Http Clients and Azure Blob Storage

Monday, March 31, 2014 by K. Scott Allen
2 comments

Although Azure Blob storage has a formal API that you can use from C#, Node, and many other environments, behind it all is a simple HTTP API. For example, with .NET you can use HttpClient to PUT a new blob into storage.

private readonly HttpClient _client = new HttpClient();
private readonly string _url = "https://pathtoblob";

public async Task WriteBlobStringAsync(string data)
{
    var content = new StringContent("Hello from C#");
    content.Headers.Add("x-ms-blob-type", "BlockBlob");

    var response = await _client.PutAsync(_url, content);
    response.EnsureSuccessStatusCode();            
}

The same can be done with Node using request.

var request = require('request');

var options = {
    url: 'https://bitmask.blob.core.windows.net/test/readme.txt?…',
    body: "Hello from Node",
    method: 'PUT',
    headers: {
        'x-ms-blob-type': 'BlockBlob'
    }    
};

var processResponse = function(error, response) {
    if (error) {
        console.log('error: ' + error);
    } else {
        console.log('response: ' + response.statusMessage);
        console.log('etag: ' + response.headers.etag);
    }    
};

request(options, processResponse);

You could also manage an upload using Node’s https module, but https is a low level API and you’d have to manage many small details, like the Content-Length header.

Uploads and Shared Access Signatures

Both these code examples expect to use URLs with Shared Access Signatures in the query string, so there is no need to know the storage account access keys or manage authorization headers. Shared access signatures allow you to grant access to various storage features in a granular fashion. There is a good overview of SAS on the Azure web site.

As an example, the following C# code will create a SAS for a “readme.txt” file in the “test” storage container. The SAS is good for ~4 hours and grants someone read and write privileges on the readme.txt blob. Note that the readme.txt blob does not have to exist before the code creates the SAS.

var storageAccount = CloudStorageAccount.Parse(
    CloudConfigurationManager.GetSetting("ConnectionNameInConfig"));
var blobClient = storageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference("test");

var blob = container.GetBlockBlobReference("readme.txt");
var sasConstraints = new SharedAccessBlobPolicy
{
    SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-15),
    SharedAccessExpiryTime = DateTime.UtcNow.AddHours(4),
    Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write
};
string sasBlobToken = blob.GetSharedAccessSignature(sasConstraints);
return blob.Uri + sasBlobToken;

The SAS combined with the blob URL will look like:

https://bitmask.blob.core.windows.net/test/readme.txt?sv=2013-08-15&sr=b&sig=---&st=2014-03-15T15%3A27%3A14Z&se=2014-03-15T19%3A42%3A14Z&sp=rw

The SAS can be handed to clients who can now directly access storage instead of streaming data through your application as an intermediary.  The example here is what Azure calls an Ad hoc SAS, because all of the details, like the permissions and expiry time are in the URL itself.

Some Useful IIS Rewrite Rules

Thursday, March 27, 2014 by K. Scott Allen
0 comments

A few months ago Mads posted some IIS URL Rewrite rules in a post titled “URL rewrite and the www subdomain”.

Years ago, when I rewrote this site in ASP.NET MVC, I found URL rewriting to be invaluable. Some URLs in the new version of this site became obsolete. For example, an efficient Gravatar implementation from the Web Helpers library replaced an Identicon HTTP handler. I wanted to explicitly purge the handler from search engine results with an HTTP 410 response. 

<rule name="obsolete identicon" stopProcessing="true">
  <match url="/IdenticonHandler.ashx" />
  <action type="CustomResponse" statusCode="410" statusReason="Gone" 
    statusDescription="…" />
</rule>

Instead of having 4 different RSS feeds for different sections of the site, I collapsed all content into a single RSS feed. All previous RSS endpoints now redirect to FeedBurner.

<rule name="article rss feed" stopProcessing="true">
  <match url="articles/rss.aspx" />
  <action type="Redirect" url="http://feeds.feedburner.com/OdeToCode" redirectType="Permanent" />
</rule>

To make routing a bit easier, I wanted to avoid processing URLs like /articles or /blogs in ASP.NET and redirect those requests to to /articles/list with rules like the following.

<rule name="avoid articles directory" stopProcessing="true">
  <match url="articles[/]?$" />
  <action type="Redirect" url="articles/list" redirectType="Permanent" />
</rule>

A tricky scenario was preserving some endpoints that used the classic ASP.NET page name of “default.aspx” in the URL. For example, the list of all blog posts used to exist at /blogs/all/default.aspx, but I wanted to redirect these requests and avoid the page name.

<rule name="default page" stopProcessing="true">
  <match url="(.*)default.aspx" />
  <conditions>
    <add input="{REQUEST_URI}" negate="true" pattern="-default.aspx$" />
  </conditions>
  <action type="Redirect" url="{r:1}" redirectType="Permanent" />
</rule>

The negation condition in the above rule avoids redirecting requests for blog posts with default.aspx in the title of the post, of which there are 1 or 2.

Finally, I use a web.config transformation to add an additional rule in production to enforce the canonical host name of odetocode.com.

<rule name="Canonical Host Name" stopProcessing="true" 
      xdt:Transform="InsertBefore(/configuration/system.webServer/rewrite/rules/rule[1])">
  <match url="(.*)" />
  <conditions>
    <add input="{HTTP_HOST}" negate="true" pattern="^odetocode\.com$" />
  </conditions>
  <action type="Redirect" url="http://odetocode.com/{R:1}" redirectType="Permanent" />
</rule>

One post I found useful when developing these rules was RuslanY’s “10 URL Rewriting Tips and Tricks”.

Tips For JavaScript Promises

Wednesday, March 26, 2014 by K. Scott Allen
1 comment

Q tipsThere are couple scenarios where I occasionally see too much JavaScript code being written with promises.

For the examples, let’s assume we are working with a simple function returning a promise like the following doWork function. This code is using q, but everything here is also true for Angular’s $q service.

var doWork = function(){
    var deferred = Q.defer();

    setTimeout(function(){
        deferred.resolve("done");
    }, 1000);

    return deferred.promise;
};

Invoking doWork to get a result is simple.

var onSuccess = function(result) {
    console.log("Success: " + result);
};

var onError = function(reason) {
    console.log("Error: " + reason);
}

doWork().then(onSuccess, onError);

Wrapping A Promise Function

The first scenario that often involves too much code is the scenario where you want to wrap a promise to add some additional calculations after the initial promise resolves, but before returning a promise to a higher level component. Examples would include processing an HTTP response to add caching or some data manipulation.

With the doWork function, we might just want to add some additional text (“with added value”) to the return.

/* don't do this */
var workWrapper = function() {
    var defer = Q.defer();
    
    doWork().then(function(result) {
        defer.resolve(result + " with added value");
    }, function(reason){
        defer.reject(reason);
    });

    return defer.promise;
};

The above code goes to a lot of extra work to create a new deferred object and handle errors, but the same result could be achieved with less code.

/* do this instead */
var workWrapper = function() {
    return doWork().then(function(result) {
        return result + " with added value";
    });
};

An error will still prorogate correctly, and the then function will capture then plain return value (a string) and wrap it in a promise. To the caller, it’s still easy to grab the final result.

workWrapper().then(onSuccess, onError);

Do I Have A Promise Or Not?

A similar scenario exists when you have a function that may or may not return a promise. For example, if request is made for some data but the data is found in a cache, a function can return the data immediately. If the data isn’t in a cache the function might need to make an asynch call to fetch the data and return a promise for the future.

The below function simulates this scenario by returning a raw string value approximately half the time, and a promise the rest of the time.

var promiseOrValue = function() {
    if(Math.random() < 0.5) {
        return "done early";
    }
    else {
        return doWork();
    }
};

The easiest way to handle this scenario is not to test the return value to see if the value is a promise, but treat everything as a promise, which is easy to do with Q.when.

Q.when(promiseOrValue()).then(onSuccess, onError);

If you own the function, it’s even nicer if the function always returns a promise, even for data that is immediately available.

var alwaysAPromise= function() {
    if(Math.random() < 0.8) {
        return Q.when("done early");
    }
    else {
        return doWork();
    }
};

Then invoking the function is as easy as invoking the original doWork function.

alwaysAPromise().then(onSuccess, onError);

A Plunkr

If you want to experiement with promises, I put together a Plunkr using Jasmine to describe some of these behaviors with tests.

Dynamic Routes with AngularJS

Monday, March 24, 2014 by K. Scott Allen
7 comments

There is a simple rule in AngularJS that trips up many people because they simply aren’t aware of the rule. The rule is that every module has two phases, a configuration phase and a run phase. During the configuration phase you can only use service providers and constants, but during the run phase you only have access to services, and not the service providers.

One scenario where the rule will trip people up is the scenario where an application needs flexible, dynamic routes. Perhaps the routes are tailored to a user’s roles, like giving additional routes to a superuser, but regardless of the specifics you probably need some information from the server to generate the routes. The typical approach to server communication is to use the $http service, so a first attempt might be to write a config function that uses $http and $routeProvider to put together information on the available routes.

app.config(function ($http, $routeProvider) {

    var routes = $http.get("userInfo");
    // ... register routes with $routeProvider
                   
});

The above code will only generate an error.

Error: [$injector:unpr] Unknown provider: $http

Eventually you’ll figure out that a config function only has access to $httpProvider, not $http. Then you might try a run block, which does give you access to $http for server communication, but …

app.run(function ($http, $routeProvider) {

    var routes = $http.get("userInfo");
    // ... register routes with $routeProvider
    
});

… there is no access to providers during a run block.

[$injector:unpr] Unknown provider: $routeProviderProvider

There are a few different approaches to tackling this problem.

One approach would be to use a different wrapper for service communication, like jQuery’s $.get, perhaps combined with manual bootstrapping of the Angular application to ensure you have everything from the server you need to get started.

An Solution With C# and Razor

Another approach would be to use server side rendering to embed the information you need into the shell page of the application. For example, let’s say you are using the following class definitions.

public class ClientRoute
{
    public string Path { get; set; }
    public ClientRouteProperties Properties { get; set; }
}

public class ClientRouteProperties
{
    public string TemplateUrl { get; set; }
    public string Controller { get; set; }
    public string Resolve { get; set; }
}

And also a ClientRouteBuilder that can generate client side routes given the identity of a  user.

public class ClientRouteBuilder
{
    public string BuildRoutesFor(IPrincipal user)
    {
        var routes = new List<ClientRoute>()
        {
            new ClientRoute { 
                Path = "/index",
                Properties = new ClientRouteProperties
                {
                    TemplateUrl = "index.html",
                    Controller = "IndexController"
                }
            }
            
            // ... more routes
        };

        if (user.IsInRole("admin"))
        {
            routes.Add(new ClientRoute
            {
                Path = "/admin",
                Properties = new ClientRouteProperties
                {
                    TemplateUrl = "admin.html",
                    Controller = "AdminController"
                }
            });
        }

        return JsonConvert.SerializeObject(routes,new JsonSerializerSettings()
        {
            ContractResolver = new CamelCasePropertyNamesContractResolver()
        });
    }
}

In a Razor view you can use the builder to emit a JavaScript data structure with all the required routes, and embed the JavaScript required to config the application in the view as well.

<body ng-app="app">
    <div ng-view>
        
    </div>
    
    <script src="~/Scripts/angular.js"></script>
    <script src="~/Scripts/angular-route.js"></script>
    <script>
        (function() {

            var app = angular.module("app", ["ngRoute"]);

            /*** embed the routes ***/
            var routes = @Html.Raw(new ClientRouteBuilder().BuildRoutesFor(User))

            /*** register the routes ***/
            app.config(function ($routeProvider) {
                routes.forEach(function(route) {
                    $routeProvider.when(route.path, route.properties);
                 });
                $routeProvider.otherwise({
                    redirectTo: routes[0].path
                });
            });
        }());    

    </script>
    @* Rest of the app scripts *@
</body>

And Remember

You can’t effectively enforce security on the client, so views and API calls still need authorization on the server to make sure a malicious user hasn’t manipulated the routes.

Durandal and Object.defineProperty

Thursday, March 20, 2014 by K. Scott Allen
6 comments

DurandalJS continues to make dramatic improvements under the direction of lead architect Rob Eisenberg. The framework is easy to pick up since the API is small, well designed, fully featured, and uses technologies familiar to many JavaScript developers, like jQuery, Knockout, and RequireJS.

I’ve been working with Durandal 2.0.1 and my favorite feature, by far, is the observable plugin which allows binding to plain JavaScript objects.

When creating the application, I only need to tell Durandal to use the observable plugin.

define(function(require) {
    var app = require("durandal/app");
    app.configurePlugins({        
        observable: true // <-
    });

    app.start().then(function() {
        app.setRoot("{viewModelName}", "entrance");
    });
});

And now all my view models can be plain, simple objects.

define(function(require) {
    var dataService = require("data/movieData");
    var viewModel = {

        movies: [],

        activate: function() {
            dataService.getAll().then(function(newMovies) {
                viewModel.movies = newMovies;
            });
        }
    };
    return viewModel;
});

How’s It Work?

At the core of the observable plugin is Object.defineProperty. This ES5 API requires a compatible browser, but fortunately even IE9 has support. The defineProperty method can build a property with get and set logic, as in the following example.

var person = {};

var propertyDefinition = function(value) {
    
    var get = function(){
        return value;
    };

    var set = function(newValue) {
        if(value != newValue) {
            value = newValue;
            console.log("Value changed to " + value);
        }
    };

    return {
        configurable: true,
        enumerable: true,
        get: get,
        set: set
    };    
};

Object.defineProperty(
    person, 
    "firstName", 
    propertyDefinition()
);

person.firstName = "Scott";
person.firstName = "Allen";
console.log(person.firstName);

With Durandal, you never have to worry about using defineProperty directly. Durandal’s observable plugin provides an API to convert all the properties of an object into Knockout compatible observables using defineProperty.

define(function(require){
    var observable = require("durandal/plugins/observable");

    var person = {
        firstName: "Scott",
        lastName: "Allen"
    };

    observable.convertObject(person);
    
    // can still use properties as properties instead of as functions
    person.firstName = "Ethan";
    console.log(person.firstName);
});

If we look at the object in the debugger, we’ll see the following.

Observable with defineProperty

But even the above code is something you don’t need to write, because Durandal will automatically convert view models into observable objects ready for 2 way data binding before bindings are applied. There’s a lot to be said for frameworks that care about making things easy for the developer.

Rethinking Biggy

Wednesday, March 19, 2014 by K. Scott Allen
5 comments

A few weeks ago Rob unveiled Biggy, a simple file based document store for .NET inspired by NeDB. Since then there’s been a few additions and Biggy also works with relational databases and MongoDB.

Looking through the code and the enhancement list, I couldn’t help wondering if Biggy might benefit from a different design. I started to open an issue on GitHub based on a completely experimental branch I created, then decided it would be better off as a post.

How It Currently Works

Looking over the Biggy implementation, every different data store becomes coupled to an InMemoryList<T> class through inheritance. The coupling isn’t necessarily wrong, but it does complicate the implementation of each new data store. For example, for JSON storage the Add method has to remember to call into the base class in order to raise the proper events:

public void Add(T item) {
  var json = JsonConvert.SerializeObject(item);
  using (var writer = File.AppendText(this.DbPath)) {
    writer.WriteLine(json);
  }
  base.Add(item);
}

The inheritance relationship also complicates life for consumers, as InMemoryList supports both Clear and Purge methods in the API, but the JSON implementation only supports Clear. Not sure if this was intentional or not, but I did find it confusing.

I also thought an alternate approach that clearly defines the responsibilities of the in-memory data manager and the backing data store might be helpful…

Separating Lists From Stores

First we’ll start off with an abstraction that clearly defines the capabilities of a Biggy in-memory list.

public interface IBiggy<T> : IEnumerable<T>
{
    void Clear();
    int Count();
    T Update(T item);
    T Remove(T item);
    T Add(T item);
    IList<T> Add(IList<T> items);
    IQueryable<T> AsQueryable();

    event EventHandler<BiggyEventArgs<T>> ItemRemoved;
    event EventHandler<BiggyEventArgs<T>> ItemAdded;
    event EventHandler<BiggyEventArgs<T>> Changed;
    event EventHandler<BiggyEventArgs<T>> Loaded;
    event EventHandler<BiggyEventArgs<T>> Saved;
}

Methods like Add and Remove will return the affected item in case something changed, like if the underlying store populates a key or version field. There are two versions of Add, because batch inserts are a common scenario.

The implementation of an IBiggy can now focus on manipulating in-memory data and firing events. All data persistence is handled by stores.

public virtual T Add(T item)
{
    _store.Add(item);
    _items.Add(item);
    Fire(ItemAdded, item: item);
    return item;
}

A Biggy store has a simple, brute force API.

public interface IBiggyStore<T>
{
    IList<T> Load();
    void SaveAll(IList<T> items);
    void Clear();     
    T Add(T item);
    IEnumerable<T> Add(IEnumerable<T> items);
}

But I’m also thinking some data stores might support additional features that make updates and queries more efficient. These capabilities are segregated into separate interfaces.

public interface IUpdateableBiggyStore<T> : IBiggyStore<T>
{
    T Update(T item);
    T Remove(T item);
}

public interface IQueryableBiggyStore<T> : IBiggyStore<T>
{
    IQueryable<T> AsQueryable();
}

And when constructing a BiggyList<T>, you have to inject a specific data store. BiggyList<T> can query the available interfaces in a data store to understand the store’s capabilities, but once the store is inject a list client never needs to know about the store.

public BiggyList(IBiggyStore<T> store)
{
    _store = store;
    _queryableStore = _store as IQueryableBiggyStore<T>;
    _updateableBiggyStore = _store as IUpdateableBiggyStore<T>;
    _items = _store.Load();
}

Now, the implementation of an actual data store doesn’t need to call into a base class or worry about raising events. The store only does what it is told. Circling back to the JSON backing store, an implementation might look like:

T IBiggyStore<T>.Add(T item)
{
    var json = JsonConvert.SerializeObject(item);
    using (var writer = File.AppendText(DbPath))
    {
        writer.WriteLine(json);
    }
    return item;
}

Another useful benefit to this approach is that each store can specify generic constraints independently of the Biggy list or other data stores. For example, a data store for SQL Server can specify that T has to implement an interface with an ID property, with one for Azure Table Storage can enforce partition and row keys. 

Is It Useful?

All the static typing and interfaces might move Biggy away from Rob’s initial vision of a lightweight and easy to use library, but I think the ability to cleanly separate stores from lists is valuable not just for extensibility but in the simplicity of the design.

Working With FIPS 140 Crypto Standards

Tuesday, March 18, 2014 by K. Scott Allen
1 comment

Kryptos sculptorTo explain the 140 series of the Federal Information Processing Standards (FIPS) in plain talk and without dozing off is challenging, but let me give it a try:

FIPS cryptographic standards specify design and implementation requirements for cryptographic modules. Software and hardware vendors can contract with accredited laboratories to validate their modules against these standards, which then allows others to use those modules in computing environments that must adhere to the U.S. government’s information processing standards.

Two questions come to mind.

First, what is a cryptographic module? A module might be a piece of software, hardware, or a combination of both. For example, RSAENH.DLL on various Windows platforms is FIPS complaint. Another example would be the Samsung crypto modules running on Android devices like the Galaxy S4.

Second question, from a software developer’s perspective, is who must use FIPS compliant modules? The obvious answer is most anyone working directly or indirectly for a department or agency of the United States federal government who is handling sensitive but unclassified data. However, FIPS has also made inroads into private sector healthcare and banking businesses, as both industries store and transmit sensitive data like credit card numbers and personal health information.

The FIPS Switch

Some businesses will enforce the use of FIPS compliant algorithms by flipping the “FIPS switch”.  This is a setting on operating systems that ensure applications only use FIPS verified cryptography algorithms. You can flip this switch on Windows, as well as OS X and other operating systems and devices.

On Windows, the FIPS switch impacts the entire system and all applications, from BitLocker to Internet Explorer and Remote Desktop. The FIPS switch will also impact the code you write. Even if you don’t want to use a FIPS compliant algorithm, if your C# code executes on a Windows machine with the FIPS switch on, you’ll have to use a FIPS compliant algorithm or the code will fail.

For example, with C#, the managed .NET implementations of AES encryption are not certified, so the following code fails with an exception.

using System.Security.Cryptography;

static class Program 
{
    static void Main()
    {
        var provider = new AesManaged();
    }
}

Unhandled Exception: System.InvalidOperationException: This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms.

Instead of using AesManaged, you’ll need to use AesCryptoServiceProvider, which calls into native, verified modules.  And just so you know, your application won’t be the only one to face these types of exceptions, as everything from Visual Studio to Internet Explorer and even frameworks like ASP.NET and databases like MongoDB have had (or still have) troubles with FIPS at one point or another. Be careful when you flip on FIPS that you don’t cripple your own development machine, you might need to reserve FIPS for a testing environment.

The FIPS Whip

One of the issues you might run into with the FIPS switch is how the system will use brute force to deny access to uncertified encryption modules. There is no awareness from the OS of why an application might choose to use a specific algorithm. In the case of AES the brute force approach might make sense, but when you start to think of algorithms like MD5 the situation is grey. MD5 might be used cryptographically to generate a message digest, but someone might have also chosen MD5 to generate a hash to use as the key value for plain text data in a distributed cache. But, MD5 is considered weak from a crypto perspective, and the following code will also fail with the exception we saw previously, even though the code is using a crypto service provider.

using System.Security.Cryptography;

static class Program 
{
    static void Main()
    {
        var provider = new MD5CryptoServiceProvider();
    }
}

To run on a computer with the FIPS switch on, then, you need to be careful about your choice of anything cryptographic.

The irony of the FIPS switch is that the system can only prevent what the system knows about. Microsoft programmed the .NET crypto classes to respond with an exception when used inappropriately, but there are other libraries and platforms that will run any kind of cryptographic algorithm on a machine with the FIPS switch on. For example, I can still run the following code in Node on a FIPS enabled Windows machine.

var crypto = require("crypto");
var message = "The Magic Words are Squeamish Ossifrage";
var hash = crypto.createHash("md5").update(message).digest("hex");

console.log(hash);  

So Is The FIPS Flag Useful?

I think the question is debatable.  The FIPS flag will keep the honest applications honest. But the FIPS flag doesn’t guarantee that an application encrypts the right data, or that an application encrypts data at the right time, or that an application developer doesn’t “work around” the FIPS flag by writing their own algorithm with XOR and clocking out. The FIPS flag also can’t stop a user from storing their passwords on a yellow sticky note affixed to the back of an LCD monitor.

The real question should be: “is the data on a FIPS enabled machine more secure than the data on a machine without the FIPS flag?”

I’d say the answer is unquestionably a “no”, and system administrators should know the FIPS flag is not a silver bullet for security.

by K. Scott Allen K.Scott Allen
My Pluralsight Courses
The Podcast!