Thoughts On JavaScript Generators

Monday, February 17, 2014 by K. Scott Allen
9 comments

In a previous post we used a Node.js script to copy a MongoDB collection into Azure Table Storage. The program flow is hard to follow because the asynch APIs of both drivers require us to butcher the procedural flow of steps into callback functions. For example, consider the simple case of executing three procedural steps:

  • Open a connection to MongoDB
  • Close the connection to MongoDB
  • Print “Finished!”

In code, the steps aren’t as straightforward:

var mongo = require('mongodb').MongoClient;

var main = function(){
    mongo.connect('mongodb://localhost', mongoConnected);
};

var mongoConnected = function(error, db){
    if(error) console.dir(error);
    
    db.close();
    console.log('Finished!');
};

main();

Some amount of mental effort is required to see that the mongoConnected function is a continuation of main. Callbacks are painful.

Make Me No Promises

broken promisesWhen anyone complains about the pain of callback functions, the usual recommendation is to use promises. Promises have been around for a number of years and there are a variety of promise libraries available for both Node and client scripting environments, with Q being a popular package for node.

Promises are useful, particularly when they destroy a pyramid of doom. But promises can’t un-butcher the code sliced into functions to meet the demands of an asynch API. Also, promises require shims to transform a function expecting a callback into a function returning a promise, which is what Q.nfcall will do in the following code.

var Q = require('Q');
var mongo = require('mongodb').MongoClient;

var main = function(){
    Q.nfcall(mongo.connect, 'mongodb://localhost2')
     .then(mongoConnected)
     .catch(function(error){
        console.dir(error);
     });
};

var mongoConnected = function(db){    
    db.close();
    console.log('Finished!');
};

main();

I’d say promises don’t improve the read-ability or write-ability of the code presented so far (though we can debate the usefulness if multiple calls to then are required). This is the current state of JavaScript, but JavaScript is evolving.

Generators

ECMAScript 6 (a.k.a Harmony) introduces the yield keyword. Anyone with some programming experience in C# or Python (and a number of other languages, except Ruby) will already be familiar with how yield can suspend execution of a function and return control (and a value) to the caller. At some later point, execution can return to the point just after yield occurred. In ES6, functions using the yield keyword are known as generator functions and have a special syntax (function*), as the following code demonstrates.

"use strict";

var numberGenerator = function*(){
    yield 1;
    yield 2;
    console.log("About to go to 10");
    yield 10;
};

for(let i of numberGenerator()){
    console.log(i);
};

A couple of other notes about the above code:

  1. The new let keyword gives us proper block scoping of variables.
  2. The new for-of statement allows looping over iterable objects. Arrays will be iterable, as will generator functions. 
  3. You’ll need to run Node version > 0.11 (currently “unstable”) with the –harmony flag for the code to execute.

The code will print:

1
2
About to go to 10
10

Instead of using for-of, a low level approach to working with generator functions is to work with the iterator they return. The following code will produce the same output.

let sequence = numberGenerator();
let result = sequence.next();
while(!result.done){   
    console.log(result.value);
    result = sequence.next();
}

However, what is more interesting about working with iterator objects at this level is how you can pass a value to the next method, and the value will become the result of the last yield expression inside the generator. The ability to modify the internal state of the generator is fascinating. Consider the following code, which generates 1, 2, and 20.

var numberGenerator = function*(){
    let result = yield 1;
    result = yield 2 * result;    
    result = yield 10 * result;    
};

let sequence = numberGenerator();
let result = sequence.next();
while(!result.done){  
    console.log(result.value); 
    result = sequence.next(result.value);
}

The yield keyword has interesting semantics because not only does the word itself imply the giving up of control (as in ‘I yield to temptation’), but also the production of some value (‘My tree will yield fruit’), and now we also have the ability to communicate back into the yield expression to produce a value from the caller. 

Imagine then using yield in a generator function to produce promise objects. The generator function can yield one or more promises and suspend execution to let the caller wait for the resolution of the promise. The caller can iterate over multiple promises from the generator and process each one in turn to push a result into the yield expressions.  What would it take to turn a dream into a reality?

Generators and Promises

It turns out that Q already includes an API for working with Harmony generator functions, specifically the the spawn method will immediately execute a generator. The spawn method allows us to un-butcher the code with three simple steps.

"use strict";

var Q = require('Q');
var mongo = require('mongodb').MongoClient;

var main = function*(){
    try {
        var db = yield Q.nfcall(mongo.connect, 'mongodb://localhost');
        db.close();
        console.log("Finished!");
    }
    catch(error) {
        console.dir(error);
    }
};

Q.spawn(main);

Not only does spawn un-butcher the code, but it also allows for simpler error handling as a try catch can now surround all the statements of interest. To gain a deeper understanding of spawn you can write your own. The following code is simple and makes two assumptions. First, it assumes there are no errors, and secondly it assumes all the yielded objects are promises.

var spawn = function(generator){
    
    var process = function(result){        
        
        result.value.then(function(value){
            if(!result.done) {
                process(sequence.next(value));   
            }
        });
    };

    let sequence = generator();
    let next = sequence.next();
    process(next);
};

spawn(main);

While the syntax is better, it is unfortunate that promises aren’t baked into all modules, as the shim syntax is ugly and changes the method signature. Another approach is possible, however.

Generators Without Promises

The suspend module for Node can work with or without promises, but for APIs without promises the solution is rather clever. The resume method can act as a callback factory that will allow suspend to work with yield.

var mongo = require('mongodb').MongoClient;
var suspend = require('suspend');
var resume = suspend.resume;

var main = function*(){
    try {
        let db = yield mongo.connect('mongodb://localhost', resume());
        db.close();        
        console.log("Finished!");
    }
    catch(error) {
        console.dir(error);
    }    
};

suspend.run(main);

And now the original ETL script looks like the following.

var azure = require('azure');
var mongo = require('mongodb').MongoClient;
var suspend = require('suspend');
var resume = suspend.resume;

var storageAccount = '...';
var storageAccessKey = '...';

var main = function *() {
    
    let retryOperations = new azure.ExponentialRetryPolicyFilter();
    let tableService = azure.createTableService(storageAccount, storageAccessKey)
                            .withFilter(retryOperations);
    yield tableService.createTableIfNotExists("patients", resume());

    let db = yield mongo.connect('mongodb://localhost/PatientDb', resume());
    let collection = db.collection('Patients');
    let patientCursor = collection.find();

    patientCursor.each(transformAndInsert(db, tableService));
};

var transformAndInsert = function(db, table){
    return function(error, patient){
        if (error) throw error;
        if (patient) {
            transform(patient);
            suspend.run(function*() {
                yield table.insertEntity('patients', patient, resume());
            });
            console.dir(patient);
        } else {
            db.close();
        }
    };
};

var transform = function(patient){    
    patient.PartitionKey = '' + (patient.HospitalId || '0');
    patient.RowKey = '' + patient._id;
    patient.Ailments = JSON.stringify(patient.Ailments);
    patient.Medications = JSON.stringify(patient.Medications);
    delete patient._id;    
};

suspend.run(main);

And this approach I like.

Building A Simple File Server With OWIN and Katana

Monday, February 10, 2014 by K. Scott Allen
7 comments

I have a scenario where I want to serve up HTML, JavaScript, and CSS files over HTTP from a .NET desktop application. This is the type of scenario Katana makes easy. Here is an example using a console application.

First, use NuGet to install a couple packages into the project.

install-package Microsoft.Owin.StaticFiles
install-package Microsoft.Owin.SelfHost

The source for the entire application is 17 lines of code, including using statements.

using System;
using Microsoft.Owin.Hosting;
using Owin;

namespace ConsoleApplication5
{
    class Program
    {
        static void Main(string[] args)
        {
            var url = "http://localhost:8080";
            WebApp.Start(url, builder => builder.UseFileServer(enableDirectoryBrowsing:true));            
            Console.WriteLine("Listening at " + url);
            Console.ReadLine();
        }
    }
}

The FileServer middleware will serve files from the same directory as the executable.

Static Files With OWIN and Katana

If you don’t want to use the default location, you can provide your own IFileSystem and serve files from anywhere. Katana currently provides two implementations of IFileSystem – one to serve embedded resources and one to serve files from the physical file system. You can construct a PhysicalFileSystem for an arbitrary location on the hard drive.

static void Main(string[] args)
{
    var url = "http://localhost:8080";
    var root = args.Length > 0 ? args[0] : ".";
    var fileSystem = new PhysicalFileSystem(root);

    var options = new FileServerOptions
    {
        EnableDirectoryBrowsing = true, 
        FileSystem = fileSystem                             
    };

    WebApp.Start(url, builder => builder.UseFileServer(options));            
    Console.WriteLine("Listening at " + url);
    Console.ReadLine();
}

The FileServer middleware is actually a composite wrapper around three other pieces of middleware – DefaultFiles (to select a default.html file, if present, when a request arrives for a directory), DirectoryBrowser (to list the contents of a directory if no default file is found), and StaticFile (to reply with the contents of a file in the file system). All three pieces of middleware are configurable through the UseFileServer extension method.

For example, the static file middleware will only serve files with a known content type. Although the list of known content types is extensive, you might need to serve files with uncommon extensions, in which case you can plug a custom  IContentTypeProvider into the static files middleware.

public class CustomContentTypeProvider : FileExtensionContentTypeProvider
{
    public CustomContentTypeProvider()
    {
        Mappings.Add(".nupkg", "application/zip");
    }
}

Now the final program looks like the following.

static void Main(string[] args)
{
    var url = "http://localhost:8080";
    var root = args.Length > 0 ? args[0] : ".";
    var fileSystem = new PhysicalFileSystem(root);
    var options = new FileServerOptions();
    
    options.EnableDirectoryBrowsing = true;
    options.FileSystem = fileSystem;         
    options.StaticFileOptions.ContentTypeProvider = new CustomContentTypeProvider();

    WebApp.Start(url, builder => builder.UseFileServer(options));            
    Console.WriteLine("Listening at " + url);
    Console.ReadLine();
}

Using Node.js To ETL Mongo Documents for Azure Table Storage

Tuesday, February 4, 2014 by K. Scott Allen
0 comments

In a previous post we saw how to work with the Windows Azure package for Node.js and interact with blob storage.

Another Azure adventure I’ve been tinkering with is moving MongoDb documents into Azure Table Storage. There are a couple challenges. Like Mongo, Table Storage is unstructured, but there are some limitations (not that Mongo is without limitations, but the limitations are different).

First, table storage requires every entity stored inside to include a partition key and row key. The two keys form a unique ID for an entity inside of table storage. Another limitation is that table storage only supports a few primitive data types including bool, string, date, double, and int, but there is no ability to store a collection of items directly. Thus, moving a BSON document into table storage requires some transformations to make collections work.

Here is a brute force node script to move data from a Mongo data source into table storage. The script uses the core mongodb package for Mongo since the data work is relatively low level work.

var azure = require('azure');
var MongoClient = require('mongodb').MongoClient;

var tableCreated = function(error) {
    if(error) throw error;
    MongoClient.connect('mongodb://localhost/PatientDb', mongoConnected);
}

var mongoConnected = function(error, db){
    if(error) throw error;

    var collection = db.collection("Patients");
    var patientCursor = collection.find();
    patientCursor.each(function(error, patient){
        if(error) throw error;
        if(patient) {
            transformPatient(patient);    
            loadPatient(patient);
        }
        else{
            db.close();
        }
    });
};

var transformPatient = function(patient){    
    patient.PartitionKey = '' + (patient.HospitalId || '0');
    patient.RowKey = '' + patient._id;
    patient.Ailments = JSON.stringify(patient.Ailments);
    patient.Medications = JSON.stringify(patient.Medications);
    delete patient._id;    
};

var loadPatient = function(patient){    
    tableService.insertEntity('patients', patient, function(error){
        if(error) throw error;
        console.log("Loaded " + patient.RowKey);
    });
};

var retryOperations = new azure.ExponentialRetryPolicyFilter();
var tableService = azure.createTableService()
                        .withFilter(retryOperations);
tableService.createTableIfNotExists('patients', tableCreated);

The solution I’m currently tinkering with is transforming collections into strings using JSON. It will take some more time and experiments to know if this approach is workable for the real application.

Azure Blobs with Node.js and Express

Monday, February 3, 2014 by K. Scott Allen
0 comments

The Node.js packages for Azure are fun to work with. Here’s a little walkthrough for building an Express app to display the blobs in an Azure storage container.

1. Create an application with Express.

2. Install the Windows Azure Client Library for node (npm install azure)

3. When Express created the application, it should have added a route to send a request for the root of the site to routes.index. In other words, the app.js file will look like the following.

var express = require('express')
  , routes = require('./routes')
  , path = require('path');

var app = express();

app.set('port', process.env.PORT || 3000);
app.set('views', __dirname + '/views');
app.set('view engine', 'jade');
app.use(app.router);

app.get('/', routes.index);

app.listen(app.get('port'), function(){
  console.log('Listening on ' + app.get('port'));
});

4. The index function is where the code consumes the Azure API and connects to the blob service, fetches a list of all the blobs in a container, and renders a view to display the blobs.

var azure = require('azure');

exports.index = function(request, response) {

    var accessKey = 'yourKey';
    var storageAccount= 'yourName';
    var container = 'yourContainerName';

    var blobService = azure.createBlobService(storageAccount, accessKey);
    blobService.listBlobs(container, function(error, blobs) {        
        response.render('index', {
             error: error, 
             container: container, 
             blobs: blobs
        });        
    });
}

You can pass the storage account name and the access key to createBlobService, or pass no parameters and define the environment variables AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_ACCESS_KEY.

5. Finally, the Jade view to dump blob information into the browser.

extends layout

block content
    h1 Blob listing for #{container}

    if error
        h3= error

    if blobs
        h3= container
        table
            tr
                th Name
                th Properties
            - each blob in blobs
                tr
                    td= blob.name
                    td
                        ul
                            - each value, name in blob.properties
                                if value
                                    li= name + ":" + value


Coming soon – Node.js and Azure Table Storage

Of Orcs and Software Craftsmanship

Monday, January 27, 2014 by K. Scott Allen
13 comments

Sometimes people will approach me and ask “what is it like to be a software craftsman?”. I’ll usually answer with: “Pffft, don’t ask me, I just cranked out 500 lines of script that are harder to read than Finnegans Wake”. At times though, I like to pretend what it might be like to be a software craftsperson.

For example, a few Saturdays ago I was excited about a new product and found myself chipping away at some features that would require parsing XML files. XML like the following.

<?xml version="1.0" encoding="utf-8" ?>
<SomeDocument
  xmlns="urn:my.org-v159"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
  <Container>
    <Section id="10399">
      <!-- other stuff that might have a Section -->
    </Section>
  </Container>
</SomeDocument>

Except, the real XML isn’t like the XML listed here.

XML OrcsThe XML listed here is like an orc in an animated Disney movie. It has a funny pink nose and buck teeth making it look more vulnerable than dangerous.

The real XML is like an orc in a movie derived from a J.R.R Tolkien novel. It is ugly, angry, and nearly incomprehensible when communicating. Its creator is a Standards Committee committed to enslaving humans, elves, software developers, and dwarves.  It is also just one orc in an army of repugnant orcs that cover a small continent.

Using this simple example though, let’s pretend the first goal is to retrieve the value of the id attribute in the Section element.

public class TheDocument
{
    public TheDocument(XDocument document)
    {
        Id = int.Parse(document
            .Element(_myOrg + "SomeDocument")
            .Element(_myOrg + "Container")
            .Element(_myOrg + "Section")
            .Attribute("id").Value);
    }

    public int Id { get; protected set; }      
    readonly XNamespace _myOrg = "urn:my.org-v159";
}

At this point a real craftsperson might realize that the id value is one piece of data in 376 total pieces of data that must be retrieved from the orc army. Since this one data point required 5 lines of code, the total processing would require 1,880 lines of code, which sounds excessive. Moreover, 1880 was a leap year, and leap years always create bugs in software, so the number itself is a bad omen and a sign that more work needs to be done. At least this is how I imagine a craftsperson to think.

I also imagine a real craftsman knows syntax and APIs pretty well, and in areas they don’t know well they will dig into documentation and figure out better ways of doing things. In this case using a different query method and the implicit conversion operator of an XAttribute cuts the LoC per data point to 4.

Id = (int)document
           .Descendants(_myOrg + "Section")
           .First()
           .Attribute("id");

However, 4 * 376 is 1,504, and 1504 was also a leap year. Coincidence? I don’t believe a craftsperson believes in coincidence, but they might believe in numerology, and regardless of their superstitions they certainly believe in error conditions. A craftsperson will understand the probability of receiving a proper XML document is low, because the Standards Committee, in an attempt to provide the ability to describe every possible variation of orc and sub-orc, built an XML schema so complex and full of malice that even the most sophisticated code generation tools will puke electronic bits on the floor when fed the first 5,000 lines of the associated schema file.

A craftsperson will know then, that it is a good idea to look at the types of error messages the software might produce if, for example, the Section element doesn’t exist.

Unhandled Exception: System.InvalidOperationException: Sequence contains no elements

XPath Orc Slayer These are the types of error messages that make debugging a software like debugging a 2 month old baby. You know the baby is unhappy because no living being makes these types of noises when content , but the baby can’t tell you exactly what it is wrong, it can only communicate with primitive shrieks that keep you awake at night.

A real craftsperson, I imagine, knows the business as well as the domain and the technology. And a real craftsperson will realize these types of blaring baby error messages will occur commonly and will never be solved without the assistance of a developer armed with stack trace and source code. It is then in the best interest of the business to spend additional time to craft a better solution.

But how to solve the problem?

A real craftsperson, I imagine, might take a step back and start to think about alternative approaches. A real craftsperson has more than a few years of experience under their belt, and will remember stories from a past age when orcs first became a tangible nuisance. It was then when the elders traveled to the Standard Mountain and forged a mighty blade. Behold! Its name is XPath the Orc Slayer.

document.XPathEvaluate("number(mo:SomeDocument/mo:Container/mo:Section/@id)")

XPathEvaluate is a strange beast. While the documentation for other XML related APIs drones on for paragraphs about the  minutiae of an XML Infoset, the documentation for XPathEvaluate consists of a single terse sentence.

Evaluates an XPath expression.

Regular developers like me look at documentation like this and scoff. What sort of laziness is this? I already know this method evaluates an XPath expression because that’s the name of the method for Lórien’s sake! But, I’ve always had a suspicion that software craftspeople maintain a cabal and communicate through a series of secret signs. Documentation like this must be a secret sign leading to a powerful tool. Because powerful tools can’t just fall from the sky like rain, they have to be hidden in plain sight so that only powerful people can find them and only in times of darkness and dire needs. Thus, a real software craftsperson will instantly recognize XPathEvaluate as a useful but too-generic tool that needs a little bit of gift wrapping to provide real value in an application.

First, a namespace resolver for the XPath expressions. A namespace resolver is basically a lookup table for the XPath engine to discover real namespace values.

class MyOrgXmlNamespaceResolver : IXmlNamespaceResolver
{
    public MyOrgXmlNamespaceResolver()
    {
        _namespaceMap = new Dictionary<string, string>()
        {
            { "mo", "urn:my.org-v159"},
            { "xsi", "http://www.w3.org/2001/XMLSchema-instance"}
        };
    }

    public IDictionary<string, string> GetNamespacesInScope(XmlNamespaceScope scope)
    {
        return _namespaceMap;
    }

    public string LookupNamespace(string prefix)
    {
        return _namespaceMap[prefix];
    }

    public string LookupPrefix(string namespaceName)
    {
        return _namespaceMap.First(kvp => kvp.Value == namespaceName).Value;
    }

    private IDictionary<string, string> _namespaceMap;
}

Also, a specialized exception class.

public class XmlParsingException : Exception
{
    public XmlParsingException(string xpath)
        :base(String.Format(NoLocate, xpath))
    {
    }

    public XmlParsingException(string xpath, Exception innerException)
        :base(String.Format(Problem, xpath), innerException)
    {            
    }

    const string NoLocate = "Could not locate {0}";
    const string Problem = "Problem locating {0}, see inner exception for details";
}

And finally, some syntactic sugar to dress XPathEvaluate like a tailor.

public static class XmlHelpers
 {
public static T XPathToValue<T>(this XNode node, string xpath)
{
    try
    {
        var queryResult = node.XPathEvaluate(xpath, _namespaceResolver)
            as IEnumerable;

        if (queryResult != null)
        {
            var firstResult = queryResult.OfType<XObject>().FirstOrDefault();                                                
            if (firstResult != null)
            {
                string value = "";
                if (firstResult is XAttribute)
                {
                    value = ((XAttribute)firstResult).Value;
                }
                else if (firstResult is XElement)
                {
                    value = ((XElement) firstResult).Value;
                }
                var converter = TypeDescriptor.GetConverter(typeof(T));
                return (T) converter.ConvertFrom(value);
            }
        }
    }
    catch (Exception ex)
    {
        throw new XmlParsingException(xpath, ex);
    }

    throw new XmlParsingException(xpath);
}

A real software craftsperson, I think, would start to worry because the amount of code here (as well as the cyclomatic complexity) is a bit much. After all, it took only 4 lines of brute force code to retrieve a single integer value from the orc army. But a software craftsperson, I imagine, is always looking for the tangible value in a piece of code, and there are two ways to see if this code provides any value. The first test of value is to consume the code.

public class TheDocument
{
    public TheDocument(XDocument document)
    {
        Id = document.XPathToValue<int>("mo:SomeDocument/mo:Container/mo:Section/@id");
    }

    public int Id { get; protected set; }
}

The consumption test passes. A developer can now focus on slaying orcs instead of XML APIs. The next test is to run the code, particularly with bad data, like a missing Section element.

Could not locate mo:SomeDocument/mo:Container/mo:Section/@id

Unlike the previous blaring baby error message (sequence contains no elements), this error is grown up. Also, it will allow a developer to switch from a mindset of “is this a bug in my code?” to “this is probably a bug in your XML!”, and upon further inspection 99% of the time there will be a bug in the XML, and developer can point out the error with an email like the following.

Version 3 revision 2402 of the specification (the only one we officially support) clearly states that the id attribute of the Section element must exist inside a Container of SomeDocument, except for the circumstances documented on pages 507-512, 693, 701, and the entirety of Appendix E. This is obvious to most people, but since this is the third time you’ve send an XML file with this same error YOU ARE CLEARLY A MORON.

I’m told that some developers write emails like these because they are filled with hubris, but I don’t believe everything I hear.

However, I do believe that a software craftsperson will think the above approach has some merit, because the code creates an easier API than XPathEvaluate and reduces the number of blaring baby error messages. The orcs must be trembling, but I think a software craftsperson will continue to look at the big picture and realize the  code still has two problems.

  1. It’s ugly
  2. It doesn’t support all the features needed to parse all the required information.

With regards to #2, the code will sometimes need to parse integers and strings from both attributes and elements. The code also needs to parse individual elements, and sometimes a collection of elements. To make things even trickier, some information is required, and some information is optional. All of this might point a real software craftsperson to using an extension method as a simple gateway to an object that can encapsulate and manage more complexity.

public static class XmlHelpers
{
    public static XPathEvaluator XPath(this XNode node, string xpath)
    {
        return new XPathEvaluator(node, xpath);   
    }        
}

The XPathEvaluator is responsible for parsing different types of values, and throwing exceptions when required information isn’t present.

public class XPathEvaluator
{
    public XPathEvaluator(XNode node, string xpath)
    {
        _node = node;
        _xpath = xpath;
        _required = true;
        _rawQueryResult = ExecuteExpression();
    }

    public XPathEvaluator Optional()
    {
        _required = false;
        return this;
    }

    public XElement Element()
    {
        var result = _rawQueryResult.OfType<XElement>().FirstOrDefault();
        Validate(result);
        return result;
    }

    public IList<XElement> Elements()
    {
        var result = _rawQueryResult.OfType<XElement>().ToList();
        Validate(result);
        return result;
    }

    public T Value<T>()
    {
        T result = default(T);
        try
        {
            var firstEntry = _rawQueryResult.FirstOrDefault();                
            if (firstEntry != null)
            {
                var rawResult = GetRawResult(firstEntry);
                Validate(rawResult);
                return ConvertResult<T>(rawResult);
            }
        }
        catch (Exception ex)
        {
            throw new XmlParsingException(_xpath, ex);
        }
        if (_required)
        {
            throw new XmlParsingException(_xpath);
        }
        return result;
    }

    private static T ConvertResult<T>(string rawResult)
    {
        var converter = TypeDescriptor.GetConverter(typeof (T));
        return (T) converter.ConvertFrom(rawResult);
    }

    private string GetRawResult(XObject firstEntry)
    {
        string rawResult = null;
        if (firstEntry != null)
        {
            if (firstEntry is XAttribute)
            {
                rawResult = ((XAttribute) firstEntry).Value;
            }
            else if (firstEntry is XElement)
            {
                rawResult = ((XElement) firstEntry).Value;
            }
        }
        return rawResult;
    }       

    void Validate(XElement element)
    {
        if (_required)
        {
            if (element == null)
            {
                throw new XmlParsingException(_xpath);
            }
        }
    }

    void Validate(IList<XElement> elements)
    {
        if (_required)
        {
            if (elements == null || !elements.Any())
            {
                throw new XmlParsingException(_xpath);
            }
        }
    }

    void Validate(string value)
    {
        if (_required)
        {
            if (value == null)
            {
                throw new XmlParsingException(_xpath);
            }
        }
    }

    IList<XObject> ExecuteExpression()
    {
        try
        {
            var result = (IEnumerable) _node.XPathEvaluate(_xpath, _namespaceResolver);
            return result.OfType<XObject>().ToList();
        }
        catch (Exception ex)
        {
            throw new XmlParsingException(_xpath, ex);
        }
    }

    readonly XNode _node;
    readonly string _xpath;
    bool _required;
    IEnumerable<XObject> _rawQueryResult;
    static IXmlNamespaceResolver _namespaceResolver = new MyOrgXmlNamespaceResolver();
}

This is quite a bit of code, but there are many orcs on the field of battle, and now developers can fight them with relatively simple code.

public class TheDocument
{
    public TheDocument(XDocument document)
    {
        Id = document.XPath("mo:SomeDocument/mo:Container/mo:Section/@id").Value<int>();
        Name = document.XPath("mo:SomeDocument/mo:Container/mo:Section/@name").Value<string>();
        Documentation = document.XPath("mo:SomeDocument/mo:Documentation").Optional().Value<string>();
        Container = document.XPath("mo:SomeDocument/mo:Container").Element();
        Extras = document.XPath("mo:SomeDocument/mo:Extra").Elements();
        Comments = document.XPath("mo:SomeDocument/mo:Comment").Optional().Elements();
    }

    public int Id { get; protected set; }
    public string Name { get; protected set; }
    public string Documentation { get; protected set; }
    public XElement Container { get; protected set; }
    public IList<XElement> Extras { get; protected set; }
    public IList<XElement> Comments { get; set; }
}

This is the type of thought process I imagine a software craftsperson might have, but I don’t know for sure.

It could all be rubbish.

And I might be an orc.

AngularJS Training

Tuesday, January 21, 2014 by K. Scott Allen
4 comments

UPDATE: The next public class will be the week of September 8th in Oslo, Norway.  

UPDATE 2: AngularJS: Get Started is now available to Pluralsight subscribers. This course is a small but focused subset of the full class. 

At the end of last year I put together and taught a 2 day workshop on AngularJS fundamentals at NDC London, which due to popular demand I’m offering as part of a larger class for ProgramUtvikling.  Feel free to contact me if you would like an on-site workshop, although my bandwidth for custom training is scarce.

AngularJS Fundamentals

From animations to testing and everything in between, this course covers the features of AngularJS with a focus on practical scenarios and real applications. We will see how to build custom services, directives, filters, and controllers while maintaining a separation of concerns with clean JavaScript code. Hands on labs will reinforce concepts.

The outline of topics:

  1. Introduction
  2. Modules and Controllers
  3. Views and Templates
  4. Services
  5. Directives
  6. Testing
  7. Routing
  8. UI and Animation

Some  of the material is based on blog posts here on OTC.

Implementing ASP.NET Identity

Monday, January 20, 2014 by K. Scott Allen
5 comments

In previous posts (listed below), we saw how the UserManager class in ASP.NET Identity provides the domain logic for an identity and membership system. Your software makes calls into a UserManager object to register and login users, and the UserManager will then call into a UserStore object to persist and retrieve data. 

Microsoft’s UserStore class uses the Entity Framework for persistence. If you don’t like or can’t use Microsoft’s UserStore class, then implementing the storage interfaces for ASP.NET identity is easy.

The goal of a custom storage class is to provide the basic CRUD operations required by the features your application needs. Since the storage features are factored into granular interfaces, your storage class can pick and choose the interfaces it needs to implement.

Here are the current interfaces you can choose from.

IUserStore<TUser, TKey>

The one required interface in the identity system is IUserStore. In the 2.0 alpha release, a TKey generic parameter allows you to specify the type of the identifier / primary key for a user, which was assumed to be a string in v1.0. This interface has 5 simple CRUD requirements:

  • Create a user
  • Update an existing user
  • Delete a user
  • Find a user by ID
  • Find a user by username

These methods each require < 10 lines of code, and the count of 10 includes checks for valid parameters and premature disposal. Each method only needs to forward data to a data framework or existing API. For example, a simple implementation of CreateAsync with the Entity Framework might look like the following (where _users is a DbSet<TUser>).

public Task CreateAsync(User user)
{
    user.Id = Guid.NewGuid();
    _users.Add(user);
    return _db.SaveChangesAsync();
}

And with MongoDB, users would probably be stored in a MongoCollection:

public Task CreateAsync(User user)
{
    user.Id = ObjectId.GenerateNewId();
    _db.Users.Insert(user);
    return Task.FromResult(0);
}

All other interfaces derive from IUserStore and add additional functionality. The following interfaces also take TUser and TKey generic type arguments. The generic arguments are omitted from the headings for aesthetic reasons.

IUserPasswordStore

Implement this interface in your custom user store if you want to store users with local hashed passwords. The interface will force you to implement methods to:

  • Get a user’s password
  • Set a user’s password
  • Check if a user has a password

IUserLoginStore

Implement this interface if you want to store 3rd party user logins, like logins from Twitter, Facebook, Google, and Microsoft. Again, the interface only requires some simple CRUD operations.

  • Add a login for a user
  • Find a user given their login data
  • Get all logins for a user
  • Remove a login for a user

IUserClaimStore

Manage System.Security.Claim information for users with three simple methods.

  • Get all claims for a user
  • Add a claim to a user
  • Remove a claim from a user

IUserRoleStore and IRoleStore

Implement IUserRoleStore if you want to associate roles with each user. There are a total of 4 methods required.

  • Add a user to a role
  • Get all the roles for a user
  • Check is a user is in a specific role
  • Remove a user from a role

The IRoleStore interface, like IUserStore, is a storage API with CRUD operations for role management. You’ll want to implement this interface and pass it to the ASP.NET Identity RoleManager.

IUserStore goes with UserManager; IRoleStore goes with RoleManager.

The IRoleStore interface requires 4 operations. 

  • Create a role
  • Delete a role
  • Update a role
  • Find a role by ID or name

IUserSecurityStampStore

The security stamp is best explained in a stackoverflow.com answer from team member Hao King.

… this is basically meant to represent the current snapshot of your user's credentials. So if nothing changes, the stamp will stay the same. But if the user's password is changed, or a login is removed (unlink your google/fb account), the stamp will change. This is needed for things like automatically signing users/rejecting old cookies when this occurs, which is a feature that's coming …

The interface requires only two methods.

  • Get the security stamp for a user
  • Set the security stamp for a user

IUserEmailStore and IUserConfirmationStore

New in Identity 2.0 are the abilities to confirm users via a token and allow for users to reset their password. To offer both features you’ll want these 2 interfaces. When combined they require the following operations. 

  • Find a user by email address
  • Get a user’s email address
  • Set a user’s email address
  • Check if a user is confirmed
  • Set a user’s confirmation flag

IUser*Store Interfaces and UserManager

When you create a new instance of a UserManager you pass in an object implementing at least IUserStore and 0 or more of the other IUser*Store interfaces. If you ask the user manager to do something that isn’t supported (like by calling FindByEmailAsync when your custom user store doesn’t support IUserEmailStore), the manager object throws an exception.

Working Implementations

There are a few OSS projects out there already providing IUser*Store implementations with various storage mechanisms. You can NuGet these implementations into a project, or peruse the source to get an idea of how to implement a custom user store.

Tugberk Ugurlu has an implementation for RavenDB: AspNet.Identity.RavenDB 

Daniel Wertheim has an implementation for CouchDB / Cloudant

SoftFluent has an implementation for CodeFluent Entities

InspectorIT has an implementation for MongoDB: MongoDB.AspNet.Identity

Stuart Leeks just published a store using Azure Table Storage: AspNet.Identity.TableStorage

ILMServices has an implementation for RavenDB, too: RavenDB.AspNet.Identity

Antônio Milesi Bastos has built a user store using NHibernate: NHibernate.AspNet.Identity

Bombsquad AB provides a user store for Elastic Search: Elastic Identity

aminjam built a user store on top of Redis: Redis.AspNet.Identity

cbfrank has provided T4 Templates to generate EF code for a “database first” user store: AspNet.Identity.EntityFramework

Previous Posts

by K. Scott Allen K.Scott Allen
My Pluralsight Courses
The Podcast!