Data Visualization With d3.js

Tuesday, July 23, 2013 by K. Scott Allen
3 comments

Last week I had an idea for visualizing the flow of patient data through an algorithmic process and tried out d3.js. d3 is short for data-driven documents. If you follow the previous link you’ll find some impressive visualizations made with d3 and SVG (the d3 Show Reel is just one example).

d3 is a bit jQuery-ish in the sense it can manipulate the DOM, select elements, and set attributes. For the work I was doing there are 2 distinguishing features of d3:

- d3 can join elements with data sources. The end result is similar to using a repeater with data binding.

- d3 can use a function to calculate attribute values instead of using hard coded values.

For example, let’s say we have the following data:.

var data =
[
    { name: "Cost" },
    { name: "Scope" },
    { name: "Time" }
];

We’ll combine the data with some d3 code:

d3.selectAll("p")
  .data(data)
  .enter()
  .append("p")
  .text(function(d) { return d.name });

The d3 API takes some work to understand, but the above code reads like this:

image1. Find all paragraph elements in the document.

2. Join the elements with the data array

3. If there is a data element with no paragraph …

4. Then append a new paragraph to the document

5. And set the text of the paragraph to the name attribute of each data item

The result being that three new paragraphs will appear on the page using text from the data array. Of course this example is uninspiring and entirely boring, but also easy to understand.

Layouts

Another great feature of d3 are the layouts the library supports. The force layout uses verlet integration (one of my favorites) to constrain object positions. Let’s use the same data as earlier, but this time use SVG and a force layout to create the famous iron triangle of project management (but in this case a triangle that no matter how far you pull of the edges, the triangle will always rebalance itself).

First, we’ll need to augment our data with some links. d3 will process the source and target values in data links as array indexes into the data array and know that cost relates to scope, scope relates to time, and time relates to cost.

var data =
  [
    { name: "Cost" },
    { name: "Scope" },
    { name: "Time" }
  ];

var dataLinks = 
  [
    { source: 0, target: 1 },
    { source: 1, target: 2 },
    { source: 2, target: 0 }
  ];

The next step is to create the force layout given an area of 960x500, keeping a distance of 220px between nodes, and maintaining a slight negative charge (repulsion) between elements. There is also a “game loop” setup using a tick function to update the screen (we’ll see it later).

var force = 
    d3.layout.force()
      .size([960, 500])
      .nodes(data) 
      .links(dataLinks)
      .linkDistance(220)
      .charge(-20)
      .on("tick", tick);

The layout is an abstraction in the sense it will set x and y properties on our data and link objects, but it is up to us to make something appear on the screen. So, we’ll use d3 to create an svg element in the document, then a <line> for each link, and a <circle> and <text> for each data element.

var svg = d3.select("body").append("svg")
            .attr("width", 960)
            .attr("height", 500);

var links = svg.selectAll(".link")
               .data(dataLinks)
               .enter().append("line")
               .attr("class", "link");

var nodes = svg.selectAll(".node")
               .data(data)
               .enter().append("circle")
               .attr("class", "node")
               .attr("r", 40)
               .call(force.drag);
    
var text = svg.selectAll(".text")
              .data(data)
              .enter()
              .append("text").attr("class", "text")
              .text(function (d) { return d.name; });

force.start();

As the force layout applies its logic it will update x and y properties on our data elements and links. Now it is our job to update all the lines, circles, and text elements on each tick loop with their associated values from the data objects. As objects are dragged around the screen, the force layout will continue to update these values.

function tick() {
    links.attr("x1", function (d) { return d.source.x; })
        .attr("y1", function (d) { return d.source.y; })
        .attr("x2", function (d) { return d.target.x; })
        .attr("y2", function (d) { return d.target.y; });

    nodes.attr("cx", function (d) { return d.x; })
        .attr("cy", function (d) { return d.y; });

    text.attr("x", function (d) { return d.x - 15; })
        .attr("y", function (d) { return d.y + 5; })
        .text(function (d) { return d.name; });
}

The end result is the “perfect project manager” who always keeps the iron triangle perfectly balanced.

Iron triangle with d3.js

Try it for yourself at http://jsfiddle.net/By5vH/3/.

Learning To Program Abstractions

Monday, July 22, 2013 by K. Scott Allen
4 comments

Last week Pluralsight published the 2nd course in my Learning to Program series. It is “Learning To Program - Part 2: Abstractions”, and the course focuses on some of the topics I had to unlearn to learn.

Unlearn to Learn?

unlearnWhen I started learning to program Basic and assembly languages on a TI-99 4/A, I  had no idea what computer programming was about and very few people who I could ask for help. Most of my time was spent trying to understand the syntax of the languages and putting together the right sequence of characters to make something work. I never gave a thought to algorithms, design, or code aesthetics, as the primary motivator was to color pixels on the screen and play musical notes.

I do remember being stumped trying to generate a random number using assembly language instructions. How do you get a giant calculator to produce an accidental value? This question led to hours of metaphysical thinking by a teenage boy. The whole idea of randomness with electronics never did click for me till Douglas Adams wrote about putting an atomic vector plotter inside a Brownian Motion producer.

University mostly reinforced the idea that computer programming was all about technical knowledge. So did every job interview I ever had.

It’s easy to be a software developer and not see the forest for the trees. It took me a long time to change my focus.

How This Course Is Different

In this course I wanted to give students a different perspective on programming than the one I started with by deemphasizing low level technical details and language syntax as much as possible. Instead I wanted to emphasize how to think about software decomposition and composition from the start, so a student would have some idea about the forest they live in. That’s one reason I chose Python as the programming language for this course.

python

Python is a beautiful, clutter free language that runs nearly everywhere. You can focus on design after learning just a few keywords and symbols. Python is also easy to work with. The REPL can provide instant feedback on both the syntax and the behavior of code.

In the course I cover everything from how to use the REPL to the different data structures available with Python. I also show how to break a program into functions and how to use a class definitions to create objects.

I talk about object oriented programming but I never talk about inheritance and polymorphism. Instead, I talk about building abstractions. Functions are abstractions, and objects are abstractions, too. When I was a student I would think that a hash table would be a perfectly reasonable abstraction for a shopping cart. I want today’s students to know the hash table can be an implementation detail and something you can encapsulate into a better abstraction for a shopping cart.

This course is about the single responsibility principle and the mantra of “design for use, not reuse”, although those words are never spoken verbatim. Knowing Big-O notation and how to work with pointers is incredibly important to build software correctly. But, I think those concepts receive too much of the spotlight in programming introductions, leaving many to believe, like I did, that you should solve every problem using lists of strings and numbers.

I hope everyone who watches enjoys the course and starts off with a better perspective on how to build software.

AngularJS: Listening for $destroy

Tuesday, July 16, 2013 by K. Scott Allen
1 comment

Angular will broadcast a $destroy event just before tearing down a scope and removing the scope from its parent.

Listening for this event is crucial for cleaning up tasks and resources that otherwise might continue to chew up memory or CPU.

As an example, the following controller continuously updates a model value in one second intervals, and these updates will continue forever, even after the controller’s view is gone and the scope is removed from its parent. Even worse, if the user is navigating back and forth to a view that loads this controller, each navigation will add another timer that runs forever.

module.controller("TestController", function($scope, $timeout) {

    var onTimeout = function() {
        $scope.value += 1;
        $timeout(onTimeout, 1000);
    };
    $timeout(onTimeout, 1000);
    $scope.value = 0;
           
});

Listening for the $destroy event is an opportunity to halt the timer. One approach is to cancel the promise returned by $timeout.

module.controller("TestController", function($scope, $timeout) {
    
    var onTimeout = function() {
        $scope.value += 1;
        timer = $timeout(onTimeout, 1000);
    };
    var timer = $timeout(onTimeout, 1000);
    
    $scope.value = 0;

    $scope.$on("$destroy", function() {
        if (timer) {
            $timeout.cancel(timer);
        }
    });
});

Chrome Workspaces: Edit Source From The Chrome Dev Tools

Monday, July 15, 2013 by K. Scott Allen
6 comments

The Workspace feature of Chrome’s Developer Tools allows you to live edit the source code to a web application and have your changes saved on your local file system. The Workspace feature is now available not yet available in the stable builds of Chrome, but sounds like it will be soon. 

Getting Workspaces Setup

The first step is to open the Settings panel by clicking the “gears” icon in the lower right of the tools window. Then select “Workspace”. Under “Folders” you can select one or more folder where your source lives, or even a parent folder like your parent folder for all git repositories. The Workspace features is fairly smart when it comes to finding source code files inside.

Mapping Folders for Chrome Dev Tools Workspaces

Theoretically you could select the root directory of a drive, but I’m not clear on the implications of granting access, and after selecting a folder you’ll have to say yes to a  popup warning about making sure you “do not expose sensitive information”. Until there is a definitive document (I couldn’t find one), I’d have to assume third party extensions would be able to poke around the file system looking for plaintext passwords inside of config files.

Once a file system location is mounted, you can start to edit files in the Sources tab of the developer tools. You can open and edit any text file from the file system tree in the Navigator window of the Sources panel, even non web files like build files or C# code.

Of course the files you will be most interested in editing are the JavaScript and CSS files that are loaded into the browser from a web application. You can “map” a file loaded into the browser to a file on the file system by right-clicking the file in the Navigator window and selecting the matching file on your hard drive.

Mapping browser files to the file system

Once that step is complete, you can now start editing the JavaScript and CSS of an application and have the changes saved (Ctrl+S) to your local file system.

Where Workspaces Is Great

For editing the CSS of an application, the Workspaces experience is unsurpassed by any other development environment. You get to see your live site running in the browser with live data. The changes you make in the CSS are instantly reflected in the browser, and when the UI looks good, pressing Ctrl+S will save your changes into your CSS file. You can edit the CSS file directly, but you can also edit CSS from the Elements pane for fine-tuning, and these changes are immediately persisted. It is the ultimate WYSIWYG designer.

Editing CSS in Chrome Workspaces

 

Where Workspaces Is Good

You can edit other files, too, like JavaScript files, but these generally require a refresh of the browser to see the changes running. For example, you can open up a file of Jasmine tests and create a new test, but running the test requires a refresh. It would be nice to have an option of automatic re-execution in the spirit of today’s automated test runners.

Obviously  a change to compiled code, like a C# file, requires a re-build outside of the browser and dev tools. It would be nice to have some hooks to do this automatically like some text editors can do.

Where Workspaces Could Improve

As for the Chrome Dev Tools becoming an everyday all day editor + IDE, it still has a ways to go. The tools have always behaved like a gifted but insubordinate child. Hitting backspace or enter at the wrong time can wipe out a style definition (fortunately Ctrl+Z undo can bring the style back), and the UI often feels cramped, jumbled (even on a big screen), and quirky (as I’m typing, an unlabeled checkbox has appeared in the middle of nowhere). The tools also feel transient, since closing the wrong browser tab will close the associated workspace, and the editing ability is still missing some key features provided by other editors, like columnar selection, auto-completion for CSS properties, and commands or triggers to interact with external tools.

Despite these shortcomings, the Chrome dev tools are still head and shoulders above the tools in other browsers. Plus, they get better with every frequent release.

Looking Back: Python

Thursday, July 11, 2013 by K. Scott Allen
4 comments

Rabbints, Campfires, PythonTrue story: My first use of Python was a little over 10 years ago.  I used Python to prevent rabbits from running into campfires and injuring themselves.

Of course the rabbits were virtual, as were the campfires. I was working with a team to create MMORPG middleware, and Python scripts were the brains for some of the lesser NPCs.

I’ve been working with Python again recently.

class Cart:

    def __init__(self):
        self.contents = dict()

    def process(self, order):        
        if order.add:            
            if not order.item in self.contents:
                self.contents[order.item] = 0
            self.contents[order.item] += 1

        if order.delete:
            if order.item in self.contents:
                self.contents[order.item] -= 1
                if self.contents[order.item] == 0:
                    del self.contents[order.item]

    def __repr__(self):
        return "Cart: {0!r}".format(self.__dict__)

It’s interesting how my opinion of Python has changed. Ten years ago the majority of my programming experience was with C, C++, and Java. My thoughts on Python were:

1. Using indentations to control block structure was weird, considering white space was insignificant everywhere else.

2. Tuples were useful.

3.  Double underscores look cool

Todays thoughts:

1. Tuples are still useful, but the REPL perhaps more so.

2. How did I miss lambdas, generators, map, filter, and reduce?

3. Double underscores are ugly, but the absence of { and } and is beautiful.

I like Python, it’s a great language. I have a better appreciation for its features today than I did 10 years ago.

AngularJS Drag and Drop Photo Directive

Wednesday, July 10, 2013 by K. Scott Allen
3 comments

Continuing from previous posts on building a file input directive and a file reader service, this post contains my first try at a drag-n-drop directive that uses the file reader service to copy an image dropped from the desktop into an img element.

As always, I welcome suggestions!

Native HTML 5 drag-and-drop is easy to work with. The directive handles the dragover, dragleave, and drop events on the target element. Dragover and dragleave are mostly about manipulating classes on the element to style it as a droppable target, as well as using e.preventDefault(), which is required for the drop event to work.

module.directive("imageDrop",
  function ($parse, fileReader, resampler) {

    return {
        restrict: "EA",
        link: function (scope, element, attrs) {

            var expression = attrs.imageDrop;
            var accesor = $parse(expression);


            var onDragOver = function (e) {
                e.preventDefault();
                element.addClass("dragOver");
            };

            var onDragEnd = function (e) {
                e.preventDefault();
                element.removeClass("dragOver");
            };

            var placeImage = function (imageData) {
                accesor.assign(scope, imageData);
            };

            var resampleImage = function (imageData) {                       
                return resampler.resample(
                    imageData, element.width(),
                    element.height(), scope);
            };

            var loadFile = function (file) {
                fileReader
                    .readAsDataUrl(file, scope)
                    .then(resampleImage)
                    .then(placeImage);
            };


            element.bind("dragover", onDragOver)
                   .bind("dragleave", onDragEnd)
                   .bind("drop", function (e) {
                       onDragEnd(e);
                       loadFile(e.originalEvent.dataTransfer.files[0]);
                   });

            scope.$watch(expression, function () {
                element.attr("src", accesor(scope));
            });
        }
    };
});

Some of the code to ensure only images are being processed is omitted, but I do want to point out the  image resizing code. It’s wrapped by the resampler service below, which in turn uses Resampler.js from the post “100% Client Side Image Resizing”.

var resampler = function ($q) {

    var resample = function (imageData, width, height, scope) {
        var deferred = $q.defer();

        Resample(imageData, width, height, function (result) {

            scope.$apply(function () {
                deferred.resolve(result);
            });
        });

        return deferred.promise;

    };

    return {
        resample: resample
    };
};

module.factory("resampler", resampler);

Getting Started With OWIN, Katana, and VS2013

Tuesday, July 9, 2013 by K. Scott Allen
5 comments

If you start almost any new ASP.NET project in the new preview of Visual Studio 2013, you’ll find a reference to the Owin package inside. OWIN (the Open Web Interface for .NET) is a specification designed to decouple web servers from the frameworks and applications they host. The OWIN goal is to provide a lightweight, modular, and portable platform for mixing and matching components, frameworks, and servers.

Katana is Microsoft’s implementation of OWIN components. The code is available on CodePlex

In a future post we can talk about how OWIN compares to System.Web, but first let’s get a simple example up and running from scratch.

In VS2013 you can start a new console mode application, then run the following commands in the package manager console:

install-package Microsoft.Owin.Hosting -IncludePreRelease

install-package Microsoft.Owin.Host.HttpListener –IncludePreRelease

install-package Microsoft.Owin.Diagnostics –IncludePreRelease

install-Package Owin.Extensions -IncludePrerelease

Inside the Main entry point for the console application, we can use the WebApp class to start an HTTP listener.

static void Main(string[] args)
{
    string uri = "http://localhost:8080/";

    using (WebApp.Start<Startup>(uri))
    {
        Console.WriteLine("Started");
        Console.ReadKey();
        Console.WriteLine("Stopping");
    }
}    

The Startup class used as the generic type parameter to WebApp.Start is a class we’ll have to implement, too.

public class Startup
{
    public void Configuration(IAppBuilder app)
    {
        app.UseWelcomePage();
    }
}

IAppBuilder is an interface we can use to compose the application for Katana to host. In this setup we’ll invoke the UseWelcomePage extension method provided by Microsoft.Owin.Diagnostics. Running the console program and pointing a browser to http://localhost:8080 produces:

katana up and running

Other extension methods allow for more fine grained control over the request and response processing. For example, UseHandlerAsync from Owin.Extensions allows for a more traditional “Hello, World” response to every request.

public class Startup
{
    public void Configuration(IAppBuilder app)
    {         
        app.UseHandlerAsync((req, res) =>
        {
            res.ContentType = "text/plain";
            return res.WriteAsync("Hello, World!");
        });
    }
}

So far this doesn’t appear much different than self-hosting WebAPI in a console application, but in some future posts we’ll dig a little deeper into some examples showing the loftier goals of OWIN and Katana.

by K. Scott Allen K.Scott Allen
My Pluralsight Courses
The Podcast!