This is the silly tale of a strange due diligence process I experienced. It happened several years ago but I couldn’t talk about it at the time.
I’ve been on both sides of the technical due diligence table, and I’ve always felt that being reviewed is easy – you simply tell the truth when asked about the software and how it’s built. Anything else can get you into trouble. It is the reviewer who has the difficult job. Someone wants to buy or invest in a company and they are depending on you to help determine the fair value. You have to look at the development methodology, the design, the architecture, the configuration management practices, the licenses, and basically ferret out any problems and risks to ensure the company is a good investment.
A few years ago I was on the receiving end of the review. The first thing the guy wanted to do was open up a source code file. I pick one and open it on the screen. The conversation goes like this:
Him: I don’t see any comments in the code.
Me: Nope!
Him: Ok, let’s take a look at this file. Hmm, I don’t see any comments in here, either!
Me: Umm .. no .. no comments.
Him: Ok, let’s take a look at this file here. Well, well, well. It doesn’t look like you guys write any comments at all!
Me: Honestly - we try to avoid commenting code.
Him (horrified): What? How on earth does anyone know what the code is doing?
Me (stunned): Well, we figure out what the code is doing .. by .. um .. reading .. the .. code.
There are people who believe that all code should be accompanied by comments (an undergrad professor come to mind), but by the turn of the century I had hoped that most people understood comments to only be used as a “here be dragons” sign. If you are depending on comments to make code consumable - you’ve already lost the battle. Sure, there are exceptions for exceptional code – like code written to work around platform bugs, code optimized for speed, and the comments that feed javadoc and intellisense for a public API, but for the most part code that requires comments to make itself maintainable is bad code. This isn’t a recent trend. The Elements Of Programming Style said in 1978:
Don’t comment bad code – rewrite it.
Eventually the business deal fell through, but it wasn’t due to the lack of comments. The company walked away from the money the investors were willing to offer. A due diligence process, like a job interview, works in both directions. In this case the competency of the due diligence team didn’t instill the confidence needed to sell them a piece of ownership.
ASP.NET AJAX 4.0 is adding client-side templates. You can bind data in the browser using declarative markup, imperative JavaScript code, or mix and match both approaches. The purely declarative approach looks like the following:
<body xmlns:sys="java script:Sys"
xmlns:dataview="java script:Sys.UI.DataView"
sys:activate="*">
…
<ul id="departmentManagerList" class="sys-template" sys:attach="dataview" dataview:serviceuri="DepartmentService.svc" dataview:query="GetAllDepartments"> <li>{{ Manager.Name }}</li> </ul>
That code is everything you need to invoke an AJAX-enabled WCF service (specifically the GetAllDepartments method of DepartmentService.svc), and bind the results to a list. Some of the binding markup looks foreign inside the HTML, and when I read it a little voice in my head says: “JavaScript should be unobtrusive”. I reply and say “that is not JavaScript”, and the voice says “but it’s still ugly”. I have to agree, but there is an alternative we’ll look at in just a moment.
Anyone who has used XAML binding extensions for Silverlight or WPF will notice a number of similarities in the AJAX 4 bits. For example, both use curly braces inside the item template to denote the path to the source property for binding (in the above code we’re drilling into a department object and pulling out a manager’s name property). The AJAX libraries will take care of replicating the data inside of <li> tags. Also, there are similar binding modes (one way, two way, and one time), and the concept of an IValueConverter for each binding (ConvertFrom and ConvertBack functions can inject custom logic at bind time).
The AJAX 4 libraries support hierarchical data binding (for master detail scenarios) and can update a binding when the underlying data changes. However, this is one place where AJAX and XAML land differ. In XAML land we use INotify* interfaces to raise change events, whereas AJAX provides a Sys.Observable class that can “mixin” functions on a target object that allow you to ultimtaely catch property change notifications, but you must use Sys.Observable.setValue to see the notifications. Infinities Loop has the scoop on this behavior.
Instead of purely declarative code, one can instantiate a DataView and write some JavaScript code to grab and process data. Since there doesn’t appear to be any pre or post request events we can hook from the DataView (which would be nice), this also gives us a chance to manipulate data from a web service before binding. The HTML will look cleaner:
<body> … <ul id="departmentManagerList" class="sys-template"> <li>{{ Manager.Name }}</li> </ul>
And we need to write a bit of code during load:
var view = $create(Sys.UI.DataView, {}, {}, {}, $get("departmentManagerList")); var service = new DepartmentService(); service.GetAllDepartments(function(result) { view.set_data(filterAndSortDepartments(result)); });
What is filterAndSortDepartments? It’s a function that uses JSINQ – LINQ to objects for JavaScript.This isn't part of ASP.NET AJAX 4.0, but a CodePlex hosted project.
function filterAndSortDepartments(departments) { return new jsinq.Enumerable(departments) .where(function(department) { return department.Employees.length > 1; }).orderBy(function(department) { return department.Name; }).toArray(); }
JSINQ has complete coverage of all the standard LINQ query operators, including join, group, groupJoin, any, all, and aggregate. The above code was written with the “extension method” syntax style, and rather makes me wish that JavaScript could take the lead from C# 3.0 lambda syntax and allow function definitions without keywords. JSINQ also offers comprehension query syntax:
function filterAndSortDepartments(departments) { var query = new jsinq.Query('\ from department in $0 \ where department.Employees.length > 1 \ orderby department.Name \ select department \ '); query.setValue(0, new jsinq.Enumerable(departments)); return query.execute().toArray(); }
In this case the declarative code is easier on the eyes, but like most declarative code it’s impossible to debug if something goes wrong. There are lot’s of possibilities when using the combination of these two libraries. Give JSINQ a try in the JSINQ playground.
This version of WWWTC is dedicated to closures. We can use closures and functional programming in general to create elegant, succinct solutions, but sometimes we can also create subtle bugs.
Exhibit A. The C# developer who wrote this code expected DoSomething to be invoked with the values 0 through 9, but something is terribly wrong.
for (int counter = 0; counter < 10; counter++) { ThreadPool.QueueUserWorkItem( state => DoSomething(counter) ); }
What’s the problem?
A JavaScript developer is working with the following markup.
<div class="header"> <div class="detail">Detail 1 …</div> </div> <div class="header"> <div class="detail">Detail 2 …</div> </div> <!-- and so on-->
When the user clicks inside a detail section, the developer wants to change the background color of all the other header sections. Thinking of one way to achieve this behavior, the developer first tests the jQuery not method.
$(function() { $(".header").each(function() { alert($(".header").not(this).length); return false; }); });
This code correctly alerts the user once and displays the total number of headers – 1.Emboldened with this proof of concept, the developer puts in the required CSS manipulation and embeds the code into a click event handler.
$(function() { $(".header").each(function() { $(".detail", this).click(function() { $(".header").not(this).css("background-color", "red"); }); }); });
Suddenly, things aren’t working and every header section turns red. What went wrong?
In my role as amateur financial analyst I like to read the publications taking an in depth look at the current financial turmoil. I then try to imagine the software tool that could have prevented the turmoil and saved the world millions billions trillions gazillions of dollars. This seems like no small feat for software considering the intricate world built by modern financial engineering - a world where terms like “collateralized debt obligation” are tossed around like dice at a craps table. However, I never imagined the answer could be so simple …
First, some background. I find the current situation interesting because ten years ago I joined an Internet startup by the name of Ultraprise.com. At Ultraprise, we tried to create an online exchange for the secondary mortgage market. Let’s say you have a bit of money saved and want to buy a house, so you find a bank that will loan you the rest of the money you need to buy the house.The bank could then wait for your to repay your loan over the next 30 years and slowly make money on the interest they charge you, but many banks just want to sell your loan to someone else and receive cash in return. Wikipedia has solid coverage of this process in the Securitization entry. The fresh cash received from the sale of your loan allows the bank to stay liquid and fund more loans. But, to sell the loans the bank needs to find a buyer, and this is where Ultraprise came in.
Fannie and Freddie pick up roughly half of the mortgages in the U.S – that is their job, but the rest of the pie is still a substantial amount of debt for sale. At Ultraprise we built a web application for banks to upload pools of loans they wanted to sell. Buyers could then login and bid on the pools. The idea was to cut out the brokers who typically sit in the middle of these transactions and take a cut. Economies of scale meant our fees would be substantially less than the typical fees.
Ultraprise was out of business after three years, despite having a substantial number of loans posted in the system. We couldn’t sell loans. Part of the problem was, I think, that the people who buy loans really like the human touch that a broker adds. Rib-eye steaks and martinis sell more products than a dull web page full of numbers.
Another problem was getting mortgage data in and out of the system. There was no standard format for pushing mortgage data. Every bank with a loan for sale required a custom import, and every bank looking to buy loans wanted to download data into their custom software for analysis and due diligence.
My experience at Ultraprise led me to believe that banks were exceedingly risk averse and highly analytical, and that they required lots of data before making decisions involving millions of dollars. Thus, I was quite surprised to learn that today’s banks have no clue about the cards they are holding.
“ … a major stumbling block for banks is having the right data on hand …” - American Banker
“…the reason that banks don’t want to lend to each other anymore is that they don’t trust that the other banks really know the value of their mortgage-backed securities … because they themselves don’t trust the value of their own.” – Aster Data
All this secrecy comes in the wake of an economic crisis brought about in part by a proliferation of financial instruments so opaque that virtually no one understood the risks. – USA Today
There have been documented changes in the mortgage industry since Ultraprise closed. For example - the lowering of lending standards allowed banks to give more money to people with lower credit scores. However, the outstanding change from my perspective was how those loans were sold to investors. What follows is an extremely gross simplification.
As subprime lending increased, the banks found it harder to sell pools of loans. The risks were too high for the big-money conservative investors like pension funds, and the possible rewards were too little for the hedge funds addicted to double digit growth. Thus, the rise of the aforementioned collateralized debt obligation, a.k.a the CDO.
The investment banks use CDOs to package mortgages, subprime loans, home equity loans, automobile loans, credit card debt, corporate debt, and used cat litter into products they slice up and sell to investors in private offerings (with the help of strippers, martinis, and appalling judgments by our trusted credit rating agencies).
However, not all of the slices from a CDO are an easy sell (particularly the ones filled with risky loans and cat liter). So … firms will create a CDO squared (CDO^2) from the undesirable slices of multiple CDOs – essentially dressing up pigs with lipstick for the next big investors ball. You can be assured that a CDO cubed (CDO^4) will then arise from the dumping grounds of multiple CDO^2s, and then … well … CDO^N should give you an appreciation of how deep into the rabbit hole an investor can fall.
Investment banks issued hundreds of billions of dollars in CDOs over the last 5 years. Why did the risk averse investors, like the banks and the pension funds, stand in line to buy these CDOs and the other credit derivatives that Warren Buffet labeled “financial weapons of mass destruction”?
In part because a CDO does a good job of obscuring it’s underlying qualities– the loans, the assets, but most of all the risk. It was all a sales job, and something that an unknown software company could never pull off.
The $765 million “Mantoloking CDO 2006-1” was underwritten by Merrill Lynch and is the prime example of a “toxic asset”. The Mantoloking, a CDO squared, was built from the unwanted slices of 126 other CDOs, and is perhaps the most infamous CDO because its spectacular losses facilitated the disappearance of at least two hedge funds. You can find the 200 page prospectus online. It’s a lot to digest. In his paper “The Credit Crunch of 2007: What went wrong? Why? What lessons can be learned?”, esteemed financial engineer John C. Hull says transparency is needed.
[CDOs] … are arguably the most complex credit derivatives that are traded. Lawyers should move with the times and define these instruments using software rather than words.
Sound familiar? To me it sounds like the software practice of using executable specifications - TDD and BDD for the financial world. No one can hide behind wordy documents and inflated credit ratings. They have to look at real numbers. The only question is - do we write these specifications in C#? Ruby? Haskell?
It turns out that Mr. Hull already had a language in mind. You can find it in his paper as footnote #8.
Given its widespread use in the financial community VBA is a natural choice for the programming language.
What? A Microsoft Excel spreadsheet with macros might have saved our banks, our brokerage accounts, and our retirement funds?
It’s a stretch, but with the proper models in place it’s certainly a step in the right direction. Mr. Hull’s paper has other prescriptions for the finance world, too, as software is only part of the solution. You can never stop anyone who wants to skip due diligence and go directly to short-sighted greed, but I’d like to think that if our industry could make good software more readily available to the business world, the problems we are experiencing today wouldn’t be quite so bad.
ASP.NET 3.5 includes a URL routing engine and IIS 7.0 can use a URL re-writing engine (x86) (x64). Routing and rewriting sound very similar, but so do robbing and rewarding – you can’t judge features using phonetics.
Ruslan Yakushev wrote a great article on this very topic last year: “IIS URL Rewriting and ASP.NET routing”. Ruslan outlines the conceptual differences between the two URL hacking approaches as:
Read the full article and I’ll think you’ll find that routing and rewriting turn out to be complementary pieces. For example, you might use rewriting for request blocking and enforcement of canonical URLs, while at the same time using ASP.NET routing for friendly URLs in your application. One advantage of using the ASP.NET routing engine in that scenario is that you can use all the routing rules to not only route requests, but also generate URLs when you put links on a page. The rewriting rules are opaque to an application.
The “Principle of Least Astonishment” (a.k.a the principle of least surprise (POLS)) is a guiding design principal for UIs, APIs, and interfaces of all types. Peter Seebach says:
Throughout the history of engineering, one usability principle seems to me to have risen high above all others. It's called the Principle of Least Astonishment -- the assertion that the most usable system is the one that least often leaves users astonished.
POLS plays a role in framework design, too. One framework behavior that I’ve always felt violated POLS was the ToString implementation of the System.Xml.XmlDocument class.
XmlDocument document = CreateSomeXml(); string result = document.ToString();
You’d think that calling ToString on an XmlDocument instance would give you a hunk of XML. But - it doesn’t matter how many nodes you have in the document, the ToString method will happily produce the following string every single time:
System.Xml.XmlDocument
This string represents the fully qualified type name of the object. Thanks, .NET framework! I wasn’t sure I was working with an XmlDocument. Now tell me, how do I get the XML out of this thing?
The new XML API in .NET revolves around XElement and friends in the System.Xml.Linq namespace. This API fixes the astonishment factor of the ToString method.
XDocument document = CreateSomeXml(); string result = document.ToString();
That above code yields real XML – angle brackets and all. In fact, LINQ to XML tends to “do the right thing” in many scenarios. For example, extracting an integer out of an attribute is just one cast operator away:
XDocument document = CreateSomeXml(); int i = (int)document.Root.Attribute("bar");
Now that we have all the surprises out of the way, we can all get back to work and make the world better.
Wait? What’s that?
Of course there are a plethora of options you might want to specify when producing a textual representation of an XDocument. There are encoding options, indentation options, new line options – the list goes on and on. You can get to these options with the WriteTo method of an XDocument, but not with ToString. ToString has to pick some defaults. One of the defaults in the ToString implementation is to omit any XML declaration, and this causes some people grief.
Just recently I came across a post in a Ruby form. The topic was a debate over the naming of mathematical functions, and Matz says:
Ah, please stop mentioning POLS whenever your personal expectation is
not satisfied. I'm sick of them.
Trying to follow the principle of least surprise is one thing, but never surprising anyone is impossible.
What has surprised you recently?
Jeffrey “Party With” Palermo recently posted on “Separating configuration from data lowers total cost of ownership”:
My recommended progressions of configuration locations are as follows. Specific requirements should be the driving factor that cause you to move configuration from one medium to the next costly alternative.
- In the code
- In a file
- In a database
- Some other system-specific external configuration store
I wanted to call out option #3, which is to put configuration information in the database. Over the years, I’ve heard numerous proposals for putting configuration information in the database. Nearly every proposal has a good counterargument.
Putting configuration information in the database is easier.
Easier than checking in code? If so, then you might have a problem with your source code repository. More often than not, “easier” means someone is trying to circumvent testing, check-in policies, revision control, or code reviews.These are all things you shouldn't avoid.
Our configuration information is very complicated.
Complexity isn’t something a database can fix. The configuration will still be complicated, and to make matters worse - it’s now in the database!
We need to access configuration values from everywhere.
If this is a highly distributed system, then this might be a good argument, but sometimes “everywhere” means “the business logic inside of stored procedures”. There is no reason to make it easier for business logic to sneak into the database.
The database should be one of your last choices as a repository for configuration information. Configuration in code is easily tested, versioned, changed, and refactored. As Jeff said – only a specific requirement should drive you to a more costly alternative like the database.