The Lost Art of TSR Programming
Abstract: Return to the glory days of DOS 2.0 and INT 21h as we write a simple Terminate and Stay Resident application using the latest software development techniques. We will construct our x86 assembler code using test driven development and mock extended memory managers.
Why Am I Here On A Saturday?
Abstract: Because even if you weren't here, you'd still be at the computer. Don't think you'd be doing chores at home, like dusting off the entertainment center, because chores are boring.
Life of a Gnat
Abstract: This session has nothing to do with GNU software, but will describe (in excruciating detail) the journey of the common fungus gnat from egg to adulthood. Pictures of mating swarms may not be appropriate for younger attendees.
P.S. In all seriousness, the spring code camps are coming to the Mid-Atlantic and the topics are far better than the ones presented above.
CMAP Code Camp – April 12th in Columbia, MD
NoVa Code Camp South – March 29thth in Woodbridge, VA
Richmond Code Camp – April 26th
The visitor pattern is an elegant solution to a specific class of problems, but also comes with drawbacks in mainstream programming languages. There are various techniques one can us to minimize the drawbacks, however. This weekend I found some old but good articles harnessing the power of visitor while removing some of the pain:
Both Brad and Kzu use reflection to effectively achieve multiple dispatch. The typical multiple dispatch mechanism used to implement the visitor pattern is a drag on maintenance and forces the pattern implementation to bleed into the visitor's targets – Brad's post shows all the gory details before he goes on the show a solution using reflection. Multiple dispatch isn't just a problem in C#, C++, and Java – it's also a problem in dynamic languages like Python and Ruby. Only a handful of languages inherently use multiple dispatch, or multimethods, with Common Lisp being the most notable.
I tried a dispatch mechanism using Expression<T> today, but (in case you don't want to read any further) did not get a satisfactory result. If you keep reading, maybe you can think of something I missed.
Given these classes:
How can you write a class like the following?
The compiler will lay down code to invoke the Visit method that accepts an Employee reference, even when we really are referencing a Manager object at runtime. We need some mechanism that will thunk us into the best method based on the runtime type of the parameter, preferably without using a switch statement. A naïve try might look like the following:
Even though this code is generating IL at runtime via Compile, it still invokes the wrong method when we reach a Manager object. At compile time the compiler generates an expression tree to invoke the Visit accepting an Employee reference. We could go into the expression and make modifications to the Body before we call Compile, or we could build an expression tree manually. Either way, we need to pick out the exact method we need at runtime using the type of the parameter and a call to GetMethod:
This code works, because we are now getting a MethodInfo object that will reference Visit(Manager employee) when we have a manger object at runtime. But, the code is starting to look just as hairy as any other code generation technique, and we haven't even tried to address caching the compiled expression for performance, or moving up to an Expression<T>.
I spent the last week of February in Hyderabad, India. This was my first trip to India, and I thought I'd share some experiences.
I flew from Washington D.C. to Hyderabad on Qatar Airlines. The longest leg, between D.C. and Doha, was on a Boeing 777-300ER – a long range jet with the largest engines in aviation history. The business class configuration on the plane made the time pass with relatively little agony. The lay-flat seats gave me 12 hours of sleep, and I used the on-demand entertainment system to pass the time with some new movies (mostly on the way home):
I highly recommend Qatar Airlines. The business lounge in Doha is comfortable, and I had time to grab a shower in the marbled and well-stocked washrooms. Service is second none, and the food is the best I've ever had in the air – smoked salmon, brie, champagne, filet mignon, prawns, and Godiva chocolates. The endless stream of food and drink was served by a team of young models – all smiling and very easy on the eye.
The downside to this experience is that my next domestic flight will feel worse than ever. Satan himself recruits airline executives from the United States to mange the first circle of hell.
I just have to tell you about the Hyderabad airport experience. My flight, like most International flights to India, arrived in the middle of the night. You'd think an airport would be empty at 3 in the morning, but you'd be wrong. There was a 30 minute wait to get through immigration, which isn't too bad, but it was amusing. Lines would close and open arbitrarily, forcing people to shuffle and maneuver into new lines. There were things that happened in this shuffle that would have resulted in bloodshed had they happened in, say, Newark, N.J. But this was India, and personal space can't afford to exist.
At one point a new line opened, but the man behind the counter shouted "women only!" So I stayed in the long line of forlorn looking men and together we stared at this long line of women. For a brief moment I was hoping this would turn into one of the scenes I've watched in Bollywood movies. You know – the line of women start singing and dancing, and then the line of men start singing and dancing, and before you know it, the entire cast is embracing and gyrating out the door to freedom. I did some light stretching to warm up in case the music started, but nothing materialized.
The arrivals area outside was a sea of humanity. People were stacked 7 deep waiting for loved ones to arrive, or perhaps they were there just to watch the spectacle. I found my driver quickly, and waited just a few minutes for him to bring up the car. I listened to the jangle of excited voices. Watched the flashing and weaving of motorbikes. Felt a tug from a young lady begging. Smelled diesel fumes from rickshaw motors. Tasted dirt in the air.
India welcomed me with an assault on all senses.
Most of my time was spent in HITECH city, slightly northwest of Hyderabad proper. The Microsoft campus there is eerily similar to the campus in Redmond, right down to the color of the glass and signage. The only major difference is the unrelenting source of heat and light that hangs in the sky over the Hyderabad campus. Redmondites might have heard of it – it's called the sun. I was on campus everyday teaching for Pluralsight.
I did get a chance to venture into the city, and also to the nearby Golconda fort. The fort is built on a hill of granite and dates back to 1143. The outer wall has a circumference of 7km. Inside are iron studded gates, mosques, temples, ornate stonework, and a blend of Hindu and Muslim architecture. The gates also featured an impressive acoustical effect. A person can stand under the carefully designed dome and clap. That clap can be heard at the top of the mountain over one kilometer away! Hundreds of years ago, the claps would signal the king's arrival, or an enemy's encroachment.
Riding in the streets of Hyderabad I witnessed only a glimpse of the diversity that is in India. A women wrapped in a brightly colored sari stands alongside a Muslim women covered in a plain black jilbab. An old farmer with a handful of vegetables stands alongside a young businessman with his cell phone. It's a dizzying mix of life in the swarming streets of an old city. And yes, the driving in India is everything you've ever heard it is – and quite possibly worse than you ever imagined.
The road is lawless, but the people are polite. I'd say hospitality in India is second to none. Everyone I talked to was warm, inviting, and eager to share a story. It was entertaining to sit down at a meal and listen to conversations. Some talks would begin in English then suddenly veer into a local tongue. Another blink of the eye - and it's English again. Fortunately, someone was always willing to translate for me.
Finally, I'd be remiss if I didn't tell you about the food. I've developed a passion for Indian food over the last 5 years, and was delighted to find a mix of both northern and southern cuisines in this centrally located city. I had vadas, idlis and somber during breakfasts. Dosas and biryanis ruled for lunch, while paneer dishes (including my favorites palek paneer and mutter paneer), and a variety of curries and tandoori dishes were fair game for dinner. I was as close to achieving a gastronomic orgasm as one can get (although I'll be honest – by the end of the week I had this undeniable craving for a New York style pizza).
One week isn't a great deal of time to spend somewhere so far from home. I was just getting over the jetlag when I stayed up to catch a 5 a.m. flight home. In the airport, I was reading the local newspaper when I came across some funny ads:
Increase your bust line with a new herbal mix developed by scientists in the United States! Start looking more attractive with a bigger bust line in 3 months! Call now!
And:
Increase your height using a new lotion developed in the United States! Don't be short any longer – just apply this lotion to the soles of your feet once a day for six months! Order now!
Funny, because I think I've seen the same rip-offs advertised in U.S. newspapers, only the lotions are developed by wise monks who live in the misty mountains of India.
I guess it's true the world over – countries far away from your own carry an aura of mystique and magic. Anything is possible over there!
I was fortunate enough to see the mystique of India up close.
Some say we are living in the information age, but I say this is the advertisement age. Marketers strive to cover every square inch of the planet and outer atmosphere with slogans and promotions. We have ambient advertising and advergames, human billboards and celebrity branding.
My prediction is that marketers will become more aggressive in placing advertisements directly inside the software used by information workers who are laden with disposable income. Microsoft is well positioned for the next wave of advertising with the addition of extension methods in C# and Visual Basic. Just imagine the number of eyeballs that will see the following methods in an Intellisense window.
Big bucks are waiting there.
The only question is – will advertisers be content to stick with the traditional camel casing of method names, or will they pay a premium for Underscored_Product_Placements?
The HTML editor in VS2008 was killing me on pages using the fancier DevExpress controls. The editor would freeze after every keystroke. Responsiveness was so bad I started using a plain text editor to make changes to those pages.
Thankfully, a hotfix is now publically available. I tested an early version of the hotfix, and it made Visual Studio 2008 a happy place to work in again.
This wasn't a problem with the DevExpress controls, by the way. Their controls get better and better with each release. My recent favorite is the ASPxPivotGrid, which can slice and dice data from a relational database, or an OLAP database. Try the online demos for yourself.
Previous entries:
If you are not familiar with branching and merging, I'd suggest Eric Sink's excellent Source Control HOWTO. My preferred branching practice is "branch for release". New development work takes place in the mainline, or the trunk of the repository. As a product nears release, schema changes become rare and code churn slows. At this point, the team creates a branch for the release. New feature development resumes in the trunk with schema changes and all. The branch receives only bug fixes. The team generally merges fixes from the branch into the trunk.
How do schema changes work in this scenario?
In The Words Of Run-D.M.C.
It's tricky.
Schema changes in a branch require some care and thought. The technique I use of forward-only, run--once schema change scripts has some issues when it comes to branches. This is certainly an area you'll want to think about when devising your schema management process.
In my case, I bet that schema changes in a branch will be rare. Like most things in our field, it's all about the tradeoffs. If you use a more aggressive branching strategy (like branch per feature), this approach doesn't work so well. Other strategies (like Ruby Migrations) can move forward or backward with changes (as you don't make irreversible changes).
The Scenario
Let's say the team baselined the database at version 01.00.0000, and wrote 45 change scripts during feature development (01.00.0001 to 01.00.0045). Feature development is done, and the product is near a 1.0 release, so a 1.0 branch is created (note: the schema version and database version don't need to coincide at all).
At the point of branching, I create a new baseline for the database. In this case, a baseline of 01.01.0000 or 02.00.0000 is fine - I just want to rev either the major or minor version number. Let's say we use 02.00.0000 for the purpose of discussion. This new baseline goes into the trunk. All new database installs (from the trunk) just run this new baseline script, as it should produce the same database as the original baseline + the 45 change scripts. I also check in an empty 02.00.0000 schema change script so that all existing databases, when updating to a new build on this code, are now "version 2" databases.
Now, imagine the team working ahead and creating 2 schema change scripts in the trunk. These changes are 02.00.0001 and 02.00.0002. At this point a bug is found in the branch, and the bug requires a schema change to fix. Ugh!
Back in the branch, the team creates schema change 01.00.0046, and fixes the bug with a combination of code and change script. All is well for production type systems that are only receiving stable builds, because those databases have never seen a v2.0 change script. We simply update those systems with the new v1.0 build. The branch build includes and applies the 46th change script. All is well, at least in the world of 1.0.
What About the Mainline?
To get this fix into the mainline, there are two options. Well, actually there are an infinite number of options to consider depending on how you apply your updates, but here are two options:
With option #1 you have to be careful because any database that updated to v2.0 will not take the 46th change script from the branch (unless you write your tools differently than I do). You have to force people to run this script manually, or you go around destroying any existing v2.0 databases (which at this point. should only be on development and test machines anyway). This is not a great option, but if you are not deep into 2.0 it is sometimes viable.
Option #2 is a bit friendlier. The v1.0 databases will pick up the fix from 01.00.0046. The v2.0 databases will pick up the fix from 02.00.0003. You have to be careful though, to write the 02.00.0003 change script so that it won't try to reapply changes if the 01.00.0046 script ran.
In other words, databases installed from the v2.0 baseline script need to apply the 02.00.0003 script, but production type databases that have been around since 1.0 will use the 01.00.0046 script, and you don't want 02.00.0003 to create an error by making changes that are already in place when the database eventually updated to v2.0.
Whew, I hope all that makes sense. Branches and schema changes always make me nervous, but fortunately, they are rare. Even when they do occur, the change scripts usually involve simple changes and not the kind of big changes you see in a script written during feature development.
Questions
In a previous post, Jason asked:
#1: I generally only baseline between releases. This means new installs might have to run a heap of update scripts, but since they are new installs and the process is automated – it's not a big deal. The benefit is I don't have to keep baseline scripts synchronized, which is one less chore to perform when updating the schema. When creating a baseline, it's easy to use a SQL diff tool to make sure the baseline is identical to a database created by running the previous baseline plus all the update scripts.
#2: The change scripts many tools build often don't work for me because of the size of the databases I work with. Some of these scripts will run for hours and hours. Most of the change scripts are hand written with loving care. It's great to have a few developers around who are good with SQL – I know not every team has this luxury, and I honestly don't have a good answer to that problem.
I Think It's Over ... For Now
I've glossed over many details, but I think this is all I have to say on the matter of database versioning for now. Hopefully, these short blog posts made some sense.
The goal of versioning a database is to push out changes in a consistent, controlled manner and to build reproducible software. Since the change scripts are all version controlled and labeled with build numbers, you can create any past version of the database that you need.
My goal in writing these posts was to get people who don't version control their database to think about starting. If you already version control your database, I hope these posts have validated your thinking, or given you ideas. It could be that you already have a much better process in place, in which case you should try to blog about your process, because there is a dearth of information on the Internet about this topic. Better yet – write an article or a book, but hurry up, we need to hear from you!
What started as a short brain dump is tuning in to a longer series of posts thanks to all the feedback and questions. In this post, I want to explain some of my thoughts on controlling objects like database views, stored procedures, functions, and triggers.
But first...
I haven't actually used a trigger in years. This isn't to say that triggers aren't valuable, but I've tended to shy away. Jon Galloway has posted a good example of what you can do with triggers.
Secondly, stored procedures have fallen out of favor in my eyes. I came from the WinDNA school of indoctrination that said stored procedures should be used all the time. Today, I see stored procedures as an API layer for the database. This is good if you need an API layer at the database level, but I see lots of applications incurring the overhead of creating and maintaining an extra API layer they don't need. In those applications stored procedures are more of a burden than a benefit.
One File per Object
My strategy is to script every view, stored procedure, and function into a separate file, then commit the files to source control. If someone needs to add a new view, they script the view into a file and commit the file to source control. If someone needs to modify a view, they modify the view's script file and commit again. If you need to delete a view from the database, delete the file from source control. It's a relatively simple workflow.
The magic happens when a developer, tester, or installer updates from source control and runs a tool that updates their local database. The tool uses a three step process:
Say What?
People often wonder why I want to drop and destroy all these objects and then re-add them. You could lose work if you update your database and haven't scripted out a view you were in the middle of creating.
The simple reason is to find problems as early as possible. If someone commits a schema change and the change removes a column used by a view, you'll find out there is an error early – hopefully before the build escapes for testing. Likewise, if someone checks in a view but forgets to publish a schema change the view needs, someone else is going to show up at their desk a few minutes later asking why they are breaking the software.
A second reason is to avoid some rare errors I've seen. Some databases have a tendency to screw up execution plans when the schema changes underneath a view. Dropping everything and starting over avoids this problem, which is really hard to track down.
Of course, this does mean a database will need a little down time to apply updates. I realize not everybody has that luxury.
Summary
Through the power of source control and automation, every database from development to production can keep a schema in synch. If you need to go back in time to see what the database looked like on July 20, 2007, or in build 1.58 of the application, you can do that with this strategy, too.
Next question: what happens if there is a branch in the source control tree? That's a topic for the next post.
OdeToCode by K. Scott Allen
Subscribe
Twitter
Search
About
Learn C#