OdeToCode IC Logo

The Law of the JSON

Monday, February 4, 2019 by K. Scott Allen

From Wikipedia: The law of the instrument is a cognitive bias that involves an over-reliance on a familiar tool.

The software industry is good at adopting standard data formats, then hyping and ultimately abusing those formats by pushing them into every conceivable nook and cranny of the trade.

Take XML, for example. What started as a simple, usable, textual data format became the lingua franca for enterprise web services. Along the way, we also used XML to build configuration files, replace make files, and implement more than a dozen significant UI frameworks.

Ten years ago, I was working with Microsoft’s triumvirate crown jewels of XML technology – WCF, WPF, and WF (Windows Workflow). Workflow is a good example of how XML transformed itself into a bloviating monster of complexity.

Example A: How to represent the integer value 5,000 in WF markup:

<ns0:CodePrimitiveExpression>
    <ns0:CodePrimitiveExpression.Value>
    <ns1:Int32 xmlns:ns1="clr-namespace:System;Assembly=mscorlib, 
                  Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
           5000
       </ns1:Int32>
    </ns0:CodePrimitiveExpression.Value>
</ns0:CodePrimitiveExpression>

Use XML, they said. It’s human readable, they said.

The Curse of Ubiquity

A few weeks ago, I became interested in cloud governance. Not an exciting topic, but when a company turns developers loose in the cloud, someone must enforce some basic rules about what services to use, and ensure the settings are in place to make the services as secure as possible. For Azure, I looked at using Azure Policy. It was obvious that I’d need to write some custom rules for Policy. To see how rule authoring works, I looked at a built-in rule that checks if data encryption is on for an Azure SQL instance:

{
  "if": {
    "allOf": [
      {
        "field": "type",
        "equals": "Microsoft.Sql/servers/databases"
      },
      {
        "field": "name",
        "notEquals": "master"
      }
    ]
  },
  "then": {
    "effect": "[parameters('effect')]",
    "details": {
      "type": "Microsoft.Sql/servers/databases/transparentDataEncryption",
      "name": "current",
      "existenceCondition": {
        "allOf": [
          {
            "field": "Microsoft.Sql/transparentDataEncryption.status",
            "equals": "enabled"
          }
        ]
      }
    }
  }
}

So, in the last 10 years, programming has evolved from writing syntax trees in XML to writing syntax trees in JSON. Actually, software has evolved from doing everything in XML to doing everything in JSON - web services, configuration files, build systems, database storage, and more. JSON’s ubiquity is surprising given that JSON doesn’t have any of the same flexibility and extensibility features as XML.

Or, as @kellabyte recently said:

Yes, I know. I lived through the AJAX days when everyone said JSON was the future. Reason being that browsers and mobile devices didn't come with XML parsers by default.

Web browsers and mobile devices today carry AI for facial recognition, support headsets for augmented reality, stream 4K video, and execute GPU accelerated 3D animations.

But parse XML documents? Nobody wants to walk in that minefield.

JSON is the easy choice.

The Easy Choice

I can understand the argument for easy. I imagine trying to design Azure Policy and thinking about implementation details for custom rules.

Can we force everyone to use Powershell? No!

JavaScript? How to sandbox the execution?

Wait, I got it – describe the code in JSON! Everyone knows JSON. All the tools support JSON. The answer must be JSON.

I’ve been down a similar road before. Years ago, I needed to create a custom ETL tool to move data between relational databases as fast as possible. At the time, Microsoft’s offering from SQL Server was SSIS 1 (SQL Server Integration Services). I spent a couple of days with SSIS and decided it was not an appropriate choice for our specific scenario. The XML was complex, hard to version, hard to test, hard to debug, slow to execute, yielded unhelpful error messages, and made a team wholly dependent on UI tools that ship with SQL Server. Not to mention, SSIS wouldn’t easily scale to meet the thousands of packages and package variations we needed to support. I had been down that road before with Microsoft’s previous ETL tools (DTS – data transformation services) and vowed never again.

Once I decided to build my own ETL tool, I needed to decide on the language for the tool. My first attempt, which survived in the wild for a brief time, relied on Ruby and a fluent API. The second attempt tried to simplify things. I needed SQL, but a way to surround the SQL with metadata.

Why not use XML? Everyone knows XML, and all the tools support XML. XML is easy!

The result used "packages" that looked something like the following 2:

<Package Type="Arracher.Core.Packages.BulkLoadPackage">
  <Name>Load Movies</Name>
  <Source>LiveDb</Source>
  <Destination>RapidDb</Destination>
  <DestinationTable>tbl_movies</DestinationTable>  
  <Query>
    <![CDATA[        
    DECLARE @StartDate datetime
    SET @StartDate = '@{StartDate}'
               
    SELECT CONVERT(varchar(1),NULL) As title, release_date  as releasedon          
    FROM dbo.movies WHERE release_date > @StartDate          
    ]]>
  </Query>
</Package>

Is it the best tool in the world? No.

But, packages are easy to version, easy to diff, easy to edit, author, and test, and best of all – a dba can open the file, copy out the SQL, tweak the SQL, and paste the SQL back in without any special editing tools and a minimal amount of fuss. The SQL can be as complicated as SQL can be.

The tool works, I believe, because the XML doesn’t get in the way. That’s the problem you can run into when you embed one language in another – one of the languages will get in the way.

Embedding

Azure Policy tries to embed Boolean rules inside of JSON, but to me, the JSON only gets in the way. It’s like embedding SQL code inside of C# or Java code – some language combinations are hard on the eyes, which makes the result hard to write, and impossible to read. You can’t just skim the code to get an idea of what might happen.

With policy, the more complex the expression, the more unreadable the code. The solution is error prone, hard to test, and therefore not scalable.

Here’s @kellabyte again, this time on embedding in JSON:

This is one of the reasons why LINQ is so compelling in C#. There’s no embedding – I have C# code inside of C# code, but some of the C# code might just translate itself into a different language and execute in a different process on some distant machine.

Despite what you might think of ORMs, or the hidden performance costs of LINQ, the feature still amazes me every time I see a query. I had the same sense of excitement the first time I ran across Gremlin-Python, too.

Summary

Truthfully, I wrote this post to organize my thoughts around Azure Policy.

Do I want to take a pass on what comes in the box with Azure? I think so.

Can I rationalize a custom solution? I think so.

I can invent my own tool for governance audits and make the rules easier to author, change, and test, as well as be more flexible.

And just for fun, I’ll write the tool in Go.

I’ll rationalize that decision in another post.

[1] The current offering for ETL (keep in mind ETL is a legacy term for a legacy form of processing) from Microsoft is Azure Data Factory (ADF). We author ADF packages in JSON. ADF v2 supports calling into SSIS XML packages. This is like candy built from data formatting standards - a hard JSON shell surrounds a creamy-sweet XML interior.

[2] I named the tool Arracher – French for “rip out”, or extract.

Composition over Inheritance in Go

Thursday, January 3, 2019 by K. Scott Allen

I’ve always admired languages that make composition as easy as inheritance. Groovy, with its @Delegate, is one example from years ago. These days I’ve been working a bit with Go. In Go, the composition features are so good you can achieve everything you need using compositional thinking.

We’ll use waterfowl as a silly example. Here’s a duck in Go.

type Duck struct {
    ID   int64
    Name string
}

I can also define a method on the Duck struct.

func (d *Duck) Eat() {
    fmt.Printf("Duck %s eats!\n", d.Name)
}

Later I might need a Mallard. A Mallard is very much like a Duck, but with a Mallard I need to track the color, too. Go doesn’t have inheritance, but embedding is elegant.

type Mallard struct {
    Duck
    Color string
}

Given a Mallard, I can reach into the ID and Name properties, or refer directly to the embedded Duck.

duck := new(Duck)
duck.ID = 1
duck.Name = "Pickles"

mallard := new(Mallard)
mallard.Color = "Green"

// copy info:
mallard.Name = duck.Name
mallard.ID = duck.ID
     
// or even better: 
mallard.Duck = *duck

And yes, I can define a method for Mallards.

func (m *Mallard) Sleep() {
    fmt.Printf("Mallard %s sleeps!\n", m.Name)
}

A mallard can now both eat and sleep.

    mallard.Eat()
    mallard.Sleep()

// Duck Pickles eats!
// Mallard Pickles sleeps!

The end result looks like inheritance, if you wear object-oriented glasses, but the mindset is entirely composition.

Experimenting with ASP.NET Core Authentication Schemes

Wednesday, January 2, 2019 by K. Scott Allen

Some software is easier to understand if you remove the software from it’s usual environment and try some experiments. ASP.NET Security components, for example. What is the impact of having multiple authentication schemes? Why does a ClaimsPrincipal have multiple identities? What does it mean to SignOutAsync on an HttpContext?

You’ll never use the following code in a real application. But, you might use this code to tinker and experiment.

First, we’ll setup two cookie authentication schemes during ConfigureServices – cookie1 and cookie2.

services.AddAuthentication(options =>
{
    options.DefaultScheme = "cookie1";
})
.AddCookie("cookie1", "cookie1", options =>
{
    options.Cookie.Name = "cookie1";
    options.LoginPath = "/loginc1";
 })
.AddCookie("cookie2", "cookie2", options =>
{
    options.Cookie.Name = "cookie2";
    options.LoginPath = "/loginc2";
});

Next, we’ll add some middleware that allows for identity sign-in and sign-out without getting bogged down in password validations.

app.Use(next =>
{
    return async ctx =>
    {
        switch(ctx.Request.Path)
        {
            case "/loginc1":
                var identity1 = new ClaimsIdentity("cookie1");
                identity1.AddClaim(new Claim("name", "Alice-c1"));
                await ctx.SignInAsync("cookie1", new ClaimsPrincipal(identity1));
                break;
            case "/loginc2":
                var identity2 = new ClaimsIdentity("cookie2");
                identity2.AddClaim(new Claim("name", "Alice-c2"));
                await ctx.SignInAsync("cookie2", new ClaimsPrincipal(identity2));
                break;
            case "/logoutc1":
                await ctx.SignOutAsync("cookie1");
                break;
            case "/logoutc2":
                await ctx.SignOutAsync("cookie2");
                break;
            default:
                await next(ctx);
                break;
        }
    };
});

app.UseAuthentication();

Now it’s time for the experiments. What happens when trying to reach pages or controllers with the following attributes?

  • [Authorize]
  • [Authorize(AuthenticationSchemes ="cookie1")]
  • [Authorize(AuthenticationSchemes ="cookie2")]
  • [Authorize(AuthenticationSchemes ="cookie1, cookie2")]

When visiting those resources, it’s educational to dump out what we know about the user given the authorize conditions, and how the output changes if we change the default auth scheme.

<h2>User</h2>
@foreach (var identity in User.Identities)
{
    <div>Authentication Type: @identity.AuthenticationType</div>
    <table class="table">
        @foreach (var claim in identity.Claims)
        {
            <tr>
                <td>@claim.Type</td>
                <td>@claim.Value</td>
            </tr>
        }
    </table>
}

I’ve also found it useful, even in real applications, to have a page that dumps out information about the available authentication schemes. Quite often the setup is obscured by helpful extension methods we use inside of ConfigureServices. A page model like the following will grab the information.

public class AuthDumpModel : PageModel
{
    private readonly AuthenticationService authenticationService;

    public AuthDumpModel(IAuthenticationService authenticationService)
    {
        this.authenticationService = (AuthenticationService)authenticationService;
    }

    public IEnumerable<AuthenticationScheme> Schemes { get; set; }
    public AuthenticationScheme DefaultAuthenticate { get; set; }
    public AuthenticationScheme DefaultChallenge { get; set; }
    public AuthenticationScheme DefaultForbid { get; set; }
    public AuthenticationScheme DefaultSignIn { get; set; }
    public AuthenticationScheme DefaultSignOut { get; set; }

    public async Task OnGet()
    {
        Schemes = await authenticationService.Schemes.GetAllSchemesAsync();
        DefaultAuthenticate = await authenticationService.Schemes.GetDefaultAuthenticateSchemeAsync();
        DefaultChallenge = await authenticationService.Schemes.GetDefaultChallengeSchemeAsync();
        DefaultForbid = await authenticationService.Schemes.GetDefaultForbidSchemeAsync();
        DefaultSignIn = await authenticationService.Schemes.GetDefaultSignInSchemeAsync();
        DefaultSignOut = await authenticationService.Schemes.GetDefaultSignOutSchemeAsync();
    }
}

And now we can see what’s installed, and where the defaults lead.

<h2>Auth Schemes</h2>

<table class="table">
    <tr>
        <th>DisplayName</th>
        <th>Name</th>
        <th>Type</th>
    </tr>
    @foreach (var scheme in Model.Schemes)
    {
        <tr>
            <td>@scheme.DisplayName</td>
            <td>@scheme.Name</td>
            <td>@scheme.HandlerType</td>
        </tr>
    }
</table>
<div>DefaultAuthenticate : @Model.DefaultAuthenticate.Name</div>
<div>DefaultForbid: @Model.DefaultForbid.Name</div>
<div>DefaultSignIn: @Model.DefaultSignIn.Name</div>
<div>DefaultSignOut: @Model.DefaultSignOut.Name</div>

REST Client is the Best Client

Thursday, November 8, 2018 by K. Scott Allen

REST Client is a Visual Studio Code extension for sending HTTP requests and viewing responses from the productive confines of Code itself.

REST Client Extension for VS Code

There is no shortage of tools for building HTTP messages, but many of these tools have become unnecessarily complex. I have to select the HTTP method from a tiny drop-down list, enter a header by typing the key and value into two separate inputs, and toggle open a third input to enter the body content. These tools try too hard.

In contrast, REST Client lets me type into an interface designed for creating and editing text – a text editor. I can check-in my experiment to source control, and use standard editor features like search and replace.

On top of all this simplicity, add in REST Client features like variables and the ability to keep multiple requests in the same file and you have a tool that is easy to use and powerful.

Give it a try!

Tracking Down a Mysterious 502.5 Error in ASP.NET Core

Monday, November 5, 2018 by K. Scott Allen

Azure Application Insights ExtensionWhen I posted 7 Tips for Troubleshooting ASP.NET Core Startup Errors, I thought I had covered every tip needed to track down any possible startup error. In particular, the last step never fails to reveal the culprit.

Then, the last step failed me.

This particular application would not run in an Azure App Service but would run everywhere else - in development, in debug, in release, and in the command line console on the App Service server. How’s that possible?

Soup du Jour

There is a proverb that says "too many cooks will spoil the broth", meaning when too many people are trying to add their own expertise on the same project, the result will be a disaster. Such is currently the case when two chefs, Visual Studio and the Azure portal, both try to cook a pot of Application Insights.

Application Insights is an invaluable resource for monitoring and troubleshooting an application, but AppInsights has also been a bit flaky to setup with ASP.NET Core in the past. Sometimes AI doesn’t work no matter how hard you try, and sometimes AI does work when you try to make it stop.

Despite the flakiness, AppInsights works great once everything is configured. It is no surprise that both the Azure Portal and Visual Studio encourage the use of AppInsights, but this leads to common questions about the best approach to use AppInsights.

Am I supposed to install the AppInsights NuGet to my project? This "build time" configuration allows an app to use the entire SDK and provide custom telemetry to AppInsights.

Or, am I supposed to setup AppInsights as an Azure App Service extension? This is known as "run-time" configuration and doesn’t require any code changes or new deployments. The official doc uses wording to encourage the "build-time" approach, but you can also find a page that says to use both approaches for the best experience.

The failing application took the "both" approach, and it turns out the "both" approach was the source of the error.

Finding the Failure

When the application failed with an HTTP 502.5 error, I went to Kudu and ran the app from the console. The application would work from the console, but consistently fail when launched from the worker process in a Windows App Service. This behavior was curious, because both approaches run the same app on the same virtual server with the same file system and the same environment variables. But, decades of debugging experience gave me the insight that something about the environment must be different.

Ultimately I found the true source of the failure by looking at the live stream of the app’s standard output stream.

ASP.NET Core live stream standard output in Azure App Service

Even this error was a bit curious since the dotnet publish operation included the required AppInsights assembly in the output. I dug in further and eventually looked at Kudu diagnostics for the w3wp.exe worker process. That’s when the answer jumped out at me.

Azure App Service Worker Process Environment Variables

ASP.NET 2.0 introduced the ASPNETCORE_HOSTINGSTARTUPASSEMBLIES environment variable seen above. The variable allows a host to inject additional startup logic to a compiled application by naming assemblies with IHostingStartup components inside. I had no idea what StartupBootstrapper could be, but a quick search revealed this assembly to be part of the Azure AppInsights extension. In the end, the AppInsights installed by Visual Studio and the AppInsights installed by the Azure extension were incompatible versions. To get the app working, I could do any of the following:

  • Uninstall the AppInsights extension in the App Service
  • Remove the AppInsights package reference from the project
  • Leave both in place, but make sure the package version and extension version are compatible

Option #1 seems like the best option, since one might never know when the extension will update and break the application again.

Summary

If you think you know exactly what your ASP.NET Core application does during Startup, you could be wrong. IHostingStartup components give platforms an extensibility point to change existing applications, but can also lead to unexpected problems when dependencies conflict. Check for ASPNETCORE_HOSTINGSTARTUPASSEMBLIES if you run into strange startup problems.

.NET Core Opinion #6 - Be Wary of GUI Build Tools

Monday, October 29, 2018 by K. Scott Allen

It’s not that GUI build tools are bad, per se, but you have to watch out for tools that use too much magic, and tools that don’t let you version control your build with the same zealousness that you use for the source code of the system you are building.

Let’s start with the 2nd type of tool.

Why Text Files Rule

Many open source .NET Core projects use AppVeyor to build, test, and deploy applications and NuGet packages. When defining an AppVeyor build, you have a choice of using a GUI, or a configuration file.

With the GUI, you can point and click your way to a deployment. This is a perfectly acceptable approach for some projects, but lacks some of the rigor you might need for larger projects or teams.

AppVeyor GUI Build

You can also define your build by checking a YAML configuration file into the root of your repository. Here’s an excerpt:

AppVeyor YAML Build

Think about the advantages of the source-controlled YAML approach:

  • You can version the build with the rest of your software

  • You can use standard diff tools to see what has changed

  • You can see who changed the build

  • You can copy and share your build with other projects.

Also note that in the screen shot above, the YAML file calls out to a build script – build.ps1. I believe you should encapsulate as many build steps as possible into a package you can run on the build server and on a development machine. Doing so allows you to make changes, test changes, and troubleshoot build steps quickly. You can use MSBuild, Powershell, Cake, or any technology that makes builds easier. Integration points, like publishing to NuGet, will stay as configurable steps in the build platform.

Azure Pipelines also offers GUI editors for authoring build and release pipelines.

Azure Pipelines GUI Editor

Fortunately, a YAML option arrived recently and is getting better with every sprint.

Configure Azure Pipelines with YAML

Magic Tools

Magic tools are tools like Visual Studio. For over 20 years, Visual Studio has met the goal of making complex tasks simple. However, any time you want to perform a more complex task, the simplicity and hidden complexity can get in the way.

For example, what does the “Build” command in Visual Studio do, exactly? Call MSBuild? Which MSBuild? I have 10 copies of msbuild.exe on a relatively fresh machine. What parameters are passed? All the details are hidden and there’s nothing Visual Studio gives me as a starting point if I want to create a build pipeline on some other platform.

Another example of magic is the Docker support in Visual Studio. It is nice that I can right-click an ASP.NET Core project and say “Add Docker Support”. Seconds later I’ll be able to build an image, and not only run my project in a container, but debug through code executing in a container.

But, try to build or run the same image outside of Visual Studio and you’ll discover just how much context is hidden by tooling. You have to dig around in the build output to discover some of the parameters, and then you'll realize VS is mapping user secrets and setting up a different environment for the container. You might also notice the quiet installation of Microsoft.VisualStudio.Azure.Containers.Tools.Targets into your project, but you won't find any documentation or source code for thus NuGet package.

I think it is tooling like the Docker tooling that gives VS Code good traction. VS Code relies on configuration files and scripts that can make complex tasks simpler without making customizations and understanding inaccessible. Want to make a simple change to the Visual Studio approach? Don’t tell me you are going to edit the IL in Microsoft’s assembly! VS Code is to Visual Studio what YAML files are to GUI editors.

Summary

To wrap up this post in a single sentence: build and release definitions need source control, too.

Major Updates for my Building Secure Services in Azure Course

Tuesday, October 23, 2018 by K. Scott Allen

I’ve completely reworked my secure services course from scratch. There’s a lot of demos across a wide range of technologies here, including:

Docker Containers

  • Building ASP.NET Core projects using Visual Studio and Docker containers.

  • Deploying container images using Docker Hub and Azure App Services for Linux

  • Setting up continuous deployment for containers

Automation and Azure Resource Manager

  • Using ARM templates to deploy and provision resources in Azure (infrastructure as code)

  • Setting up Azure Key Vault

  • Storing secrets in Key Vault for use in ARM templates

Microservices and Container Orchestration

  • Using the new IHttpClientFactory and resiliency patterns for HTTP networking in ASP.NET Core

  • Container orchestration using Docker compose

  • Creating and using an Azure Container Registry (ACR)

  • Deploying multiple images using ACR And Compose

Cloud Identity

  • Creating your own test instance of Azure Active Directory

  • Authentication with OpenID Connect (OIDC) and Azure Active Directory

  • Securing APIs using Azure Active Directory and JWT tokens

  • Invoking secure APIs

  • Setting up an Azure B2C instance and defining your own policies

  • Securing an application using Azure B2C.

Note: this updated course is an hour shorter than the original course. Pluralsight authors generally want to make courses longer, not shorter, but I learned how to tell a better story this second time around. Also, the Docker story and tooling is markedly improved from last year, which saves times.

Building Secure Services in Azure

I hope you enjoy the course!