OdeToCode IC Logo

Choosing an Approach to Work with Azure REST APIs

Tuesday, January 30, 2018 by K. Scott Allen

Azure Resource Manager APIsThe Azure REST APIs allow us to interact with nearly every type of resource in Azure programmatically. We can create virtual machines, restart a web application, and copy an Azure SQL database using HTTP requests. There's a few choices to make when deciding how to interact with these resource manager APIs, and some potential areas of confusion. In this post and future posts I hope to provide some guidance on how to work with the APIs effectively and avoid some uncertainties.

Pure HTTP

If you can send HTTP messages, you can interact with the resource manager APIs at a low level. The Azure REST API Reference includes a list of all possible operations categorized by resource. For example, backing up a web site. Each endpoint gives you a URI, the available HTTP methods (GET, PUT, POST, DELETE, PATCH), and a sample request and response. All these HTTP calls need to be authenticated and authorized, a topic for a future post, but the home page describes how to send the correct headers for any request.

These low levels APIs are documented and available to use, but generally you want to write scripts and programs using a slightly higher level of abstraction and only know about the underlying API for reference and debugging.

Fortunately, specifications for all resource manager APIs are available in OpenAPI / Swagger format. You can find these specifications in the azure-rest-api-specs GItHub repository. With a codified spec in hand, we can generate wrappers for the API. Microsoft has already generated wrappers for us in several different languages.

Using a Client Library SDK

Microsoft provides Azure management libraries that wrap these underlying APIs for a number of popular languages. You can find links on the Microsoft Azure SDKs page. When looking for a management SDK, be sure to select a management SDK instead of a service SDK. A blob storage management SDK is an SDK for creating and configuring a storage account, whereas the service SDK is for reading and writing blobs inside the storage account. A management SDK generally has the name "management" or "arm" in the name (where arm stands for Azure Resource Manager), but the library names are not consistent across different languages. Instead, the names match the conventions for the ecosystem, and Node packages follow a different style than .NET and Java. As an example, the service SDK for storage in Node is azure-storage-node, whereas the management package is azure-arm-storage.

Using the Command Line

In addition to SDKs, there are command line utilities for managing Azure. PowerShell is one option. In my experience, PowerShell provides the most complete coverage of the management APIs, and over the years I've seen a few operations that you cannot perform in the Azure portal, but can perform with PowerShell.

However, my favorite command line tool is the cross-platform Azure CLI. Not being a regular user of PowerShell, I find the CLI easier to work with and the available commands are easier to discover. That being said, Azure CLI doesn't cover all of Azure, although new features arrive on a regular cadence.

In general, stick with the command line tools if you have quick, simple scripts to run. Some applications, however, require more algorithms, logic, heuristics, and cooperation with other services. For these scenarios, I'd prefer to work with an SDK in a programming language like C#.

Speaking of which ...

Choices for C# Developers

If you are a C# developer who wants to manage Azure using C# code, you have the option of going with raw HTTP messages using a class like HttpClient, or using the SDK. Use the SDK. There is enough flexibility in the SDKs to do everything you need, and you don't need to build your own encapsulation of the APIs.

You do need to choose the correct version of the SDKs. If you search the web for examples of managing Azure from C# code, you'll run across NuGet packages with the name Microsoft.WindowsAzure.Management.*. Do not use these packages, they are obsolete.  Make sure you are using packages that start with Microsoft.Azure.Management.* (no Windows in the name).

One caveat to these packages is that the classes inside are auto-generated from the OpenAPI specs, so they tend to feel quirky and discoverability can be difficult. I've found using a good code navigation tool like dotPeek allows me to find the model that I want (model classes represent the data returned by a given API endpoint, i.e. a resource in REST terms), and then use "Find References" to see the operations that act on the model resource.

For C#, It's often easier to work with the Fluent .NET management APIs. These fluent management APIs build on top of the underlying management package, but have extension methods tailored for discoverability and readability. For example, Microsoft.Azure.Management.Sql allows you to use the management API from generated .NET code. The Microsoft.Azure.Management.Sql.Fluent package adds discoverability and readability to the code. Note that not all management packages have a fluent counterpart, however, and not all operations might be exposed from the fluent interface.

Summary

This post covers some of the options and decision points for working with the Azure Resource Manager API. In future posts we'll see how to write C# code for interacting with the base API and the fluent API. First, however, we'll have to setup a service principal so our code can be authenticated and authorized to execute against our resources.

Also see: Setting Up Service Principals to Use the Azure Management APIs and Working with Azure Management REST APIs

New Pluralsight Course on Packaging and Deploying ASP.NET Core

Monday, January 29, 2018 by K. Scott Allen

Recorded many months ago in my previous life, this new course shows how to deploy ASP.NET Core into Azure using a few different techniques. We'll start by using Git, then progress to using a build and release pipeline in Visual Studio Team Services. We'll also demonstrate how to use Docker and containers for deployment, and how to use Azure Resource Manager Templates to automate the provisioning and updates of all Azure resources. 

Lars has a blog post with a behind the scenes look, and you'll find the new course on Pluralsight.com.

Recording of Packaging and Deploying ASP.NET Core

Interacting with Azure SQL Using All Command Line Tools

Tuesday, January 23, 2018 by K. Scott Allen

Microsoft's collection of open source command line tools built on Python continues to expand. Let's take the scenario where I need to execute a query against an Azure SQL database. The first step is poking a hole in the firewall for my current IP address. I'll use the Azure CLI 2.0:

λ az login
To sign in, use a web browser to open the page https://aka.ms/devicelogin 
 and enter the code DRAMCY103 to authenticate.
[
  {
     ...subscription 1 ...
  },
  {
     ... subscription 2 ...
  }
]

For the firewall settings, az has firewall-rule create command:

λ az sql server firewall-rule create -g resourcegroupname 
   -s mydbserver -n watercressip --start-ip-address 173.169.164.144 
   --end-ip-address 173.169.164.144
{
  "endIpAddress": "173.169.164.144",
  ...
  "type": "Microsoft.Sql/servers/firewallRules"
}

Now I can launch the mssql-cli tool.

λ mssql-cli -S mydbserver.database.windows.net 
            -U appusername  -d appdbname

Auto-complete for columns works well when you have a FROM clause in place (maybe LINQ had it right after all).

mssql-cli at work

If I'm in transient mode, I'll clean up and remove the firewall rule.

λ az sql server firewall-rule delete 
   -g resourcegroupname -s mydbserver -n watercressip               

The mssql-cli has a roadmap, and I'm looking forward to future improvements.

Task.CompletedTask and Task.Result Tips

Thursday, January 18, 2018 by K. Scott Allen

In any method returning a Task, it is desirable to avoid using Task.Run if you can compute a result without going async. For example, if logic allows you to short-circuit a computation, or if you have a fake method in a test returning a pre-computed answer, then you don't need to use Task.Run.

Example:

Here is a method that is not going to do any work, but needs to return a task to fulfill an interface contract for a test:

public Task ComputationWithSideEffects(string someId)
{
    return Task.Run(() => {
        // i'm just here to simulate hard work
    });
}

Instead of returning the result of Task.Run, there are two helpers on the Task class that make the code more readable and require a bit less runtime overhead.

For the above scenario, I'd rather use Task.CompletedTask:

public Task ComputationWithSideEffects(string someId)
{
    return Task.CompletedTask;
}

What if the caller expects to receive a result from the task? In other words, what if you return a Task<T>? In this scenario, if you already have an answer, use Task.FromResult to wrap the answer.

public Task<Patience> FindPatience()
{
    if(cachedPatience != null) 
    {
        return Task.FromResult(cachedPatience);
    }
    else
    {
        return ImAlreadyGone();
    }
}

The Cloud is a Small World

Tuesday, January 16, 2018 by K. Scott Allen

See the source image

Last night, I began to wonder about my virtual neighbors here at OdeToCode.

You see, when you provision an Azure App Service, Azure will give your service a public IP address. The IP address will stay with the App Service into the future, as long as you don't delete the App Service.

The IP address is a virtual address shared by many App Services that run on the same "stamp" in a region (where a stamp is essentially a cluster of servers). If, for some reason, you don't want a public address or don't want to share an address, another approach is to setup an isolated App Service, but isolation is pricey. For most of us, using the shared, virtual, and public IP is fine, as we can use custom domains and SSL certificates, and everything just works as expected.

But, back to last night.

What if I could wander around my stamp like a character in a Gibson novel? Who would I see? Do I share an IP address with a celebrity website? Do I live in a criminalized neighborhood where bewildered netizens show up after being click-jacked? Do I have any neighbors who would lend me 2 eggs, and a cup of all-purpose flour if in a pinch?

First step, finding the IP address for OdeToCode.

λ nslookup odetocode.com
Server:  UnKnown
Address:  192.168.200.1

Non-authoritative answer:
Name:    odetocode.com
Address:  168.62.48.183

Taking this IP address to hostingcompass.com, I can see there are 84 known web sites hosted on the IP address (and this wouldn't include sites fronted by a proxy, like Cloudflare, or without a custom domain, I think).

What amazing is not just how many websites I recognized, but how many websites are run by people I personally know. For example:

http://rachelappel.com/

http://nickberardi.com/

http://andrewconnell.com/

https://www.alvinashcraft.com/

http://www.schwammysays.net/

The neighborhood also includes an inordinate number of bars and restaurants, as well as a photographer and investment advisor. Cheers!

Streaming APIs In ASP.NET Core

Thursday, January 11, 2018 by K. Scott Allen

Antietam Creek

The topics this week continue to be stream related.

When building an API, some teams have a strict rule to always use ToList in a LINQ query and materialize the query results before sending the results back to an HTTP client. There are some advantages to this eager approach:

1) You’ll fail fast, compared to deferred execution.

2) You’ll never worry about controller code using IQueryable operators that can destroy a query plan.

However, every application may require a different set of rules. For APIs that move heavy amounts of data, chewing up memory to materialize results can lead to high memory usage, paging, and sluggish performance. Let’s look at an example using the following, simple model.

public class Widget
{
    public int Id { get; set; }
    public string Name { get; set; }
}

Imagine we have 1 million widgets to move across the Internet. We’ll use the following controller action to respond to a request for all the widgets.

[HttpGet]
public List<Widget> Get()
{
    var model = _widgetData.GetAll();
    return model;
}

Notice the return type of List<Widget>, meaning the GetAll method has already placed every widget into memory. ASP.NET Core handles the request well. For the client, the 35 megabytes of JSON streams out of the server with a chunked encoding, which is ideal.

HTTP/1.1 200 OK 
Transfer-Encoding: chunked 
Content-Type: application/json; charset=utf-8 
Server: Kestrel

The downside is looking at the memory consumed on the server. After just three requests, the dotnet server process is using over 650MB of memory without serving any other requests. Move 4 million widgets at a time and the process is over 1.3GB.

ASP.NET Core Memory Usage ToList

Using IEnumerable

Assuming your data source can stream results (like the firehose cursor of SQL Server), you can keep more memory available on the server by moving data with IEnumerable or IQueryable all the way to the client.

[HttpGet]
public IEnumerable<Widget> Get()
{
    // GetAll now returns IEnumerable, 
    // not using ToList, but perhaps using
    // AsEnumerable
    var model = _widgetData.GetAll();            
    return model;
}

With multiple inbound requests, moving even 4 million Widgets requires less than 250MB of heap space over time.

ASP.NET Core Memory Usage IEnumerable

Streaming yields a considerable savings in a world where memory is still relatively precious. Don’t let development rules overrule your context or take away your precious resources, and always use benchmarks and tests to determine how your specific application will behave.

My Microsoft Azure Channel on Pluralsight

Wednesday, January 10, 2018 by K. Scott Allen

Still catching up on Pluralsight announcements from last year ...

The "Everything .NET Developers Should Know About Microsoft Azure" channel is a list of the Azure courses I made last year. 10+ hours of content, with more to come and updates this year.

Everything .NET Developers Should Know About Microsoft Azure