Tracking Down an AspNetCore Build Error

Building AspNet Core from source is straightforward, unless, like me, you run into strange errors. My first build compiled over 600 projects successfully, but ended up failing and showing the following message.

Build FAILED.

ALINK : error AL1078: Error signing assembly -- Access is denied. [...\LocalizationSample.csproj]
ALINK : error AL1078: Error signing assembly -- Access is denied. [...\LocalizationWebsite.csproj]
ALINK : error AL1078: Error signing assembly -- Access is denied. [...\RazorPagesWebSite.csproj]
ALINK : error AL1078: Error signing assembly -- Access is denied. [...\BasicTestApp.csproj]
    0 Warning(s)
    4 Error(s)

I first noticed that all four failures were because of AL.exe, a.k.a ALINK, the little known assembly linker that ships with .NET. Compared to a linker in the world of C++, the .NET linker is an obscure tool that is rarely seen unless a build needs to work with satellite assemblies generated from .resx files, or sign an assembly with a public/private key pair.

The next bit I noticed was that the first two projects involved localization, so I was sure the build errors were some sort of problem with satellite assemblies or resource files. Unfortunately, the error message access is denied is short on helpful details. I went looking for obvious problems and verified that projects existed in the correct directories, and that no files were missing.

Generating a Build Log

In need of more information, I decided to create a detailed log of of my AspNetCore build. Building all of AspNetCore requires running a build script, one of build.cmd, build.ps1, or build.sh, depending on your OS and mood. All of these scripts accept parameters. For example, build -BuildNative will build the native C/C++ projects, which includes the AspNetCoreV2 IIS module. The default is to only -BuildManaged. You can also thunk MSBuild parameters through the build script, so I used the following to create a binary log of detailed verbosity, and avoided writing to the console to save time:

build.cmd -bl -verbosity:detailed -noconsolelogger  

All AspNetCore build output goes into an artifacts folder, including the binary log. To view the binary log I use MSBuildStructureLog, a tool I've written about previously.

A word of warning - AspNetCore build logs are not the kind of logs that you can pack into an overnight bag and sling over your shoulder. If you want to open and view the logs I recommend a machine with at least 16GB of RAM, but 32 is better. The machine needs to handle a process with 10+ GB committed or else be sucking mud.

Once the build log loads, it is easy to find build errors.

The command line for each of the 4 build errors looked like the following:

CommandLineArguments = ...\al.exe /culture:es-ES /delaysign- /keyfile: and so on...

I tried the command line myself, and sure enough, there is an access denied error.

The good news, I thought to myself, was that I can easily reproduce the build error without running the entire build. The bad news, I thought to myself, was that I still don't know what resource is denying access. Time to bring in more tools.

Tracking Down Access Denied with SysInternals Process Monitor

The Sysinternals Suite has been invaluable for debugging over the years. Process Monitor in particular is the first tool that comes to mind when I need to track down file system activity.

If you've ever worked with procmon, you'll know the tool can capture an overwhelming amount of data. The first thing to do is to add one or more filters to the capture. In the screen shot below I am only showing file system activity for any process named al.exe. With the filter in place I can execute the command line, and sure enough, procmon shows the access denied error.

The access denied message occurs on a file inside the ProgramData\Microsoft\Crypto\RSA\MachineKeys folder. It appears to me, based on my limited knowledge of the Windows crypto API, that al.exe is trying to write an ephemeral private key into the machine keys folder, but my user account doesn't have permission to create the file.

Now I know why the build is failing, but there is no clear solution to fix the problem.

What to do?

Stay tuned for the next blog post where I will reveal the unexciting solution to this problem in a stupendously boring fashion. Sneak preview: the solution does not involve sudo or changing folder permissions to give my account full control...