Tracking Down a Mysterious 502.5 Error in ASP.NET Core

When I posted 7 Tips for Troubleshooting ASP.NET Core Startup Errors, I thought I had covered every tip needed to track down any possible startup error. In particular, the last step never fails to reveal the culprit.

Then, the last step failed me.

This particular application would not run in an Azure App Service but would run everywhere else - in development, in debug, in release, and in the command line console on the App Service server. How’s that possible?

Soup du Jour

There is a proverb that says "too many cooks will spoil the broth", meaning when too many people are trying to add their own expertise on the same project, the result will be a disaster. Such is currently the case when two chefs, Visual Studio and the Azure portal, both try to cook a pot of Application Insights.

Application Insights is an invaluable resource for monitoring and troubleshooting an application, but AppInsights has also been a bit flaky to setup with ASP.NET Core in the past. Sometimes AI doesn’t work no matter how hard you try, and sometimes AI does work when you try to make it stop.

Despite the flakiness, AppInsights works great once everything is configured. It is no surprise that both the Azure Portal and Visual Studio encourage the use of AppInsights, but this leads to common questions about the best approach to use AppInsights.

Am I supposed to install the AppInsights NuGet to my project? This "build time" configuration allows an app to use the entire SDK and provide custom telemetry to AppInsights.

Or, am I supposed to setup AppInsights as an Azure App Service extension? This is known as "run-time" configuration and doesn’t require any code changes or new deployments. The official doc uses wording to encourage the "build-time" approach, but you can also find a page that says to use both approaches for the best experience.

The failing application took the "both" approach, and it turns out the "both" approach was the source of the error.

Finding the Failure

When the application failed with an HTTP 502.5 error, I went to Kudu and ran the app from the console. The application would work from the console, but consistently fail when launched from the worker process in a Windows App Service. This behavior was curious, because both approaches run the same app on the same virtual server with the same file system and the same environment variables. But, decades of debugging experience gave me the insight that something about the environment must be different.

Ultimately I found the true source of the failure by looking at the live stream of the app’s standard output stream.

Even this error was a bit curious since the dotnet publish operation included the required AppInsights assembly in the output. I dug in further and eventually looked at Kudu diagnostics for the w3wp.exe worker process. That’s when the answer jumped out at me.

ASP.NET 2.0 introduced the ASPNETCORE_HOSTINGSTARTUPASSEMBLIES environment variable seen above. The variable allows a host to inject additional startup logic to a compiled application by naming assemblies with IHostingStartup components inside. I had no idea what StartupBootstrapper could be, but a quick search revealed this assembly to be part of the Azure AppInsights extension. In the end, the AppInsights installed by Visual Studio and the AppInsights installed by the Azure extension were incompatible versions. To get the app working, I could do any of the following:

• Uninstall the AppInsights extension in the App Service
• Remove the AppInsights package reference from the project
• Leave both in place, but make sure the package version and extension version are compatible

Option #1 seems like the best option, since one might never know when the extension will update and break the application again.

Summary

If you think you know exactly what your ASP.NET Core application does during Startup, you could be wrong. IHostingStartup components give platforms an extensibility point to change existing applications, but can also lead to unexpected problems when dependencies conflict. Check for ASPNETCORE_HOSTINGSTARTUPASSEMBLIES if you run into strange startup problems.