Streaming APIs In ASP.NET Core

Thursday, January 11, 2018

Antietam Creek

The topics this week continue to be stream related.

When building an API, some teams have a strict rule to always use ToList in a LINQ query and materialize the query results before sending the results back to an HTTP client. There are some advantages to this eager approach:

1) You’ll fail fast, compared to deferred execution.

2) You’ll never worry about controller code using IQueryable operators that can destroy a query plan.

However, every application may require a different set of rules. For APIs that move heavy amounts of data, chewing up memory to materialize results can lead to high memory usage, paging, and sluggish performance. Let’s look at an example using the following, simple model.

public class Widget
{
    public int Id { get; set; }
    public string Name { get; set; }
}

Imagine we have 1 million widgets to move across the Internet. We’ll use the following controller action to respond to a request for all the widgets.

[HttpGet]
public List<Widget> Get()
{
    var model = _widgetData.GetAll();
    return model;
}

Notice the return type of List<Widget>, meaning the GetAll method has already placed every widget into memory. ASP.NET Core handles the request well. For the client, the 35 megabytes of JSON streams out of the server with a chunked encoding, which is ideal.

HTTP/1.1 200 OK 
Transfer-Encoding: chunked 
Content-Type: application/json; charset=utf-8 
Server: Kestrel

The downside is looking at the memory consumed on the server. After just three requests, the dotnet server process is using over 650MB of memory without serving any other requests. Move 4 million widgets at a time and the process is over 1.3GB.

ASP.NET Core Memory Usage ToList

Using IEnumerable

Assuming your data source can stream results (like the firehose cursor of SQL Server), you can keep more memory available on the server by moving data with IEnumerable or IQueryable all the way to the client.

[HttpGet]
public IEnumerable<Widget> Get()
{
    // GetAll now returns IEnumerable, 
    // not using ToList, but perhaps using
    // AsEnumerable
    var model = _widgetData.GetAll();            
    return model;
}

With multiple inbound requests, moving even 4 million Widgets requires less than 250MB of heap space over time.

ASP.NET Core Memory Usage IEnumerable

Streaming yields a considerable savings in a world where memory is still relatively precious. Don’t let development rules overrule your context or take away your precious resources, and always use benchmarks and tests to determine how your specific application will behave.


Comments
gravatar Andrew Thursday, January 11, 2018
Sorry for my ignorance and lack of knowledge, why does the 2nd Get() consume less memory? Is it because the results are both fetched from the _widgetData and returned to client by pages of data? (does it use HTTP range requests?). So basically the server never retrieves all results from _widgetData.GetAll(). Thanks!
gravatar Stefan Thursday, January 11, 2018
I think t is more serious the disadvantage keeping open a datasource's connection and other datasource resources allocated, . Instead we implement server-paging. and clients request page results based on a predefined key-sorting.
gravatar Jeff Thursday, January 11, 2018
FWIW we recommend HTTP streaming for all large N1QL requests in Couchbase. The difference in performance is huge; no OOM exceptions, nothing goes to the LOH (GC purrs) and memory consumption is a constant.
Alex Friday, January 12, 2018
Or just use a cache for a “GetAll” method. No open connections and no exponential memory consumption.
gravatar SB Monday, January 15, 2018
@Stefan API developers may needs to return large volumes of data, so paging/sorting isn't necessarily possible (as K. Scott stated, in the opening paragraphs). Also, your concern regarding the data source's connection is somewhat misguided. The connection is only opened at the point at which the query is enumerated; therefore, whether the query is enumerated within the action method or by the framework, to generated the response, the connection is open for exactly the same amount of time. However, there is (possibly*) an issue with deferring the enumeration of the query to the framework: the framework won't dispose the ' _widgetData' and you can't place it in a using statement, as that will cause an error when the framework tries to enumerate it. * resources will be disposed by the built-in DI container, if the service was created by the container, or within the controller's Dispose method (as long as the developer overrides the base implementation).
Your Comment