Introduction
I have been working recently on integration tests, an important topic for building sustainable software. I then discovered ASP.Net Core's TestServer to perform fast integration testing. But is it really faster than a classic ASP.Net Core server? Let's see that.
The test project
The project we are going to work on is a simple online sales website. It contains some microservices, all written using a functional architecture. We will focus on the Shopping Cart microservice. It contains an endpoint to add a product to a cart: we will use it to perform some benchmarking.
[HttpPost]
public async Task<ActionResult> AddProductToCart([FromBody] AddProductToCartDto addProductToCartDto)
{
/*...*/
}
ASP.Net Core TestServer
ASP.Net Core TestServer is a web server you can find in ASPNetCore.TestHost package. It makes the handling of HTTP requests fast because they don't go through a normal network stack.
BenchmarkDotNet
We will use BenchmarkDotNet to compare the performance between the two approaches. It is a small and extremely well-made library to carry out performance measurements. The purpose of this article is not to show how it works, for that you need to refer to its documentation.
The benchmark classes
ServerBase.cs
To make a fair comparison, the context must be the same between the two servers. So we will pool as much code as possible between the two approaches, i.e.:
Creation of an in-memory SQLite database.
Initialization of the test data.
Initialization of DI services.
Initialization of the web server.
These initializations are done in the constructor of an abstract ServerBase.cs class:
public abstract class ServerBase
{
protected WebApplicationBuilder WebAppBuilder { get; set; }
protected WebApplication WebApp { get; set; }
protected CallistoDbContext DbContext { get; set; }
protected SqliteConnection Connection { get; set; }
protected HttpClient? Sut { get; set; }
private readonly string _baseUrl;
protected ServerBase(string baseUrl)
{
WebAppBuilder = WebApplication.CreateBuilder();
WebAppBuilder.Services.AddControllers().PartManager.ApplicationParts.Add(new AssemblyPart(typeof(CartController).Assembly)); // Necessary to add controllers from other projects
Connection = new SqliteConnection("DataSource=file::memory:"); // We will use an in-memory Database
Connection.Open();
// Database Initialization
var options = new DbContextOptionsBuilder<CallistoDbContext>().UseSqlite(Connection);
DbContext = new CallistoDbContext(options.Options);
DbContext.Database.Migrate();
DbContext.Product.Add(new ProductModel("2EJX-6HCC", "PS5", "Playstation 5", 500, DateTime.UtcNow));
DbContext.SaveChanges();
// Services configuration
WebAppBuilder.Services.AddSingleton<CallistoDbContext>(_ => DbContext);
WebAppBuilder.Services.AddTransient<ICartRepository, CartRepository>();
WebAppBuilder.Services.AddTransient<IProductRepository, ProductRepository>();
WebAppBuilder.WebHost.UseUrls(baseUrl);
_baseUrl = baseUrl;
}
}
Finally, we will add a test method to call the endpoint for adding a product to a cart. The first call adds a product to a new cart (because no CartId is given), then we add 10 times the same product in this cart:
public async Task RunTest()
{
var addProductToCart = new
{
CartId = "",
ProductId = "2EJX-6HCC",
Quantity = 1
};
var response = await Sut!.PostUtf8Async($"{_baseUrl}/carts", addProductToCart);
var cartId = (await response!.Content.ReadFromJsonAsync<List<Read.Entities.Cart>>())!.ElementAt(0).CartId;
// Adding 10 times the same product in the cart
for (int i = 0; i < 10; i++)
{
addProductToCart = new
{
CartId = cartId,
ProductId = "2EJX-6HCC",
Quantity = 1
};
await Sut.PostUtf8Async($"{_baseUrl}/carts", addProductToCart);
}
}
ClassicServer.cs
This class contains initializations specific to the test with the classic server. It inherits from ServerBase.cs:
using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
namespace Callisto.Sales.Benchmark
{
internal class ClassicServer : ServerBase
{
public ClassicServer(string baseUrl) : base(baseUrl)
{
WebAppBuilder.Services.AddHttpClient();
var serviceProvider = WebAppBuilder.Services.BuildServiceProvider();
Sut = serviceProvider.GetService<HttpClient>()!;
WebApp = WebAppBuilder.Build();
WebApp.MapControllers();
WebApp.Start();
}
}
}
TestServer.cs
This class contains initializations specific to testing with the ASP.Net Core TestServer; it also inherits from the ServerBase.cs class:
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.Hosting;
namespace Callisto.Sales.Benchmark
{
internal class TestServer : ServerBase
{
public TestServer(string baseUrl) : base(baseUrl)
{
WebAppBuilder.WebHost.UseTestServer(); // Using TestServer
WebApp = WebAppBuilder.Build();
WebApp.MapControllers();
WebApp.Start();
Sut = WebApp.GetTestClient(); // Getting an HttpClient directly from the app.
}
}
}
The benchmark class
The benchmark class is as follows:
using BenchmarkDotNet.Attributes;
namespace Callisto.Sales.Benchmark
{
[SimpleJob(launchCount: 5, warmupCount: 2, iterationCount: 10)]
public class ClassicServerVSTestServerBenchmark
{
private readonly ClassicServer _classicServer;
private readonly TestServer _testServer;
public ClassicServerVSTestServerBenchmark()
{
_classicServer = new ClassicServer("http://localhost:5000");
_testServer = new TestServer("http://localhost:5001");
}
[Benchmark]
public async Task Classic_Server_Benchmark()
{
await _classicServer.RunTest();
}
[Benchmark]
public async Task Test_Server_Benchmark()
{
await _testServer.RunTest();
}
}
}
And finally, to run the benchmark, we create a simple console project:
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;
namespace Callisto.Sales.Benchmark
{
internal class Program
{
static void Main(string[] args)
{
BenchmarkRunner.Run<ClassicServerVSTestServerBenchmark>();
}
}
}
First results
Now, let's run the benchmark:
dotnet run -c Release
After a few minutes, I get these results:
| Method | Mean | Error | StdDev |
|---------------------------- |-------------:|-------------:|-------------:|
| Classic_Server_Benchmark | 455,511.2 us | 24,192.14 us | 48,314.32 us |
| Test_Server_Benchmark | 454,201.1 us | 7,812.13 us | 15,780.88 us |
Intermediate conclusion: the gain with the test server is not obvious (~1.5 ms/test). I was a bit disappointed... So I wondered what the result would be without database interaction, only focusing on the network part. Let's see that.
A benchmark without database interaction
In the Shopping Cart microservice, I added a stub endpoint that does strictly nothing.
Then, in ServerBase.cs, I added a test for this endpoint :
public async Task RunTestWithoutDatabaseInteraction()
{
await Sut.GetAsync($"{_baseUrl}/carts/stub");
}
I then added the following two new benchmarks in ClassicServerVSTestServerBenchmark.cs :
[Benchmark]
public async Task Classic_Server_Without_Database_Interaction_Benchmark()
{
await _classicServer.RunTestWithoutDatabaseInteraction();
}
[Benchmark]
public async Task Test_Server_Without_Database_Interaction_Benchmark()
{
await _testServer.RunTestWithoutDatabaseInteraction();
}
Here are the results :
| Method | Mean | Error | StdDev |
|------------------------------------------------------ |-------------:|-------------:|-------------:|
| Classic_Server_Without_Database_Interaction_Benchmark | 2,137.9 us | 32.98 us | 61.94 us |
| Test_Server_Without_Database_Interaction_Benchmark | 177.6 us | 6.99 us | 13.48 us |
This time, the gain brought by the TestServer is clear: the calls to an endpoint are 12x faster! We gain ~2 ms per endpoint call, which is consistent with the results from the previous benchmark.
Conclusion
ASP.Net Core's TestServer only saves time on network calls by a few milliseconds. Perhaps the gains could be significant on a broad base of integration tests.
In any case, your database will remain the bottleneck for integration testing.
About the cover image
The Source of the Universe. In the desert of Atacama in Chile, the Milky Way seems to spring from a small and perfectly calm water hole...