.NET Steps – End to End Testing

In my last post, I talked about integration testing and during my investigation, I run into something a bit “confusing” – what is the line that separates integration tests from end to end tests?

If you try to do a quick google search, you are gonna get some conflicting answers. What most of them agree on is that it is kind of testing that simulates real life user flow. And if you look at Microsoft documentation they are calling it integration tests. I mean, isn’t calling an API a thing that users do when they are using your API? I have no idea.

So, I’m gonna stick with the following: when you call API endpoint, you are simulating real life user, hence, writing an end to end test.

Like I mentioned in my previous post, there are some challenges when your want to test your application. Most of them being how to handle state of external systems.

But, I’m wondering, can we battle those challenges with the same approach?

Do you really think that there would be blog post about it if we couldn’t?

Implementation

If you don’t recall (or you didn’t read last post), we have a simple application that consists of 2 endpoints

GET /users -> which gets users from the database
GET /pets -> which gets pets from some external API

Let’s talk about approach I’m going to take

I’m going to use xUnit
I’m going to use FluentAssertions
Before application starts, I want to run docker containers – one for database and that will act as mock for external api.
Then I want to run my application using Program.cs (like you would normally run your .NET Web Api application). This means that everything in our Startup.cs will be configured (since we have automatic migrations, this means that they will be applied to)
I want to test my endpoints by going to an actual url (eg. I want to make a GET /api/v1/users)
Before every test, I want to insert some initial state where I need to
After every test, I want to cleanup that state (eg, after the test database would be in a state after we applied migrations)

Now, let’s try to create “end to end” tests.

First, lets add secrets.json file to our Company.Shorts project

{
  "ExternalApiSettings:Url": "Url",
  "PostgresAdapterSettings:Url": "Url",
  "ExampleAdapterSettings:Url": "Url"
}

Lets create a xUnit Project in our tests folder called: Company.Shorts.EndToEnd.Tests
Let’s add dependencies:
- TestContainers – for running our docker containers
- TestContainers.PostgreSql – for running our docker container for database
- Microsoft.Data.SqlClient – for our custom attributes
- XUnit.DependencyInjection – so that we can inject fixtures inside another fixture
At the end, the Company.Shorts.EndToEnd.Tests.csproj should look like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="FluentAssertions" Version="6.11.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="7.0.5" />
    <PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.1" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
    <PackageReference Include="Testcontainers" Version="3.2.0" />
    <PackageReference Include="Testcontainers.PostgreSql" Version="3.2.0" />
    <PackageReference Include="Xunit.DependencyInjection" Version="8.7.1" />
    <PackageReference Include="xunit" Version="2.4.2" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.5">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="coverlet.collector" Version="3.1.2">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\src\Company.Shorts\Company.Shorts.csproj" />
  </ItemGroup>

  <ItemGroup>
    <ContentWithTargetPath Include="Resources\**">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <TargetPath>Resources\%(RecursiveDir)\%(Filename)%(Extension)</TargetPath>
    </ContentWithTargetPath>
  </ItemGroup>

  <ItemGroup>
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
  </ItemGroup>

</Project>

After that, we need to create:
- PostgresDatabaseFixture – for creating a container for database
- MockWebServerFixture – for creating fa container for mock server
- WebApplicationFactoryFixture – that will be used to start our API and replace one or more settings files (for our example we need to replace postgre connection string with one from created by PostgresDatabaseFixture and we need to replace external api base address from one once created by MockWebServerFixture.
- MockWebServerContainerConstants – to keep constants in one place
- PostgresSqlContainerConstants – to keep constants in one place
- CollectionFixtureConstants – to keep constants in one place
- IntegrationTestCollection – to define what fixtures should run inside one test context
- Startup – to register dependency injection for fixtures (we are injection 2 fixtures in WebApplicationFactoryFixture)

namespace Company.Shorts.EndToEnd.Tests.Internal.Fixtures
{
    using Company.Shorts.EndToEnd.Tests.Internal.Postgres;
    using DotNet.Testcontainers.Builders;
    using System;
    using Testcontainers.PostgreSql;

    public class PostgresDatabaseFixture : IDisposable
    {
        private bool _disposed;
        private readonly IEnviromentVariableManager eventVariableManager = new PostgresEnviromentVariableManager();

        public PostgresDatabaseFixture()
        {
            this.PqsqlDatabase = new PostgreSqlBuilder()
                .WithImage(PostgreSqlContainerConstants.Image)
                .WithDatabase(PostgreSqlContainerConstants.Database)
                .WithUsername(PostgreSqlContainerConstants.Username)
                .WithPassword(PostgreSqlContainerConstants.Password)
                .WithExposedPort(PostgreSqlContainerConstants.Port)
                .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(PostgreSqlContainerConstants.Port))
                .Build();

            this.PqsqlDatabase.StartAsync().Wait();

            this.eventVariableManager.Set(PqsqlDatabase.GetConnectionString());
        }

        public PostgreSqlContainer PqsqlDatabase { get; }

        protected virtual void Dispose(bool disposing)
        {
            if (!_disposed)
            {
                if (disposing)
                {
                    PqsqlDatabase.DisposeAsync().AsTask().Wait();
                }

                _disposed = true;
            }
        }

        public void Dispose()
        {
            Dispose(disposing: true);
            GC.SuppressFinalize(this);
        }
    }
}

namespace Company.Shorts.EndToEnd.Tests.Internal.Fixtures
{
    using Company.Shorts.EndToEnd.Tests.Internal.MockWebServer;
    using DotNet.Testcontainers.Builders;
    using DotNet.Testcontainers.Containers;
    using System;

    public class MockWebServerFixture : IDisposable
    {
        private bool _disposed;

        private readonly IEnviromentVariableManager enviromentVariableManager = new MockWebServerEnvironmentVariableManager();

        public MockWebServerFixture()
        {
            this.MockWebServerContainer = new ContainerBuilder()
                .WithImage(MockWebServerContainerConstants.Image)
                .WithPortBinding(MockWebServerContainerConstants.Port)
                .WithWaitStrategy(Wait.ForUnixContainer().UntilMessageIsLogged("INFO 1080 started on port: 1080"))
                .Build();

            this.MockWebServerContainer.StartAsync().Wait();

            this.Url = $"http://{this.MockWebServerContainer.Hostname}:{this.MockWebServerContainer.GetMappedPublicPort(MockWebServerContainerConstants.Port)}";

            this.enviromentVariableManager.Set(this.Url);
        }

        public IContainer MockWebServerContainer { get; }

        public string Url { get; }

        protected virtual void Dispose(bool disposing)
        {
            if (!_disposed)
            {
                if (disposing)
                {
                    this.MockWebServerContainer.DisposeAsync().AsTask().Wait();
                }

                _disposed = true;
            }
        }

        public void Dispose()
        {
            Dispose(disposing: true);
            GC.SuppressFinalize(this);
        }
    }
}

namespace Company.Shorts.EndToEnd.Tests.Internal.Fixtures
{
    using Company.Shorts;
    using Microsoft.AspNetCore.Mvc.Testing;
    using System;
    using System.Net.Http;

    public class WebApplicationFactoryFixture : IDisposable
    {
        private bool _disposed;

        public WebApplicationFactoryFixture(PostgresDatabaseFixture fixture, MockWebServerFixture webServerFixture)
        {
            EnvironmentUtils.SetTestEnvironment();

            var application = new WebApplicationFactory<Program>().WithWebHostBuilder(conf =>
            {
                conf.UseSetting("PostgresAdapterSettings:Url", fixture.PqsqlDatabase.GetConnectionString());
                conf.UseSetting("ExternalApiSettings:Url", webServerFixture.Url);
            });

            HttpClient = application.CreateClient();
        }

        public HttpClient HttpClient { get; }

        protected virtual void Dispose(bool disposing)
        {
            if (!_disposed)
            {
                if (disposing)
                {
                    HttpClient.Dispose();
                }

                _disposed = true;
            }
        }

        void IDisposable.Dispose()
        {
            // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method
            Dispose(disposing: true);
            GC.SuppressFinalize(this);
        }
    }
}

namespace Company.Shorts.EndToEnd.Tests.Internal.Fixtures
{
    public static class MockWebServerContainerConstants
    {
        public const string Image = "mockserver/mockserver";
        public const int Port = 1080;
    }
}

namespace Company.Shorts.EndToEnd.Tests.Internal.Fixtures
{
    public static class PostgreSqlContainerConstants
    {
        public const string Image = "postgres:15.1-alpine";
        public const string Database = "ShortsUserDb";
        public const string Username = "postgres";
        public const string Password = "postgres";
        public const int Port = 5432;
    }
}

namespace Company.Shorts.EndToEnd.Tests.Internal.Fixtures
{
    public static class CollectionFixtureConstants
    {
        public const string Integration = "Integration";
    }
}

namespace Company.Shorts.EndToEnd.Tests.Internal.Fixtures
{
    using Xunit;

    [CollectionDefinition(CollectionFixtureConstants.Integration)]
    public class IntegrationTestCollection
        : ICollectionFixture<PostgresDatabaseFixture>,
        ICollectionFixture<MockWebServerFixture>,
        ICollectionFixture<WebApplicationFactoryFixture>
    {
    }
}

namespace Company.Shorts.EndToEnd.Tests
{
    using Company.Shorts.EndToEnd.Tests.Internal.Fixtures;
    using Microsoft.Extensions.DependencyInjection;

    public class Startup
    {
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddSingleton<PostgresDatabaseFixture>();
            services.AddSingleton<MockWebServerFixture>();
        }
    }
}

At the end, lets write our tests

namespace Company.Shorts.EndToEnd.Tests
{
    using Company.Shorts.Application.PetsAggregate.Queries;
    using Company.Shorts.EndToEnd.Tests.Internal.Fixtures;
    using Company.Shorts.EndToEnd.Tests.Internal.MockWebServer;
    using FluentAssertions;
    using Newtonsoft.Json;
    using System.Collections.Generic;
    using System.Threading.Tasks;
    using Xunit;

    [Collection(CollectionFixtureConstants.Integration)]
    public class PetsTest
    {
        private readonly WebApplicationFactoryFixture app;

        public PetsTest(WebApplicationFactoryFixture app)
        {
            this.app = app;
        }

        [Fact]
        [MockWebServerSeed("/Resources/ExternalApi/get-pets-expectation.json")]
        public async Task Test()
        {
            var response = await this.app.HttpClient.GetAsync("/api/v1/pets");

            var result = JsonConvert.DeserializeObject<List<PetResponse>>(await response.Content.ReadAsStringAsync());

            result?.Count.Should().Be(2);
        }
    }
}

namespace Company.Shorts.EndToEnd.Tests
{
    using Company.Shorts.Domain;
    using Company.Shorts.EndToEnd.Tests.Internal.Fixtures;
    using Company.Shorts.EndToEnd.Tests.Internal.Postgres;
    using FluentAssertions;
    using Newtonsoft.Json;
    using System.Collections.Generic;
    using System.Threading.Tasks;
    using Xunit;

    [Collection(CollectionFixtureConstants.Integration)]
    public class UsersTest
    {
        private readonly WebApplicationFactoryFixture app;

        public UsersTest(WebApplicationFactoryFixture app)
        {
            this.app = app;
        }

        [Fact]
        [PostgresSeed("/Resources/Users/get-users.json")]
        public async Task GetUsers_Should_ReturnTwoUsers()
        {
            var response = await this.app.HttpClient.GetAsync("/api/v1/users");

            var result = JsonConvert.DeserializeObject<List<User>>(await response.Content.ReadAsStringAsync());

            result?.Count.Should().Be(2);
        }
    }
}

There are also 2 attributes present, one for the database and the other for the external api. Essentially, they are doing the same thing. Before each test, insert something from a file, after each test, clean everything. Implementation can be found on github because at the end of a day I am a lazy bastard.

Conclusion

Is this end to end test or integration test? I have no idea. Maybe? At the end of the day, I don’t think it’s really important how we call it but what it does. And I think it does a good job in testing your application.

In this example, we had 2 different scenarios:

Testing database -> this one is pretty straightforward and it imitates real life system
Testing external api -> this one is essentially a mock. We are “trusting” documentation provided to us by some external provider (eg. Swagger docs). in order to make our tests stable. Of course, there are situation where documentation is not maintained, but should we really test that as well? I don’t think so. What I’m trying to say is, using this approach doesn’t guarantee that your implementation will work, but it gets pretty close.

Github