Float In Code

Debugging: Only building docker image extra npm packages are downloaded

2021-10-04T13:05:36+00:00

I'm working on the hybrid angular web application. We did update the angular version and released it to our QA environment, everything went fine until we tried to release it to our other QA environment which is using docker. Angular build failed with the following error, indicating that two type definition packages are exporting the same types.

In this blog post, I will walk through the thought process that helped me debug this problem.

Locally on our windows laptop build works fine, azure windows pipelines work fine, but docker builds fail everywhere.

#12 87.41
#12 87.41 ERROR in node_modules/@types/googlemaps/reference/map.d.ts:1:1 - error TS6200: Definitions of the following identifiers conflict with those in another file: BicyclingLayer, Map, HYBRID, ROADMAP, SATELLITE, TERRAIN, MapTypeRegistry, TrafficLayer, TransitLayer
#12 87.41
#12 87.41 1 declare namespace google.maps {
#12 87.41   ~~~~~~~
#12 87.41
#12 87.41   node_modules/@types/google.maps/index.d.ts:18:1
#12 87.41     18 declare namespace google.maps {

From the error above we can see that there are two packages @types/googlemaps and @types/google.maps which provide the same type of definitions.

Thought process

We upgraded the angular major version, this resulted in a large portion of our other packages being upgraded as well. The first thought was that some have a new dependency, but this doesn't explain why our windows build agents can build successfully. When building on windows @types/google.maps package is not present in node_modules directory.

After some searching in the project, installed packages and online, still nothing. The next step that came to my mind is to check the docker file system for this mysterious package. With a few commands, I had docker up and running with the project inside it.

 docker build . -t ng-debug-build
 
 docker run -it ng-debug-build /bin/ash

This showed that docker indeed had @types/google.maps installed. Can npm packages have dependencies based on the operating system? New google quest was born and search continued. The answer is - no.

Well if we don't have it on windows build, we don't have it in our package.json, it must be installed by someone. Using bisection search on dependencies and devDependencies I was able to find that @typesmarkerclusterer package was installing additional dependencies when building project in docker.

I'm still not sure why we don't have issues on our windows azure build agent, it might be that we have some sort of cache that I'm not aware of.

The issue was that @types/google-map package changed its dependency. Our angular application has @types/googlemaps but when this package changed its dependence conflicts arose. @types/markerclustererplus has wildcard major version dependency on @types/google-maps, and this situation is a perfect example of why you shouldn't do that!

Solution

Since we are using yarn we override internal package versions. We simply rolled back to @types/google-maps version that uses same @types/googlemaps dependency.

    "resolutions": {
        "@types/markerclustererplus/**/@types/google-maps": "3.2.1"
    }

This is a perfect example demonstrating how careful you should be about versions that you specify in dependencies. Overall I'm pretty happy with this bug that we had, it led to great examples of bad code, helped me learn new things, such as resolutions, and showed that problem solving has room for improvement, since some steps took a bit longer than anticipated.

Snap to the road: Google vs Mapbox vs Bing

2021-09-20T08:34:00+00:00

Recently did some research about snap-to-road functionality. In this blog post, I will try to share results and compare some edge cases where coordinates are missing or GPS noise is present.

Tech stack: Angular, leaflet with map box map provider.

Red - actual coordinates
Blue - Mapbox
Green - Google
Purple - Bing

Bing road interpolation works over longer road periods, but at the same time, it ignores visite to gas stations. Mapbox and Google acknowledge fact that there were few coordinates at the gas station

Mapbox, Google, and actual coordinates go as a straight line, however, bing interpolates large distance coordinate jump.

A strange example of how Google decided to stop interpolation and just draw straight lines for some reason. Bing is the winner for this scenario, perfectly snapped to the road.

Great example showing different interpolation algorithms between Mapbox and Google. Google makes a corner, while Bing just continues straight.

Goole tries to make a corner but gives up in the middle.

Mapbox only provides snap to closest road and this is clearly visible when we compare with Google and Bing snap to the road with interpolation.

Latitude,Longitude or Longitude,Latitude

This was a long-lasting question and now I know that there is no correct answer.

Latitude,Longitude - Google, Bing

Longitude,Latitude - Mapbox

Final thoughts

It won't be fair to compare accuracy, I'm drawing all roads on MapBox tiles.

Bing will try to keep going straight when interpolating, while Google will wiggle around and try to include all points in the interpolated route. This brings to the main difference, Bing is more likely to remove original points to make a route while Google is more likely to add points in between.

There is no winner, this research showed fundamental differences between these providers. Based on the situation all of them are valid options.

.NET Transaction scope

2021-09-06T03:30:00+00:00

This blog post was moved from the old blog.

One way to learn something new is to write unit tests exploring a new library, that is what I'll be doing in this blog post. For tests, we will use xUnit and for database Npgsql.

Let's explore different use cases of transaction scope. We will see how transactions behave when they are nested and how exceptions are handled on different nesting levels.

Since we are using async/await workflow we need to enable TransactionScopeAsyncFlowOption.Enabled on our transactions.

Enable transaction scope

Include Enlist=true in your connection string.
Call connection.EnlistTransaction(Transaction.Current);

Newer versions of Npgsql have enlisting enabled by default.

Setup functions and constants used throughout testing

private readonly string ConnectionString = "DATABASE=transaction_test;PORT=5432;USER ID=postgres;PASSWORD=postgres;HOST=localhost;";

private readonly string InsertTableAEntryQuery = @"INSERT INTO public.TestTableA(value) VALUES (@value);";
private readonly string InsertTableBEntryQuery = @"INSERT INTO public.TestTableB(value) VALUES (@value);";

private readonly string TableARowCountQuery = @"SELECT count(*) FROM public.TestTableA;";
private readonly string TableBRowCountQuery = @"SELECT count(*) FROM public.TestTableB;";

private async Task ClearDatabase()
{
    await using var connection = new NpgsqlConnection(this.ConnectionString);

    await connection.ExecuteAsync("DELETE FROM public.TestTableA");
    await connection.ExecuteAsync("DELETE FROM public.TestTableB");
}

private async Task<int> TableARowCount()
{
    await using var connection = new NpgsqlConnection(ConnectionString);
    return await connection.QuerySingleAsync<int>(TableARowCountQuery);
}

private async Task<int> TableBRowCount()
{
    await using var connection = new NpgsqlConnection(ConnectionString);
    return await connection.QuerySingleAsync<int>(TableBRowCountQuery);
}

private async Task ThrowException()
{
    await using var connection = new NpgsqlConnection(this.ConnectionString);
    using var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled);
    throw new Exception();
}

White smoke test to make sure we have everything setup correctly

[Fact]
public async Task ConnectionIsWorking()
{
    await ClearDatabase();
    await using var connection = new NpgsqlConnection(ConnectionString);

    // Note: no need to open connection
    // Dapper will open it for us

    await connection.ExecuteAsync(InsertTableAEntryQuery, new { value = 123 });
    await connection.ExecuteAsync(InsertTableBEntryQuery, new { value = 321 });

    Assert.Equal(1, await TableARowCount());
    Assert.Equal(1, await TableBRowCount());
}

Throwing exception before transaction complete should roll back transaction

[Fact]
public async Task SimpleTransactionRollbackOnException()
{
    await ClearDatabase();

    using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
    {
        try
        {
            await using var connection = new NpgsqlConnection(ConnectionString);

            await connection.ExecuteAsync(InsertTableAEntryQuery, new { value = 123 });

            await ThrowException();

            scope.Complete();
        }
        catch { }
    }

    Assert.Equal(0, await TableARowCount());
    Assert.Equal(0, await TableBRowCount());
}

Even if we catch exception, we still can't complete transaction since connection state is set to aborted

[Fact]
public async Task SimpleTransactionRollbackOnException2()
{
    await ClearDatabase();
    try
    {
        using var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled);
        await using var connection2 = new NpgsqlConnection(ConnectionString);

        try
        {
            await using var connection = new NpgsqlConnection(ConnectionString);
            await connection.ExecuteAsync(InsertTableAEntryQuery, new { value = 123 });

            await ThrowException();

        }
        catch { }

        scope.Complete();
    }
    catch (TransactionAbortedException)
    {
        Assert.True(true);
    }
    catch
    {
        Assert.True(false);
    }
}

Forgetting to call Complete will fail to save changes silently

[Fact]
public async Task SimpleTransactionRollbackWithoutComplete()
{
    await ClearDatabase();

    using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
    {
        await using var connection = new NpgsqlConnection(ConnectionString);

        await connection.ExecuteAsync(InsertTableAEntryQuery, new { value = 123 });
        await connection.ExecuteAsync(InsertTableBEntryQuery, new { value = 321 });

        //Forgot to call complete
        //scope.Complete();
    }

    Assert.Equal(0, await TableARowCount());
    Assert.Equal(0, await TableBRowCount());
}

Nesting transaction scopes and having multiple complete functions will work fine

[Fact]
public async Task NestedTransactionsBothHavingComplete()
{
    await ClearDatabase();

    using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
    {
        await using var connection = new NpgsqlConnection(ConnectionString);

        await connection.ExecuteAsync(InsertTableAEntryQuery, new { value = 123 });

        await NestedTransactionWithCompleteInside();

        scope.Complete();
    }

    Assert.Equal(1, await TableARowCount());
    Assert.Equal(1, await TableBRowCount());
}

private async Task NestedTransactionWithCompleteInside()
{
    using var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled);
    await using var connection = new NpgsqlConnection(ConnectionString);

    await connection.ExecuteAsync(InsertTableBEntryQuery, new { value = 321 });

    scope.Complete();
}

Root transaction will fail when nested transaction isn't completed

[Fact]
public async Task NestedTransactionsOnlyRootWithComplete()
{
    await ClearDatabase();

    try
    {
        using var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled);
        await using var connection = new NpgsqlConnection(ConnectionString);

        await connection.ExecuteAsync(InsertTableAEntryQuery, new { value = 123 });

        await NestedTransactionWithoutCompleteInside();

        scope.Complete();
    }
    catch (TransactionAbortedException)
    {
        Assert.True(true);
    }
    catch
    {
        Assert.True(false);
    }
}

private async Task NestedTransactionWithoutCompleteInside()
{
    using var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled);
    await using var connection = new NpgsqlConnection(ConnectionString);

    await connection.ExecuteAsync(InsertTableBEntryQuery, new { value = 321 });
}

Summary

If we nest transactions, all of them succeed or all of them will fail.
Don't forget to call transaction scope complete method.
Transaction complete method will fail if connection is in invalid state due to exception.

Custom API version selector

2021-08-23T03:08:00+00:00

In this blog post, I will demonstrate how to write a custom .NET CORE API version selector. While it's easy and libraries provided all the necessary information to do that, you should weigh the pros and cons before doing so. The implementation that I will share below is used in API which communicates with the web and two mobile apps (IOS, Android), while at first, this custom version selection seemed a great idea, after some time it looks more like a ticking time bomb.

Source code at GitHub

Implementation

Install Microsoft.AspNetCore.Mvc.Versioning nuget package.

        [HttpGet]
        [ApiVersion("1.0")]
        public WeatherResponse Get()
        {
            var rng = new Random();
            var respones = new WeatherResponse
            {
                Forecasts = Enumerable.Range(1, 5).Select(index => new WeatherForecast
                {
                    Date = DateTime.Now.AddDays(index),
                    TemperatureC = rng.Next(-20, 55),
                    Summary = Summaries[rng.Next(Summaries.Length)]
                }),
                ApiVersion = "1.0"
            };

            return respones;
        }

        [HttpGet]
        [ApiVersion("2.0")]
        public WeatherResponse Get_v2()
        {
            var rng = new Random();
            var respones = new WeatherResponse
            {
                Forecasts = Enumerable.Range(1, 5).Select(index => new WeatherForecast
                {
                    Date = DateTime.Now.AddDays(index),
                    TemperatureC = rng.Next(-20, 55),
                    Summary = Summaries[rng.Next(Summaries.Length)]
                }),
                ApiVersion = "2.0"
            };

            return respones;
        }

services.AddApiVersioning(options =>
{
    options.ReportApiVersions = true;
    options.AssumeDefaultVersionWhenUnspecified = true;
    //Important to version reader ALWAYS report version as missing
    options.ApiVersionReader = new HeaderApiVersionReader("custom-version-selection-works-only-if-api-version-is-not-specified");

    options.ApiVersionSelector = new HighestUpToSpecifiedIncludedVersionSelector(new ApiVersion(2, 0));
});

ApiVersion has extension to parse version object from string, SelectVersion method has all the information required to decide which method to select. What's left is for us to make decision and implement our custom version selector.

    public class HighestUpToSpecifiedIncludedVersionSelector : IApiVersionSelector
    {
        private readonly ApiVersion _defaultVersion;

        public HighestUpToSpecifiedIncludedVersionSelector(ApiVersion defaultVersion)
        {
            _defaultVersion = defaultVersion;
        }

        public ApiVersion SelectVersion(HttpRequest request, ApiVersionModel model)
        {
            if (model == null)
                throw new ArgumentNullException(nameof(model));

            var requestedVersion = ParseVersion(request) ?? _defaultVersion;

            var selectedVersion = model.ImplementedApiVersions.Count switch
            {
                0 => _defaultVersion,
                1 => model.ImplementedApiVersions[0].Status == null ? model.ImplementedApiVersions[0] : _defaultVersion,
                _ => model.ImplementedApiVersions
                   .Where(v => v.Status == null && v <= requestedVersion)
                   .Max(v => v) ?? _defaultVersion
            };

            return selectedVersion;
        }

        private static ApiVersion ParseVersion(HttpRequest request)
        {
            if (!request.Headers.TryGetValue("api-version", out var version))
                return null;

            if (string.IsNullOrEmpty(version) || version.Count != 1)
                return null;

            return ApiVersion.TryParse(version[0], out var apiVersion) ? apiVersion : null;
        }
    }

Sending request without version will execute default version endpoint, which in this case is 2.0

Specifying version that doesn't exist we get smallest up to that version, in this case it's 1.0

Summary

We used this approach in mobile application. Would I use it again ? No. As of right now we don't make any breaking changes to the API, only small bug fixes. With this versioning style, if we already have v2.0 endpoint and mobile app is using it, we can't create new v2.0 endpoint for other controller, if we do, mobile app will start to use it immediately and will break.

It's good to know that you can change version selection, but it's best to fail if endpoint or version is not found.

Caprover: best way to deploy .NET and Docker

2021-08-09T02:36:00+00:00

Caprover quickly became my favorite server management tool. If you are a solo developer or working in a smaller team, you should definitely give it a try.

What's included:

Docker registry - but you can easily connect to any other registry
Docker build engine - just push your code and Caprover will build and deploy it
Webhooks - integrate with any Git repository and build\deploy on push
Server metrics
Lets Encrypt certificates - Caprover will issue and renew certificates for you
Custom nginx configs - you are not locked in doing things in only one way, if you need web sockets or extra http headers - you can modify configuration
One-click-apps - select app from list, enter required variables (name, version ...) and deploy. Similar to docker-compose where you can run multiple services from single configuration. Also it's easy to create your own one-click-app configurations

What's missing

Logging - to view nginx or other service logs you have to SSH to server
No build steps - you can't run tests or do anything else, just build and push to registry

I have been using Caprover for a half year now and moved all my services to it, so right now I have +10 services running, including web apps, databases, WordPress, and more.

Caprover is too simple to write about and its documentation covers all the details, but at the same time Caprover is too important not to mention it.

To find more visit Caprover.com

Mapster: generate dto, async after map actions, dependency injection

2021-07-26T02:35:00+00:00

Mapster brings some new ideas to object mapping world. Model generation, async custom actions after map, dependency injection. In this post, I will try to explore these new ideas. Source code can be found on GitHub.

Getting started

In your solution folder run the following commands to enable dotnet tools and install mapster code generator.

dotnet new tool-manifest

Enabling tool manifest in the solution directory will create a local configuration file, which will help other team members to install the same tools for the project.

dotnet tool install Mapster.Tool

Mapster provides several ways to generate mapping files, for this example, let's generate automatically after build.

    <ItemGroup>
        <Generated Include="**\*.g.cs" />
    </ItemGroup>
    <Target Name="CleanGenerated">
        <Delete Files="@(Generated)" />
    </Target>

<Target Name="Mapster" AfterTargets="AfterBuild">
    <Exec WorkingDirectory="$(ProjectDir)" Command="dotnet tool restore" />
    <Exec WorkingDirectory="$(ProjectDir)" Command="dotnet mapster model -a &quot;$(TargetDir)$(ProjectName).dll&quot;" />
    <Exec WorkingDirectory="$(ProjectDir)" Command="dotnet mapster extension -a &quot;$(TargetDir)$(ProjectName).dll&quot;" />
    <Exec WorkingDirectory="$(ProjectDir)" Command="dotnet mapster mapper -a &quot;$(TargetDir)$(ProjectName).dll&quot;" />
</Target>

One thing to keep in mind, code generation is done after build. If you change mapster configs and start the project, nothing will change. Correct flow is build project and then start it.

    public class MapsterConfig : ICodeGenerationRegister
    {
        public void Register(CodeGenerationConfig config)
        {
            config.AdaptTo("[name]Dto")
                .ForType<WeatherForecast>();
        }
    }

Advanced model generation

public class MapsterConfig : ICodeGenerationRegister
{
    public void Register(CodeGenerationConfig config)
    {
        config.AdaptTo("[name]Dto")
            .ForType<WeatherForecast>(
            cfg =>
            {
                cfg.Ignore(poco => poco.TemperatureC);
                cfg.Map(poco => poco.Summary, "MappedSummary");
            });
    }
}

The model can be modified by ignoring, renaming, changing the type of the property. When you change the property name, you MUST map that property manually.

TypeAdapterConfig<WeatherForecast, WeatherForecastDto>
    .NewConfig()
    .Map(dest => dest.MappedSummary, src => src.Summary);

Type safety

RequireExplicitMapping will make sure that all mappings should be registered. RequireDestinationMemberSource checks if all destination properties have mappings. .Compile() will perform all checks at compile time and throw exceptions if there are any issues.

TypeAdapterConfig.GlobalSettings.RequireExplicitMapping = true;
TypeAdapterConfig.GlobalSettings.RequireDestinationMemberSource = true;
TypeAdapterConfig.GlobalSettings.Compile();

After map actions

AfterMapping extension allows you to execute code after mapping is done and do final modifications to result if needed.

TypeAdapterConfig<WeatherForecast, WeatherForecastDto>
    .NewConfig()
    .Map(dest => dest.MappedSummary, src => src.Summary)
    .AfterMapping((dest, d) => { d.MappedSummary = "AfterMapping " + d.MappedSummary; });

Dependency Injection and async after map actions

For dependency injection to work we need to change how mapper is used. Configuration and mapping must be switched from static instances to services. First, we must configure required services by adding singleton for all mapping configurations and scoped mapping service.

public void ConfigureServices(IServiceCollection services)
{
    //configure mapster
    var config = new TypeAdapterConfig();
    services.AddSingleton(config);
    services.AddScoped<IMapper, ServiceMapper>();

    TypeAdapterConfig.GlobalSettings.RequireExplicitMapping = true;
    TypeAdapterConfig.GlobalSettings.RequireDestinationMemberSource = true;
    TypeAdapterConfig.GlobalSettings.Compile();
    //-------
}

For testing purposes, I created three services all of them return time when they were created.

public void Configure(IApplicationBuilder app, IWebHostEnvironment env, TypeAdapterConfig config)
{
    //Mapping configurations
    config.NewConfig<WeatherForecast, WeatherForecastDto>()
            .Map(dest => dest.MappedSummary, src => src.Summary)
            .AfterMappingAsync(async (dest) =>
            {
                Thread.Sleep(1000);
                var transient = await MapContext.Current.GetService<TransientService>().DateAsync();
                var scoped = await MapContext.Current.GetService<ScopedService>().DateAsync();
                var singleton = await MapContext.Current.GetService<SingletonService>().DateAsync();

                dest.MappedSummary = $"{transient}, {scoped}, {singleton}";
            });
    //--------

Mapping function itself also change to async version.

private readonly IMapper _mapper;

public WeatherForecastController(IMapper mapper)
{
    _mapper = mapper;
}

[HttpGet]
public async Task<IEnumerable<WeatherForecastDto>> Get()
{
    var result = new List<WeatherForecastDto>();

    foreach (var forecast in Forecasts())
    {
        result.Add(await _mapper.From(forecast)
            .AdaptToTypeAsync<WeatherForecastDto>());
    }

    return result;
}

Now that we have all the required services, we can do a simple test. Execute endpoint twice and compare dates to see if service instances were created as expected.

First-run results.

Second run results.

Transient service results change for each temperature value, scoped service time is the same for request and singleton always shows the same date. From this simple test we see that DI and service scope in mapping works as we expect.

Summary

Mapster brings really great additions to otherwise mundane mapping. Code generation, after map actions, dependency injection are things that more and more often come to mind while working on a large legacy project, and from the initial look, Mapster handles these cases pretty well.

There are still some questions unanswered: how does it handle complex objects? testing I saw in github that some people had issues with concurrency. Code maintainability, since part of logic, would be moved to mapping.

I'm looking forward to using this library in my next project.

Pro Windows Subsystem for Linux (WSL) book “review”

2021-07-12T14:16:00+00:00

Pro Windows Subsystem for Linux (WSL)

Review is in quotation marks because this is not a full review, this is a summary and tips of what I learned from this book. When reading it my main focus was to improve my Kubernetes, terraform script workflow. Which involve windows file/directory navigation, editing files on WSL, managing WSL distributions.

Default parameters

First lets identify which distributions are installed, which are default and make sure we starting them with correct user.

wsl.exe --set-default-version 2

wsl.exe --list --verbose --all

 NAME                   STATE           VERSION
* Ubuntu                 Running         2
  docker-desktop-data    Running         2
  docker-desktop         Running         2

Name - <wsl distro name> in following commands

wsl.exe --set-default <wsl distro name>

wsl.exe --set-version <wsl distro name> 2

<wsl distro name> config --default-user <new user name>

File permissions

Even though file permissions doesn't really fit into what I was looking in this book, wanted to write this down for future reference:

Three groups (<user><group><other>):

User - by default user who created
Group - users in a group with assigned access to files
Other - all other users

Symbolic form:

r - Read
w - Write
x - Execute
- (just dash) for no permission

Example: rwx-r-xr-

rwx - for user
-r- -for group
xr- - for other

Numeric form

0 - No permissions, ---
1 - Execute only, --x
2 - Write only, -w-
3 - Write and execute, -wx
4 - Read, r--
5 - Read and execute, r-x
6 - Read and write, rw-
7 - Read, write and execute, rwx

Previous example can be written: 745

Convert between Linux and Windows paths

One of the biggest issue for me right now are navigating windows directories in Linux and vice versa.

wslpath -w /mnt/c/Users/sarun

wslpath "C:\Users\sarun"

explorer.exe .

It's also possible to browse all WSL distribution files using File Explorer, navigate to: \\wsl$

Windows X server

Install x server

configuration:

https://stackoverflow.com/a/61110604

WSL distribution settings

sudo apt update
sudo apt install xfce4 xfce4-goodies

Append following lines to ~/.bashrc:

export DISPLAY=$(awk '/nameserver / {print $2; exit}' /etc/resolv.conf 2>/dev/null):0
export LIBGL_ALWAYS_INDIRECT=1

Reboot WSL distribution.

Windows settings

Install X Server: VcXsrv Windows X Server download | SourceForge.net

Change VcXsrv launcher DPI settings.

Start xLaunch.exe and select display settings, most of them are optional and you can choose what you like. "Disable access control" is required, without it WSL distribution can't communicate with host.

Display settings, I found that multiple windows works best for me.

When new WSL distribution connects, you can execute or start some default applications.

Disable access control must be enabled.

Running applications

Now that we have everything set up we can run some applications.

You can run individual applications, like xCalc

Run individual applications.

Or you can start full desktop experience by running startxfce4

Full desktop experience in WSL 2.

Please note that not all applications will start, some of them might have issues or crash.

Summary

Considering my initial goal and what this book provided, I'm really happy and would recommend this book. I took only small portion what it provides and will definitely come back to it when my needs increase.

ASP.NET CORE Required property validation

2021-06-26T10:07:58+00:00

System.ComponentModel.DataAnnotations.Required validation doesn't work how most people imagine it working. Required - it does have value, if the property is nullable and value is null - this validation will fail. However, if we have value type property - then we always have value, and required validation never fails.

Model binder creates an empty model and assigns values from JSON. If a JSON value is missing for enum or integer - we will get default value, and sometimes this is not desired behavior. We can approach this problem from two sides, we can make value types nullable or we can use JSON validation and fail inside the model binder.

Lets explore these two validation types by creating small project, start with empty ASP.NET core API project.

Validation with System.ComponentModel.DataAnnotations

using System.ComponentModel.DataAnnotations;

namespace RequiredValidation
{
    public class WeatherModel
    {
        [Required]
        public int Temperature { get; set; }

        [Required]
        public string Title { get; set; }
    }
}

Validation with NewtonsoftJson

 Microsoft.AspNetCore.Mvc.NewtonsoftJson

    public void ConfigureServices(IServiceCollection services)
    {

        services
            .AddControllers()
            .AddNewtonsoftJson();

using Newtonsoft.Json;

namespace RequiredValidation
{
    public class Weather2Model
    {
        [JsonProperty(Required = Required.Always)]
        public int Temperature { get; set; }

        [JsonProperty(Required = Required.Always)]
        public string Title { get; set; }
    }
}

Endpoints

Controller endpoints in both cases are identical, no additional changes are needed. To make testing easier I just return same model.

        [HttpPost]
        [Route("Update")]
        public IActionResult UpdateWeather(WeatherModel weather)
        {
            return Ok(weather);
        }

        [HttpPost]
        [Route("Update2")]
        public IActionResult UpdateWeather2(Weather2Model weather)
        {
            return Ok(weather);
        }

Testing DataAnnotations validation

Lets start by testing DataAnnotations validation "Update" endpoint.

If we ignore value type property, it will get default value.

If we ignore string property with required attribute, we get error message, as expected.

Testing JSON property validation

We get error message when integer is not present in request.

Same as above, when string is not present in request we get error message.

Summary

Back to C# basics, value types always have a default value - unless they are nullable. Adding required on value type property doesn't make sense since it will be with default value either it's zero or first enum value.

ASP.NET CORE Integration tests: Entity framework DB first

2021-05-10T03:10:52+00:00

In this short blog post I want to show how useful EF DB first approach is when testing. While working with legacy project you might now have endpoints to set all data in DB for test, also validating exact data in DB get easier.

Generate code from the database

Install two Nuget packages in a project that will contain DB files: Microsoft.EntityFrameworkCore.Design and Npgsql.EntityFrameworkCore.PostgreSQL don't forget to rebuild the solution.

Next install dotnet-ef global tool and invoke it with following configuration. Tool has variety of configuration parameters, but for the sake of simplicity we will use only minimal set.

dotnet ef dbcontext scaffold "Server=127.0.0.1;Port=6444;Database=weather;User Id=admin_user;Password=secretPassword123!;" Npgsql.EntityFrameworkCore.PostgreSQL --output-dir Weather --no-onconfiguring --force

At this point we should have all necessary files to change dapper into EF.

Only a few changes are needed, create DataAccess inject database string, and use an entity framework in your project or tests. Source code is always available at GitHub

In memory

Why ? If someone is using EF code first on enterprise, please let me know how it's going.

Summary

I find Entity Framework is really helpful in tests, you can easily add data to DB and validate if tests performed correct changes to DB. When the database changes, you only need to run generate.cmd script to update all code and if there are issues, they are in compile-time, which makes maintenance and updates easy.

ASP.NET CORE Integration tests: Database with API

2021-04-26T01:56:54+00:00

Now that we have database migrations and API integration tests set up we can combine them together. In this post, I will go through some tips and tools that proved to be really helpful. Source code.

Double-check that you are using test DB

The first thing that you should be doing is make sure you are running tests in test environment. Usually when working we switch our connection strings to our QA or even one of our production databases, the least that we want is to corrupt these databases.

This check will vary from project to project, I usually check client or user count in DB, for tests, there shouldn't be that many, at least for our tests.

var maxAllowedEntries = 100;

using var conn = new NpgsqlConnection(settings.Value.ConnectionString);

var entries = await conn.ExecuteScalarAsync<int>("select count(*) from weather_predictions.predictions");

return entries < maxAllowedEntries;

Disable test concurrency

We have one test DB, logically we can only run one test at a time. This can be controlled with collections.

[Collection(nameof(NotThreadSafeResourceCollection))]
public class WeatherForecastControllerDatabaseTests
{

[CollectionDefinition(nameof(NotThreadSafeResourceCollection), DisableParallelization = true)]
public class NotThreadSafeResourceCollection
{
}

Clear database after each test with Respawn

It is possible to restart docker and rebuild the entire database after each test but that would take a lot of time. Respawn NuGet allows us to clear the database, either full or partial with initialization scripts.

var checkpoint = new Checkpoint
{
    TablesToIgnore = new string[] { },
    SchemasToInclude = new string[] { },
    DbAdapter = DbAdapter.Postgres
};
using var conn = new NpgsqlConnection(settings.Value.ConnectionString);
await conn.OpenAsync();

await checkpoint.Reset(conn);

Comparing objects

For object comparison, I highly recommend using Compare-Net-Objects, this NuGet will cover all the use cases that you need.

//When we ignore collection order we must specify key which will be used to match actual with expected
comparer.Config.IgnoreCollectionOrder = true;
comparer.Config.CollectionMatchingSpec.Add(typeof(WeatherForecast), new List<string> { nameof(WeatherForecast.Date) });

//Summary will be random, we don't really care about in this test. Check if it's not null.
comparer.Config.CustomPropertyComparer<WeatherForecast>(obj => obj.Summary,
    new CustomComparer<string, string>((expected, actual) => !string.IsNullOrEmpty(actual)));

Summary

In order to run tests fluently some gluing is needed. The first thing should be security, make sure you are using test DB and not production or another shared database. Since only one test can be running at any given time - make sure to configure concurrency, otherwise, you will have to debug a lot of random failing tests due to multiple tests writing data to DB. Lastly comparing objects should be easy, instead of writing tons of helpers invest some time and look into Compare-Net-Objects it's highly customizable and will cover all your cases.

	Google	MapBox	Bing
Price	10 USD per 1000 requests	2 USD per 1000 requests	Unable to find, need to contact with exact quote.
Snap-to-road	Great	Doesn't have	Best
Road selection	Tries to make much more turns and include all original points in the route.	The closest road can be configured with max distance from the point (5-50meters).	Will try to interpolate a straight road and will ignore some coordinates to do so.