Mocking Entity Framework DbContext with Moq

When we have to test methods that involves Entity Framework, a typical choice that we have to face is use integration tests, with an effective database, or unit tests.

If we choice the first option, with a database like SQL LocalDB, we’ll have performance problems because the cost of the database creation and the data inserts in the test startup is very high, and in order to guarantee the initial conditions we’ll have to do it for each test.

What we can do is use a mock framework that help us to mockup the entity framework context; it would be an in-memory db context, like the in-memory db context of .NET Core, that we have seen in this post.

The factory

In pratice, mocking a class means substitute the real implementation of a method with our custom behaviour; what we can do for every method of the class is setup returns values of the method; therefore we don’t need the real implementation of the class, we have mocked the methods.

In our case, we can setup the EF DbSet with an in-memory list and the methods that use the context will no longer need the real database, they will use our lists provided with the mock.

Anyway, before to do it we have another problem; how we can provide our in-memory db context to the methods that we need to test?

In the real world, it’s likely that the context will be instantiated with the using statement in every method, like this:


public async Task<List<Person>> GetPersons(string query)
{
using (var db = new Context())
{
.....
}
}

This is a problem, because we are not able to inject our mocked context in the class.

But we can solve it with a Factory service, a singleton service that returns instances of the db context:


public class ContextFactory
{
private Type _dbContextType;
private DbContext _dbContext;

public void Register<TDbContext>(TDbContext dbContext) where TDbContext : DbContext, new()
{
_dbContextType = typeof(TDbContext);
_dbContext = dbContext;
}

public TDbContext Get<TDbContext>() where TDbContext : DbContext, new()
{
if (_dbContext == null || _dbContextType != typeof(TDbContext))
{
return new TDbContext();
}

return (TDbContext)_dbContext;
}
}

We have two methods, with the Register method we can setup a specific db context implementation; with the Get method we can get an instance of a db context, that is the registered implementation if we have one, otherwise the default implementation.

We can now inject this service as a dependency and use it:


public class PersonService
{
private readonly ContextFactory _contextFactory;

public PersonService(ContextFactory contextFactory)
{
_contextFactory = contextFactory;
}

public async Task<List<Person>> GetPersons(string query)
{
using (var db = _contextFactory.Get<Context>())
{
return await db.Persons.Where(p => p.TaxCode.Contains(query) || p.Firstname.Contains(query) || p.Surname.Contains(query)).ToListAsync();
}
}
}

Now we are ready to mock the EF context.

The mock

The framework that I use for this purphose is moq and I can install it with nuget:

install-package moq

It’s likely that you use async methods of entity framework; if yes, in order to mock we need to create an in-memory DbAsyncQueryProvider, and you can find the implementation here.

The Unit Testing used for this example is NUnit and I can configure the mocked context in the setup method; the first step is prepare a list of queryable objects:


[SetUp]
public void Setup()
{
var persons = new List<Person>() {
new Person() { TaxCode = "taxcode1", Firstname = "firstname1", Surname = "surname1" },
new Person() { TaxCode = "taxcode2", Firstname = "firstname2", Surname = "surname2" }
};
var queryable = persons.AsQueryable();
}

Now I’m ready to setup the mock:


MockSet = new Mock<DbSet<Person>>();

MockSet.As<IQueryable<Person>>().Setup(m => m.Expression).Returns(queryable.Expression);
MockSet.As<IQueryable<Person>>().Setup(m => m.ElementType).Returns(queryable.ElementType);
MockSet.As<IQueryable<Person>>().Setup(m => m.GetEnumerator()).Returns(queryable.GetEnumerator);

MockSet.As<IQueryable<Person>>().Setup(m => m.Provider).Returns(new AsyncQueryProvider<Person>(queryable.Provider));
MockSet.As<IDbAsyncEnumerable<Person>>().Setup(m => m.GetAsyncEnumerator()).Returns(new AsyncEnumerator<Person>(queryable.GetEnumerator()));

In order to mock an IQueryable, I have to setup returns values for Expression method, ElementType and GetEnumerator; every time these methods will be invoked in the queries executions, the values that I setup in the Returns expression will be returned.

I need to do the same operations for Provider method and GetAsyncEnumerator, but, since async methods are involved, I need to use the custom classes AsyncQueryProvider and AsyncEnumerator of the in-memory DbAsyncQueryProvider.

The mock for the Add and Remove operations are simplier:


MockSet.Setup(m => m.Add(It.IsAny<Person>())).Callback((Person person) => persons.Add(person));
MockSet.Setup(m => m.Remove(It.IsAny<Person>())).Callback((Person person) => persons.Remove(person));

Since Add and Remove methods returns nothing, we use Callback methods instead of Returns.

The last step is setup the factory service context with the mocked version:


MockContext = new Mock<Context>();
MockContext.Setup(m => m.Persons).Returns(MockSet.Object);

var contextFactory = new ContextFactory();
contextFactory.Register(MockContext.Object);
PersonService = new PersonService(contextFactory);

First of all I setup the DbSet of the mocked context with the mocked DbSet.

Then I register the mocked context in the factory service and then I pass the factory service as a dependency of the service to be test.

With these lines of code I have mocked the entity framework context with an in-memory instance and leveraging the context factory I was able to inject the mocked context to the service.

You can find the source code here.

 

Advertisements
Mocking Entity Framework DbContext with Moq

Consume Web API with an Angularjs service

This topic concerns a problem that I faced in these days, that was the building of an Angularjs service to consume Web API’s, witch in my case were implemented with OData protocol.

The start point are some Web API’s, one for every entity of the project, that they responds to a specific url, for example:

I wouldn’t want to implement a service for every single Web API that do the same operations, so the idea is implement a generic javascript class with the basic methods and a factory that generates instances of this class; every module will have an own instance with some custom options, such as the service url.

So what I want to do are two things, a generic class that implements commons methods for CRUD operations and then a factory that generates instances of this service.

Generic service

Angularjs has several provider that we can use to instantiate services; the provider is the lower level and is use usually to configure reusable services that can be used in different applications.

Factory and services are syntactic sugar on top of a provider, and allow us to define singleton services in our application; the last two are constant and value that they provides values.

One of the main debates on Angularjs is when we use a factory or a service.

The main difference from these is that in the first one we needs to return an instance of the service in this way:


(function (window, angular) {
'use-strict'
angular.module('odataModule', [])
.factory('odataService', [function() {
returns {
.....
}
}]);
})(window, window.angular)

In the second one, we don’t need to do this, the angular service deal with object instantiation:


(function (window, angular) {
'use-strict'
angular.module('odataModule', [])
.service('odataService', [function() {
.....
}]);
})(window, window.angular)

What I want to implement is a javacript class that will have these properties/methods:

  • A url property, the Web API reference
  • An entityClass property, is the the javascript object that represents the entity to be managed
  • A isNew property, that allow the service to understand that the object need to be added (post) or saved (patch)
  • A getEntities method that retrieves the list of the entities
  • A createEntity method that returns a instance of a new entity
  • A getEntity method to retrieves a single entity
  • A saveEntity method that deal with the post/patch of the entity
  • And finally a method deleteEntity

An angular service help in this work by letting me write cleaner code that factory; a concrete implementation is:


(function (window, angular) {
'use-strict'
angular.module('odataModule', [])
.constant('blogsUrl', '/odata/Blogs')
.constant('postsUrl', '/odata/Posts')
.service('odataService', ['$http', function($http) {
function Instance(url, entityObject) {
this.isNew = false;
this.url = url;
this.entityClass = entityObject;
}

Instance.prototype.getEntities = function () {
return $http.get(this.url);
}

Instance.prototype.createEntity = function() {
this.isNew = true;
return angular.copy(this.entityClass);
}

Instance.prototype.getEntity = function (id) {
return $http.get(this.url + '(guid\'' + id + '\')');
}

Instance.prototype.saveEntity = function (entity) {
if (this.isNew) {
this.isNew = false;
return $http.post(this.url, entity);
}
else
return $http.patch(this.url + '(guid\'' + entity.Id + '\')', entity);
}

Instance.prototype.deleteEntity = function (entity) {
return $http.delete(this.url + '(guid\'' + entity.Id + '\')');
}
this.GetInstance = function(url, entityObject) {
return new Instance(url, entityObject);
};
}]);
})(window, window.angular)

The Instance class is the implementation of the common methods that we needed.

The last row is the main difference from a factory; in a service, the container function is a javascript function with a constructor and the service returns an instance of it, so I can define a method GetInstance that will returns an instance of my custom class.

If I had used a factory, the last rows would be the following:


return {
GetInstance: function(url, entityObject) {
return new Instance(url, entityObject);
}

Much better the implementation with the service.

Factories of the entities

Now I need to define a specific factory for every entity:


(function (window, angular) {
'use-scrict';
angular.module('blogsModule', ['ui.router', 'odataModule'])
.factory('blogsFactory', ['odataService', 'blogsUrl', function (odataService, blogsUrl) {
var blogEntity = { Name: '', Url: '' };
return odataService.GetInstance(blogsUrl, blogEntity);
}])
})(window, window.angular)

So far so good, I have an instance of the service with specific parameters, and now I can use the factory for crud operations in my controller:


(function (window, angular) {
'use-scrict';
angular.module('blogsModule', ['ui.router', 'odataModule'])
....
.controller('blogsController', ['$scope', '$state', 'blogsFactory', function ($scope, $state, blogsFactory) {
$scope.Title = 'Blogs';

blogsFactory.getEntities().then(function (result) {
$scope.blogs = result.data.value;
});

$scope.create = function() {
$state.go('main.blog', { id: null });
};

$scope.edit = function (blog) {
$state.go('main.blog', { id: blog.Id });
};
}])
.controller('blogController', ['$scope', '$state', '$stateParams', 'odataService', function ($scope, $state, $stateParams, odataService) {
$scope.Title = 'Blog';
var service = new odataService.GetInstance('/odata/Blogs', { Name: '', Url: '' });

$scope.save = function () {
service.saveEntity($scope.blog).then(function () {
$state.go('main.blogs');
});
};

$scope.delete = function() {
service.deleteEntity($scope.blog).then(function () {
$state.go('main.blogs');
});
};

$scope.close = function() {
$state.go('main.blogs');
};

if ($stateParams.id === '')
$scope.blog = service.createEntity();
else {
odataService.getEntity($stateParams.id).then(function (result) {
$scope.blog = result.data;
});
}
}]);
})(window, window.angular)

With few lines of code I have managed the crud operations.

In my case the odata module can be improved with checks about the entity validations, by using the $metadata informations returned by the odata services and I can retrieve also the entity fields without passing those as parameters, but, anyway, this is a good starting point.

 

 

 

 

Consume Web API with an Angularjs service

Web API in ASP.NET Core

In this post I’ll talk about the Web API in ASP.NET Core and how we can implement a Restful service with this new framework.

One of the main difference in ASP.NET Core is that Web API have an unified class with the classic MVC Controllers.

So the declaration of a Web API in ASP.NET Core is look like this:


public class PostController : Controller
{
}

The attribute of the actions verbs is pretty similar to MVC 5.

We can define the controller route as well and we can do that with the Route attribute:


[Route("api/[controller]")]
public class PostController : Controller
{
}

We can implement an action in the same way as the Web API 2:


[HttpGet("{offset}", Name = "GetPost")]
 public IActionResult Get(int offset)
 {
 var result = _postService.GetPosts(offset, true);

return Ok(result);
 }

With the HttpGet attribute we can specify the verb of the method, the parameters and the action url.

So far so good, now there is the last question about this topic, that is the content negotiation.

Since ASP.NET Core MVC and Web API frameworks are unified, with the attribute Produces we can define the response content type for the controllers.

So, if we want to return a content type in json format, we need to apply this attribute:


[Produces("application/json")]
[Route("api/[controller]")]
public class PostController : Controller
{
}

With this attribute the content type of the request will be application/json, otherwise the controller will returns a response 415 – Unsupported media type.

The source code of the topic is here.

 

Web API in ASP.NET Core

Unit Testing in .NET Core with xUnit

In this post we see how to do unit testing in .NET Core with xUnit.

xUnit is a popular framework, with a lot of features and helpers to develop quickly our unit testings and implement them with a clean syntax; it’s very efficient and the result is that the tests execution is very fast.

In the previous post we implemented a  basic class with an in-memory DbContext, that we’ll use in the unit test.

The service

We can suppose that our sut is a service, like CategoryService.

The service has some methods to retrieve the category from the database with Entity Framework.

The service look like this:


public class CategoryService
{
readonly BlogContextFactory _contextFactory;

public CategoryService(BlogContextFactory contextFactory)
{
_contextFactory = contextFactory;
}

public IEnumerable<Category> GetCategories()
{
using (var db = _contextFactory.GetNewDbContext())
{
return db.Categories.OrderBy(c => c.Name).ToList();
}
}

public async Task<Category> AddCategory(Category category)
{
using (var db = _contextFactory.GetNewDbContext())
{
category.Id = Guid.NewGuid();

await db.Categories.AddAsync(category);
await db.SaveChangesAsync();

return category;
}
}

public async Task UpdateCategory(Category category)
{
using (var db = _contextFactory.GetNewDbContext())
{
db.Categories.Update(category);
await db.SaveChangesAsync();
}
}

public async Task DeleteCategory(Category category)
{
using (var db = _contextFactory.GetNewDbContext())
{
db.Categories.Remove(category);
await db.SaveChangesAsync();
}
}
}

Unit Test class

Now we can implement the Unit Test class; we can use the base class DatabaseFixture and pass the context factory configured with an in-memory DbContext to the service:


public class CategoryServiceTests : DatabaseFixture
{
readonly CategoryService _categoryService;

public CategoryServiceTests() : base("CategoryContext")
{
_categoryService = new CategoryService(ContextFactory);
}

[Fact]
public void should_return_categories_list()
{
var categories = _categoryService.GetCategories();
categories.Should().NotBeNullOrEmpty();
categories.Count().Should().BeGreaterThan(0);
}

[Fact]
public async Task should_add_new_category()
{
var category = new Category()
{
Id = new Guid("{1225FE5B-9C46-4BA3-9233-9337DDC5F478}"),
Name = "Test Category"
};

category = await _categoryService.AddCategory(category);

using (var db = ContextFactory.GetNewDbContext())
{
var newCategory = await db.FindAsync<Category>(category.Id);
newCategory.Should().NotBeNull();
}
}

[Fact]
public async Task should_edit_category()
{
using (var db = ContextFactory.GetNewDbContext())
{
var category = await db.FindAsync<Category>(new Guid("{BE7EB1E1-FB67-4FE1-A96E-4721D37AFE29}"));
category.Name = "category modified";
await _categoryService.UpdateCategory(category);

category = await db.FindAsync<Category>(new Guid("{BE7EB1E1-FB67-4FE1-A96E-4721D37AFE29}"));
category.Name.ShouldBeEquivalentTo("category modified");
}
}

[Fact]
public async Task should_delete_category()
{
var category = new Category() { Id = new Guid("{5D134CF2-EF37-4663-B3D2-64F2E3DBF3BE}"), Name = "Category1" };
await _categoryService.DeleteCategory(category);

using (var db = ContextFactory.GetNewDbContext())
{
category = await db.FindAsync<Category>(new Guid("{5D134CF2-EF37-4663-B3D2-64F2E3DBF3BE}"));
category.Should().BeNull();
}
}
}

The Fact attribute is the xUnit attribute used for the test methods.

As discussed above the ContextFactory is configured to returns an instance in-memory of the DbContext.

Now we can run the unit test by opening a command console, going to the UnitTest project folder and running the command:

dotnet test

The command will run the tests and will show us the results.

You can find the source code here.

 

Unit Testing in .NET Core with xUnit

In-memory DbContext with Entity Framework Core

Frequently, when we need to test a EF context from our unit test class, we need to implement a stub of the context manually, in order to have, for example, an in memory representation of the context.

In entity framework core this is useless because one of the options available to initialize a context is to use an in memory database.

With this option we can populate an in memory database that could be used, for test purposes.

Let’s go to take a look about the configurations that we need to do.

Context

The first thing is define a DbContext.

With Entity Framework Core, the things are quite different because the DbContext class has a new constructor that accept a parameter of type DbContextOptions; this parameter will be used to define for example the connection string of the context.

public class BlogContext : DbContext
{
public DbSet<Category> Categories { get; set; }
public DbSet<Tag> Tags { get; set; }
public DbSet<Post> Posts { get; set; }
public DbSet<Reply> Replies { get; set; }
public DbSet<PostCategory> PostCategories { get; set; }
public DbSet<PostTag> PostTags { get; set; }
public BlogContext(DbContextOptions<BlogContext> options) : base(options) { }
}

Context factory

Basically, what I want is a context factory that help me to get a new instance of a context with specific options.


public class BlogContextFactory
{
private readonly DbContextOptions<BlogContext> _options;

public BlogContextFactory(DbContextOptions<BlogContext> options)
{
_options = options;
}

public BlogContext GetNewDbContext()
{
return new BlogContext(_options);
}
}

The factory accept a DbContextOptions parameter in the constructor and returns a new instance of the BlogContext with these options.

Registration

Another important thing is the registration of the factory in a IoC container, that in my case is Autofac.

This is a mandatory stuff if I want to inject the factories in the services that depends on it:

var options = new DbContextOptionsBuilder<BlogContext>().UseSqlServer(_configuration.GetConnectionString("BlogContext")).Options;
builder.Register(b => new BlogContextFactory(options)).SingleInstance();

I’m using Autofac, so I register BlogContextFactory as single instance with specific options.

The first row is the most important; with the method UseSqlServer I’m saying that the db context will be configured with a sql server database, and obviously I’m giving the connections string as a parameter of the method.

In memory option

As well as I use this options to configure a db context with a sql database, I can do the same thing to configure an in memory database.


public abstract class DatabaseFixture : IDisposable
{
protected readonly BlogContextFactory ContextFactory;
readonly DbContextOptions<BlogContext> _options;

public DatabaseFixture(string databaseName)
{
_options = new DbContextOptionsBuilder<BlogContext>().UseInMemoryDatabase(databaseName).Options;
ContextFactory = new BlogContextFactory(_options);

var categories = new List<Category>()
{
new Category() { Id = new Guid("{5D134CF2-EF37-4663-B3D2-64F2E3DBF3BE}"), Name = "Category1" },
new Category() { Id = new Guid("{BE7EB1E1-FB67-4FE1-A96E-4721D37AFE29}"), Name = "Category2" },
new Category() { Id = new Guid("{BBB5D5DD-A746-4A87-9204-90EE87B5F560}"), Name = "Category3" }
};

var tags = new List<Tag>()
{
new Tag() { Id = new Guid("{859A4C65-F688-4F9F-8575-F389841665F8}"), Name = "Tag1" },
new Tag() { Id = new Guid("{A57D65BE-8A33-48E8-A3E4-3EF5D110A961}"), Name = "Tag2" },
new Tag() { Id = new Guid("{2D1AC63C-3D48-45A3-8F22-C9670A9D4F84}"), Name = "Tag3" }
};

var posts = new List<Post>()
{
new Post() { Id = new Guid("{7660E171-0776-451C-AD2C-1CA6F46EDB30}"), Title = "Post1", Content = "Content1", Published = true },
new Post() { Id = new Guid("{7221B957-E3F2-461F-A4F8-3D13699880A1}"), Title = "Post2", Content = "Content2", Published = false },
new Post() { Id = new Guid("{D2E934DB-7B64-42A7-953F-213EA771F596}"), Title = "Post3", Content = "Content3", Published = false }
};

var replies = new List<Reply>()
{
new Reply() { Id = new Guid("{9344DCAF-154E-4167-8EEF-0372896F5469}"), CreateUser = "user1", Content = "Content1", Published = true, Post = posts[0] },
new Reply() { Id = new Guid("{1DB75F77-A50E-4F90-83F8-711D3025EBA4}"), CreateUser = "user2", Content = "Content2", Published = true, Post = posts[0] },
new Reply() { Id = new Guid("{6772F1D7-E6D6-47DB-8724-E46353BFDFDD}"), CreateUser = "user3", Content = "Content3", Published = false, Post = posts[0] }
};

var postCategories = new List<PostCategory>()
{
new PostCategory() {
Id = new Guid("{267A0AE0-2E91-4FA5-B45C-361B8AAE635D}"),
Post = posts[0],
Category = categories[0] }
};

var postTags = new List<PostTag>()
{
new PostTag() {
Id = new Guid("{DA6CEA55-610F-4956-913F-D8F409014AC9}"),
Post = posts[0],
Tag = tags[0] }
};

using (var blogContext = ContextFactory.GetNewDbContext())
{
blogContext.Set<Category>().AddRange(categories);
blogContext.Set<Tag>().AddRange(tags);
blogContext.Set<Post>().AddRange(posts);
blogContext.Set<Reply>().AddRange(replies);
blogContext.Set<PostCategory>().AddRange(postCategories);
blogContext.Set<PostTag>().AddRange(postTags);
blogContext.SaveChanges();
}
}

public void Dispose()
{
using (var blogContext = ContextFactory.GetNewDbContext())
{
blogContext.RemoveRange(blogContext.Categories);
blogContext.RemoveRange(blogContext.Tags);
blogContext.RemoveRange(blogContext.Posts);
blogContext.RemoveRange(blogContext.Replies);
blogContext.RemoveRange(blogContext.PostCategories);
blogContext.RemoveRange(blogContext.PostTags);
blogContext.SaveChanges();
}
}
}

With the UseInMemoryDatabase method I can configure an instance of the db context with an in memory database.

The only things that I need to do is specify the name of the database (a string) and then populate the context with the entities collections.

After that, my in memory db context will be available for the using.

The source code of this topic is here.

In-memory DbContext with Entity Framework Core

Webpack with Angular2

In a previous post I spoke about the configurations to manage bundles in angular 2 with Systemjs.

For this purphose, the latest versions of Angular CLI uses webpack and we can use it as module bundler in the new angular 2/4 applications.

The reason of this choice is that webpack has a lot of plugins that they help us with a refined management of the modules bundling.

Entries

When we configure webpack, the first thing that we need to do is define the entries of the bundles.

Every bundle has the main entry point, that is the module where webpack start to build the bundle hierarchy.

I like to build two different bundle, the first one for the vendors libraries and the second one for the application modules.

The  main entry for the vendors is a typescript class that looks like this:


import '@angular/platform-browser';
import '@angular/platform-browser-dynamic';
import '@angular/core';
import '@angular/common';
import '@angular/http';
import '@angular/router';
import '@angular/animations';

import 'jquery';
import 'lodash';
import 'ng2-toastr';
import 'ng2-translate';
import 'reflect-metadata';
import 'zone.js';
import 'rxjs';

The second one is the main module for the app:


export { PostModule } from "./post.module";

Webpack is able to load these modules and build the dependencies of the childs descendans, and process them.

Loaders

Loaders are specific webpack tools that are able to process specific file types and to produce the right bundle outputs.

For example, in order to make webpack to be able to resolve the angular html templates or the router modules, or more simply to compile typescript files, specific loaders are needed.

Below you can see a bunch of loaders used in an Angular 2 application:


module: {
rules: [
{
test: /\.ts$/,
loaders: [
{
loader: 'awesome-typescript-loader',
options: { configFileName: 'tsconfig.json' }
}, 'angular2-template-loader', 'angular2-router-loader'
]
},
{
test: /\.html$/,
loader: 'html-loader'
},
{
test: /\.(png|jpe?g|gif|svg|woff|woff2|ttf|eot|ico)$/,
loader: 'file-loader?name=images/[name].[hash].[ext]'
},
{
test: /\.css$/,
exclude: helpers.root('src', 'app'),
loader: ExtractTextPlugin.extract({ fallbackLoader: 'style-loader', loader: 'css-loader?sourceMap' })
},
{
test: /\.css$/,
include: helpers.root('src', 'app'),
loader: 'raw-loader'
}
]
}

Plugins

These are utilities that we can help us in some additional tasks of the webpack script.

For example, we would like to share a common chunk and we would want that the chunk, if used from different module of the application, is not repeated in the bundle but is present only once.

CommonsChunkPlugin deal with this work:


plugins: [

.....

new webpack.optimize.CommonsChunkPlugin({
name: ['vendor']
}),

.....

]

Another requirement could be write the script references in the index.html file of the application.

HtmlWebpackPlugin can help us in this work:


plugins: [

.....

new HtmlWebpackPlugin({
filename: 'index.html',
template: 'src/index.html',
chunks: ['app', 'vendor']
}),

new HtmlWebpackPlugin({
filename: 'index.admin.html',
template: 'src/index.admin.html',
chunks: ['app.admin', 'vendor']
}),

.....
]

Output

In this section we can configure the file results of the webpack process.

We have to produce these files in the public folder of the application; the output section is look like this:


output: {
path: helpers.root('wwwroot'),
filename: 'js/[name].js',
chunkFilename: 'js/[id].chunk.js'
}

 

You can find the source code here.

 

Webpack with Angular2

Manage attachments chunks with ASP.NET Web Api

In the previous post I spoke about a custom MultipartFormData stream provider and how it can help us to manage some custom informations included in a request message.

In that example I generated chunks form a file and I sent those to a rest service (AKA Web API) with some additional informations that were then retrieved from the custom provider.

Now I want to use these informations to manage the upload session and merge all the chunks when received.

What I need to do is define the models involved in the process and the service that manage the chunks.

Models

We have to define two stuff, the first one is the model for the chunk:


public class ChunkMetadata
{
public string Filename { get; set; }
public int ChunkNumber { get; set; }

public ChunkMetadata(string filename, int chunkNumber)
{
Filename = filename;
ChunkNumber = chunkNumber;
}
}

The ChunkNumber property deserves an explanation; is the number associated to the chunk and will be useful to understand the correct order when we’ll have to merge all of them.

The second one is the model of the session, that is the bunch of the chunks that compose the file.

First of all we define the interface:


public interface IUploadSession
{
ConcurrentBag<ChunkMetadata> Chunks { get; set; }
string Filename { get; }
long Filesize { get; }
bool AddChunk(string filename, string chunkFileName, int chunkNumber, int totalChunks);
Task MergeChunks(string path);
}

The FileName and Filesize are closely tied to the session; we need AddChunk and MergeChunks methods as well.

We also need a thread safe collection for the chunks that compose the session, so we define a CuncurrentBag collection, that is the thread safe representation of the List.

Now we can implement the model:


public class UploadSession : IUploadSession
{
public string Filename { get; private set; }
public long Filesize { get; private set; }
private int _totalChunks;
private int _chunksUploaded;

public ConcurrentBag<ChunkMetadata> Chunks { get; set; }

public UploadSession()
{
Filesize = 0;
_chunksUploaded = 0;
Chunks = new ConcurrentBag<ChunkMetadata>();
}

public bool AddChunk(string filename, string chunkFileName, int chunkNumber, int totalChunks)
{
if (Filename == null)
{
Filename = filename;
_totalChunks = totalChunks;
}

var metadata = new ChunkMetadata(chunkFileName, chunkNumber);
Chunks.Add(metadata);

_chunksUploaded = Interlocked.Increment(ref _chunksUploaded);
return _chunksUploaded == _totalChunks;
}

public async Task MergeChunks(string path)
{
var filePath = path + Filename;

using (var mainFile = new FileStream(filePath, FileMode.Create))
{
foreach (var chunk in Chunks.OrderBy(c => c.ChunkNumber))
{
using (var chunkFile = new FileStream(chunk.Filename, FileMode.Open))
{
await chunkFile.CopyToAsync(mainFile);
Filesize += chunkFile.Length;
}
}
}

foreach (var chunk in Chunks)
{
File.Delete(chunk.Filename);
}
}
}

The implementation is quite simple.

The AddChunk method add the new chunk to the collection, then increment the _chunksUploaded property with the thread safe operation Interlocked.Increment; at the end, the method returns a bool that is true if all the chunks are received, otherwise false.

The MergeChunks method deal with the retrieve of all the chunks from the file system.

It gets the collection, order by the chunk number, read the bytes from the chunks and copy those to the main file stream.

After all, the chunks are deleted.

Service

The service will have an interface like this:


public interface IUploadService
{
Guid StartNewSession();
Task<bool> UploadChunk(HttpRequestMessage request);
}

In my mind, the StartNewSession method will instantiate a new Session object and assign a new correlation id that is the unique identifier of the session.

This is the implementation:


public class UploadService : IUploadService
{
private readonly Context _db = new Context();
private readonly string _path;
private readonly ConcurrentDictionary<string, UploadSession> _uploadSessions;

public UploadService(string path)
{
_path = path;
_uploadSessions = new ConcurrentDictionary<string, UploadSession>();
}

public async Task<bool> UploadChunk(HttpRequestMessage request)
{
var provider = new CustomMultipartFormDataStreamProvider(_path);
await request.Content.ReadAsMultipartAsync(provider);
provider.ExtractValues();

UploadSession uploadSession;
_uploadSessions.TryGetValue(provider.CorrelationId, out uploadSession);

if (uploadSession == null)
throw new ObjectNotFoundException();

var completed = uploadSession.AddChunk(provider.Filename, provider.ChunkFilename, provider.ChunkNumber, provider.TotalChunks);

if (completed)
{
await uploadSession.MergeChunks(_path);

var fileBlob = new FileBlob()
{
Id = Guid.NewGuid(),
Path = _path + uploadSession.Filename,
Name = uploadSession.Filename,
Size = uploadSession.Filesize
};

_db.FileBlobs.Add(fileBlob);
await _db.SaveChangesAsync();

return true;
}

return false;
}

public Guid StartNewSession()
{
var correlationId = Guid.NewGuid();
var session = new UploadSession();
_uploadSessions.TryAdd(correlationId.ToString(), session);

return correlationId;
}
}

In the StartNewSession method we use the thread safe method TryAdd to add a new session to the CuncurrentBag.

About the UploadChunk method, we seen the first part of the implementation in the previous post.

Once the metadata is retrieved from the request, we try to find the session object with a thread safe operation.

If we don’t find the object, of course we need to throw an exception because we expect that the related session exists.

If the session exists, we add the chunk to the session and we check the result of the operation.

If is the last chunk, we merge all of them and we can do a database operation if needed.

Controller

The implementation of the controller is very simple:


public class FileBlobsController : ApiController
{
private readonly IUploadService _fileBlobsService;
private readonly Context _db = new Context();

public FileBlobsController(IUploadService uploadService)
{
_fileBlobsService = uploadService;
}

[Route("api/fileblobs/getcorrelationid")]
[HttpGet]
public IHttpActionResult GetCorrelationId()
{
return Ok(_fileBlobsService.StartNewSession());
}

[HttpPost]
public async Task<IHttpActionResult> PostFileBlob()
{
if (!Request.Content.IsMimeMultipartContent())
throw new Exception();

var result = await _fileBlobsService.UploadChunk(Request);

return Ok(result);
}
}

You can find the source code here.

Manage attachments chunks with ASP.NET Web Api