Custom MultipartFormDataStreamProvider in C#

Frequently, when we manage multipart/form requests and we send them to the server, we might want to add some additional informations.

Perhaps we might want to split a big file in chunks and we might want to add some additional informations like the id of the upload session, the chunk number, the file name and the total chunks number that compose the file.

Suppose that we use for the client side Angularjs, the code of the controller is quite simple:


.....

public AddAttachment(event) {
let attachments = event.target.files;
if (attachments.length > 0) {
let file: File = attachments[0];

this.$http.get(this.url + "/GetCorrelationId").then((correlationId) => {
let chunks = this.SplitFile(file);

for (let i = 0; i < chunks.length; i++) {
let formData = new FormData();
formData.append("file", chunks[i], file.name);
formData.append("correlationId", correlationId.data);
formData.append("chunkNumber", i + 1);
formData.append("totalChunks", chunks.length);

this.$http.post(this.url, formData, { headers: { "Content-Type": undefined } }).then((result) => {
if(result.data) {
this.Load();
}
});
}
});
}
}

private SplitFile(file: File): Array<Blob> {
let chunks = Array<Blob>();
let size = file.size;
let chunkSize = 1024 * 1024 * 10;
let start = 0;
let end = chunkSize;

while (start < size) {
let chunk = file.slice(start, end);
chunks.push(chunk);
start = end;
end += chunkSize;
}

return chunks;
}

.....

The AddAttachment method is invoked by the view; once the file is retrieved, the split method generate the array of chunks.

Then, with the $http factory we send every single chunks to the server with additional metadata.

In order to read these datas from the server side, we need to implement a custom MultipartFormData stream provider.

The first step is define the interface of our provider:


public interface ICustomMultipartFormDataStreamProvider
{
string ChunkFilename { get; }
int ChunkNumber { get; }
string CorrelationId { get; }
string Filename { get; }
int TotalChunks { get; }
void ExtractValues();
}

The interface has the same properties sent by the client, and a method that deal with extract the values from the message.

Now we can proceed with the implementation:


public class CustomMultipartFormDataStreamProvider : MultipartFormDataStreamProvider, ICustomMultipartFormDataStreamProvider
{
public string Filename { get; private set; }
public string ChunkFilename { get; private set; }
public string CorrelationId { get; private set; }
public int ChunkNumber { get; private set; }
public int TotalChunks { get; private set; }

public CustomMultipartFormDataStreamProvider(string rootPath) : base(rootPath) { }

public CustomMultipartFormDataStreamProvider(string rootPath, int bufferSize) : base(rootPath, bufferSize) { }

public override Task ExecutePostProcessingAsync()
{
foreach (var file in Contents)
{
var parameters = file.Headers.ContentDisposition.Parameters;
var filename = ExtractParameter(parameters, "filename");
if (filename != null) Filename = filename.Value.Trim('\"');
}

return base.ExecutePostProcessingAsync();
}

public void ExtractValues()
{
var chunkFileName = FileData[0].LocalFileName;
var correlationId = FormData?.GetValues("correlationId");
var chunkNumber = FormData?.GetValues("chunkNumber");
var totalChunks = FormData?.GetValues("totalChunks");

if (string.IsNullOrEmpty(chunkFileName) || correlationId == null || chunkNumber == null || totalChunks == null)
throw new Exception("Missing values in UploadChunk session.");

ChunkFilename = chunkFileName;
CorrelationId = correlationId.First();
ChunkNumber = int.Parse(chunkNumber.First());
TotalChunks = int.Parse(totalChunks.First());
}

private NameValueHeaderValue ExtractParameter(ICollection<NameValueHeaderValue> parameters, string name)
{
return parameters.FirstOrDefault(p => p.Name.Equals(name, StringComparison.OrdinalIgnoreCase));
}
}

The class inherits from MultipartFormDataStreamProvider base class and implements our interface.

Two methods are implemented; the first one override ExecutePostProcessingAsync and in this method we retrieve the name of the main file.

The second one extract the custom parameters from the FormData; we retrieve also the chunk filename from the FileData object; this information is included as default information in the MultipartFormData message.

Now the informations are retrieved and we can use the custom provider in a service:


public async Task<bool> UploadChunk(HttpRequestMessage request)
{
var provider = new CustomMultipartFormDataStreamProvider(_path);
await request.Content.ReadAsMultipartAsync(provider);
provider.ExtractValues();

.....
}

The metadata will be available in the provider object.

You can find the source code here.

 

 

Advertisements
Custom MultipartFormDataStreamProvider in C#

Real-time search with ASP.NET and Elasticsearch

A common problem that we are faced when we have deployed our applications is improve the performance of a page or feature.

In my case for example I had a field where I could search and select a city, so the starting elements were a lot and the search was quite slow; I wanted a better user experience.

We can solve performance problems like these with the help of a cache or a full-text search.

I have chosen the last one and elastic search as full-text engine, so I’ll describe the steps that I followed to configure and use it in my application.

Installation

The first step is install the elastic search server, that you can download here.

Once installed we have to start it by executing the following executable:

<Installation path>\bin\elasticsearch.bat

This is the server log:

log

The server will take care to index the content that we will pass to it; in order to do that we need a client to use in our application; in my case the application was .NET and I used NEST.

NEST

As said above, NEST is an elastic search high level client for .NET applications.

The first step is install it in the application with nuget:

Install-package NEST

And in the package.config we’ll have:

package

Now we have all the necessary tools and we can develop the code for the search feature.

Client

We define a client class that has one responsability, that is setup the url and the default index of the client, and that instantiate it:


public class ElasticSearchClient
{
privatereadonlyIElasticClient _client;

publicElasticSearchClient(IElasticClient client)
{
_client = client;
}

publicElasticSearchClient(string uri, string indexName) : this(CreateElasticClient(uri, indexName)) {}

publicIElasticClientGetClient()
{
return_client;
}

privatestaticElasticClientCreateElasticClient(string uri, string indexName)
{
var node = newUri(uri);
var setting = newConnectionSettings(node);
setting.DefaultIndex(indexName);
returnnewElasticClient(setting);
}
}

Once instantiated, the class returns a new instance of the client; we can register it in the startup class of the application with autofac:


public partial class Startup
{
publicvoidConfiguration(IAppBuilder app)
{
var builder = newContainerBuilder();

builder.Register(c => newElasticSearchClient("http://localhost:9200", "cities"))
.AsSelf()
.SingleInstance();
...
}
}

Service base class

A service that uses an elasticsearch index should be able to do some basic operations, that concerns the logics of the full-text indexes.

We have to deal with the initialize a specific index, populate the index with the contents, obviously performs a search on the index with specific parameters.

So, we have to define an interface like this:


internal interface IElasticSearchService<T> where T : class
{
voidInit();
voidCheckIndex();
voidBulkInsert(List<T> objects);
IEnumerable<T> Search(string query);
}

I like to separate the init method, that create the index, from the checkindex method, that check if the index already exists.

Now we can implement the basic service:


public class ElasticSearchService<T> : IElasticSearchService<T> where T : class
{
protectedreadonlyContext Db = newContext();
protectedreadonlyElasticSearchClient ElasticSearchClient;
protectedreadonlystring IndexName;

publicElasticSearchService(ElasticSearchClient elasticSearchClient, string indexName)
{
ElasticSearchClient = elasticSearchClient;
IndexName = indexName;
}

publicvirtualvoidInit()
{
CheckIndex();
BulkInsert(Db.Set<T>().ToList());
}

publicvoidCheckIndex()
{
if (IndexExist()) return;
var response = CreateIndex();

if (!response.IsValid)
{
thrownewException(response.ServerError.ToString(), response.OriginalException);
}
}

publicvoidBulkInsert(List<T> objects)
{
var response = ElasticSearchClient.GetClient().IndexMany(objects, IndexName);
if (!response.IsValid)
{
thrownewException(response.ServerError.ToString(), response.OriginalException);
}
}

publicvirtualIEnumerable<T> Search(string query)
{
var results = ElasticSearchClient.GetClient().Search<T>(c => c.From(0).Size(10).Query(q => q.Prefix("_all", query)));

returnresults.Documents;
}

protectedvirtualIResponseCreateIndex()
{
var indexDescriptor = newCreateIndexDescriptor(IndexName).Mappings(ms => ms.Map<T>(m => m.AutoMap()));
returnElasticSearchClient.GetClient().CreateIndex(indexDescriptor);
}

protectedboolIndexExist()
{
returnElasticSearchClient.GetClient().IndexExists(IndexName).Exists;
}
}

The constructor accept the client and the index name.

We define a virtual init method, that check if the index exists and do a bulkinsert of a list of object; the method is virtual, we think that a derived service could override the method.

This bulkinsert method leverage the client to index the object list and the search method implements a basic search, that searchs in all the fields of the objects by using the special field _all, which contains the concatenate values of all fields.

The method returns the first 10 elements.

Createindex create a specific index with automap option, that infers the elasticsearch fields datatypes from the POCO object that we pass to it; it’s protected, so the derived class could use it.

IndexExists check if an index exists and it can be used from the derived class as well.

Service

Now we can implement a specific service, that inherits from ElasticSearchService class.

In this example I need to search in a list of cities and related districts, so I need to override the CreateIndex method like this:


public sealed class CitiesService : ElasticSearchService<City>
{
publicCitiesService(ElasticSearchClient elasticSearchClient, string indexName): base(elasticSearchClient, indexName) {}

protectedoverrideIResponseCreateIndex()
{
var indexDescriptor = newCreateIndexDescriptor(IndexName).Mappings(
ms => ms.Map<City>(m => m.AutoMap().Properties(ps =>
ps.Nested<District>(n => n
.Name(nn => nn.District)
.AutoMap()))));

returnElasticSearchClient.GetClient().CreateIndex(indexDescriptor);
}

publicoverrideIEnumerable<City> Search(string query)
{
var results = ElasticSearchClient.GetClient().Search<City>(c => c.From(0).Size(10).Query(q => q.Prefix(p => p.Name, query) || q.Term("district.name", query)));

returnresults.Documents.OrderBy(d => d.Name);
}
}

What I need to do is automap the city object and the district, that is a closely related entity of the city; so I have to map the District property as nested with the automap option as well.

Thus I will able to search for all the properties of the city and the district.

The other method that I override is the Search method; I search partially in the name of the city (Prefix) and the specific term in the district name (Term) and I returns the first 10 elements.

Now I have to register the service with autofac:


public partial class Startup
{
public void Configuration(IAppBuilder app)
{
var builder = newContainerBuilder();

builder.Register(c => newElasticSearchClient("http://localhost:9200", "cities"))
.AsSelf()
.SingleInstance();

builder.Register(c => newCitiesService(c.Resolve<ElasticSearchClient>(), "cities"))
.AsSelf()
.AsImplementedInterfaces()
.SingleInstance();

...
}
}

The last step is initialize the full text index of my service:


public partial class Startup
{
publicvoidConfiguration(IAppBuilder app)
{
var builder = newContainerBuilder();

builder.Register(c => newElasticSearchClient("http://localhost:9200", "cities"))
.AsSelf()
.SingleInstance();

builder.Register(c => newCitiesService(c.Resolve<ElasticSearchClient>(), "cities"))
.AsSelf()
.AsImplementedInterfaces()
.SingleInstance();

...

InitElasticSearchServices(containerBuilder);
}

privatestaticvoidInitElasticSearchServices(IContainer containerBuilder)
{
var citiesServices = containerBuilder.Resolve<CitiesService>();
citiesServices.Init();
}
}

I make a new instance of the service and call the Init method of the ElasticSearchService that we have seen above.

This method will create and populate the index.

Web API

Now I can use the service in my Web API, like this:


public class CitiesController : ApiController
{
privatereadonlyCitiesService _elasticSearchService;

publicCitiesController(CitiesService elasticSearchService)
{
_elasticSearchService = elasticSearchService;
}

// GET: api/Cities
publicIEnumerable<City> GetCities(string query)
{
return_elasticSearchService.Search(query);
}

protectedoverridevoidDispose(bool disposing)
{
base.Dispose(disposing);
}
}

You can find the source code of this topic here.

Real-time search with ASP.NET and Elasticsearch