Friday, 21 July 2017

C#: How to I extend the timeout of a WebClient call? (30 seconds is just not long enough)

My Azure function is calling web services that take longer than 30 seconds to complete, so I needed to extend the default timeout the WebClient class.

I created the following base class:

using System;
using System.Net;

namespace MyHelpers
{
    public class MyWebClient : WebClient
    {
        public int Timeout { get; set; }

        public WebDownload() : this(180000) { }

        public WebDownload(int timeout)
        {
            this.Timeout = timeout;
        }

        protected override WebRequest GetWebRequest(Uri address)
        {
            var request = base.GetWebRequest(address);
            if (request != null)
            {
                request.Timeout = this.Timeout;
            }
            return request;
        }
    }
}

I set the default to 3 minutes. I can then use it like I would a usual WebClient call:

private string ExecuteAPI(string url, Dictionary<string, string> parameters)
{
    try
    {
        using (var client = new MyWebClient())
        {
            foreach (var param in parameters)
            {
                client.Headers.Add(param.Key, param.Value);
            }

            return client.DownloadString(url);
        }
    }
    catch
    {
        return string.Empty;
    }
}

Thursday, 20 July 2017

C#: How do I split a generic list into smaller chunks?

When trying to send some records to an Azure Service Bus payload, I found that due to the limits of the tenant subscription, the Service Bus could not handle the large payload I was sending. I have 256K to deal with and 1000+ records to load.

A simple solution is to group the items in a logical manner, and then split them up into equal sized batches. LINQ to the rescue.

Grouping the items is done as follows:
var items = new List<MyObject>();
// add all the items the list
var records = items.GroupBy(a => a.MyGroupingProperty);

Then batch it up:
var batchSize = 10;
var batchedItems = records
.Select((x, i) => new { Index = i, Value = x })
.GroupBy(x => x.Index / batchSize)
.Select(x => x.Select(v => v.Value).ToList())
.ToList();

batchedItems is now grouped into chunks of 10. I can now iterate though the batches and handle them accordingly:

foreach(var batchedItem in batchedItems)
{
// magic goes here.
}


Wednesday, 19 July 2017

Entity Framework: No Entity Framework provider found for the ADO.NET provider with invariant name 'System.Data.SqlClient'

The solution is simple - install the Entity Framework Nuget package to the solution startup project.

Entity Framework: No connection string named 'ImportEntities' could be found in the application config file.

I got this message when writing a PoC application for Azure. The message is quite self explanatory, but the key is WHICH PROJECT TO UPDATE. I have multiple projects, but my startup was a Console Application (for testing).

The solution is to add the setting (in app.config or web.config) to the solution STARTUP PROJECT.

<configuration>
  <connectionStrings>
    <add name="ImportEntities" connectionString="...." />
  </connectionStrings>
</configuration>

Azure Blob Storage: How can I quickly read a text file?

My current project has the requirement to read a large text file from blob storage and then process the contents. Getting the file is easy:

var storageAccount = CloudStorageAccount.Parse(CONNECTIONSTRING);
var container = storageAccount.CreateCloudBlobClient().GetContainerReference(containerName);
var blob =  container.GetBlobReference(fileName);

And the data can be easily read:

Stream stream = new MemoryStream();
blob.DownloadToStream(stream);
stream.Position = 0;

string text = "";
using (StreamReader reader = new StreamReader(stream))
{
text = reader.ReadToEnd();
}
var lines = text.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);

The real time saver came when I had to process the contents. A traditional IEnumerable was too slow, but luckily Parallel saved the day:

Parallel.For(0, lines.Length - 1, i =>
{
DoSomethingWithTheRow(lines[i]);
});

Its lightening fast.

A word of warning: I passed a generic into the function (List<MyCustomType>) and found out very quickly that it is not thread safe. I changed to ConcurrentQueue<MyCustomType> and the issue was resolved.

Monday, 17 July 2017

Azure API: A route named 'swagger_docs' is already in the route collection. Route names must be unique

While creating some PoC APIs, I encountered this little gem.

Fortunately, a simple fix is available: In the 'Settings' tab of the 'Publish' dialog, click 'Remove additional files at destination' (in the File Publish Options expander).


You can then republish the API and the issue should be resolved.

Tuesday, 11 July 2017

C#: How can I create a class dynamically from text?

The following code is a really simple way to create a class dynamically from string.

 dynamic data = new ExpandoObject();

IDictionary<string, object> dictionary = (IDictionary<string, object>)data;
dictionary.Add("FirstName", "Richard");
dictionary.Add("Surname", "Leeman");

Console.WriteLine(data.FirstName + " " + data.Surame);