Azure Blob Storage: How can I quickly read a text file?

My current project has the requirement to read a large text file from blob storage and then process the contents. Getting the file is easy:

var storageAccount = CloudStorageAccount.Parse(CONNECTIONSTRING);
var container = storageAccount.CreateCloudBlobClient().GetContainerReference(containerName);
var blob =  container.GetBlobReference(fileName);

And the data can be easily read:

Stream stream = new MemoryStream();
blob.DownloadToStream(stream);
stream.Position = 0;

string text = "";
using (StreamReader reader = new StreamReader(stream))
{
text = reader.ReadToEnd();
}
var lines = text.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);

The real time saver came when I had to process the contents. A traditional IEnumerable was too slow, but luckily Parallel saved the day:

Parallel.For(0, lines.Length - 1, i =>
{
DoSomethingWithTheRow(lines[i]);
});

Its lightening fast.

A word of warning: I passed a generic into the function (List<MyCustomType>) and found out very quickly that it is not thread safe. I changed to ConcurrentQueue<MyCustomType> and the issue was resolved.

Comments

Post a Comment

Popular posts from this blog

SharePoint 2013: Error updating managed account credentials

I encountered the following message when trying to change a password for a managed account: Error deploying administration application pool credentials. Another deployment may be active. An object of the type Microsoft.SharePoint.Administration.SPAdminAppPoolCredentialDeploymentJobDefinition named "job-admin-apppool-change" already exists under the parent Microsoft.SharePoint.Administration.SPTimerService named "SPTimerV4".  Rename your object or delete the existing object. The error message is self explanatory - Rename your object or delete the existing object.  I choose the latter. Powershell to the rescue. $job = Get-SPTimerJob -Identity "job-admin-apppool-change" $job.Delete() Reset the password and all should be good.
Read more

Error deploying Nintex workflow: An item with the same key has already been added

My client recently upgraded their version of Nintex Workflow 2010 and suddenly some of my workflows started to fail on deployment. The following error cropped up: Error deploying workflow: Server was unable to process request. ---> An item with the same key has already been added. ---> An item with the same key has already been added. WTF???? A little digging unveiled the problem - I had two items in a 'Create Item' shape with the SAME DISPLAYNAME. I needed to move the second item to a new 'Update Item' shape in order for it to work. I order to get the workflow to deploy, I had to manually remove the 'duplicate' item from the serialized XML. Hooray!! It was then a simple matter of creating a new shape to update the 'duplicate' value.
Read more