Thursday, 20 July 2017

C#: How do I split a generic list into smaller chunks?

When trying to send some records to an Azure Service Bus payload, I found that due to the limits of the tenant subscription, the Service Bus could not handle the large payload I was sending. I have 256K to deal with and 1000+ records to load.

A simple solution is to group the items in a logical manner, and then split them up into equal sized batches. LINQ to the rescue.

Grouping the items is done as follows:
var items = new List<MyObject>();
// add all the items the list
var records = items.GroupBy(a => a.MyGroupingProperty);

Then batch it up:
var batchSize = 10;
var batchedItems = records
.Select((x, i) => new { Index = i, Value = x })
.GroupBy(x => x.Index / batchSize)
.Select(x => x.Select(v => v.Value).ToList())

batchedItems is now grouped into chunks of 10. I can now iterate though the batches and handle them accordingly:

foreach(var batchedItem in batchedItems)
// magic goes here.

No comments:

Post a comment