Resolving slow row inserts in vb.net

wow who would have thought, adding rows to a datatable with a strongly typed dataset is about 10 times faster than an untyped dataset. It’s definately worth the effort if you are importing large CSVs.

 

I was writing a smal lapp that takes the content from 3 CSV files and loads them into datatables. Then I merge them into a fourth with some business logic thrown in along the way.

Inserting rows without a strongly types dataset was taking ages. I am importing about 50,000 rows and I swear it was taking around 20 mins to run. Of that maybe 3-4 minutes is loading the csv into the datatables. Because we are reading CSV the format is a little unpredictable. I changed that and wrote new exporters so that the formats all match then added a dataset and manually defined the datatable. Now when I read the csv into tables I declare them with the type. The inserts are faster like done from 15 minutes to maybe under a minute.

 

Amazing who would have thought!

 

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply