I am developing a file parser in C# such that I have a file with many many rows. I want to write that to database.

I have created a loop in which I read the file and issue an insert.

However, what I would like to do is read say 100 rows then commit all 100, then read the next 100 and commit them and so on an so forth.

My problem (a bit conceptual) is how to find out the last set of rows as that could be less than 100.

For e.g. if my file has 550 rows, I can read 5 chunks of 100 and commit them using modulus operator on the line counter but what happens to the last batch of 50?

Any ideas in this area will be highly appreciated.

Thanks.

share|improve this question

3 Answers

Top off my mind here:

First option, do a quick count of '\n', thus getting the number of lines. Then, read chunks in multiple of 2 such that the result is very near to the number of lines, for example if 501 lines exist, read chunks of 250 and 250. Then only 1 line is left. When you first run the code, check for odd or even. If odd, remember to commit the last line, else if the number of lines are even, there is no need to compensate for the last line. Hope this helps.

P.S: After dividing the number into multiple of 2, if it is very large, you can further divide into 2 (Divide and conquer!)

share|improve this answer

If your read function returns false when there isn't anything else to read. then you can try this.

int i = 1;
while (read())
{
     if (i % 100 == 0)
     {
        for int (j = 0; j < 100; j++)
        { 
            write();
        }
     }
     i++;
}
for (int j = 0; j < i % 100; j++)
{
     write();
}
share|improve this answer

Something like this, perhaps? The following is pseudocode:

int rowCount = 0;
while more records to read
{
   // Get record
   // Parse record
   // Do something with results

   if(++rowCount % 100 == 0)
   {
      // Commit
      rowCount = 0;
   }
}
// If you have 0 records, you just did a commit (unless you had zero records to read,
// but I assume you can code your way around that!)  So you only need a final commit 
// if you had more than 0 records still uncommitted. 

if(rowCount > 0) 
{
   // Commit remaining records
}
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.