Tell me more ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 0 down vote favorite

I have a 260k line csv file that has two columns. I have read in the csv file using fgetcsv and have a while loop which reads every line in the file. In the loop I am trying to add the values from the second column to an array.

When I have the line in to add to the array, my PHP freezes and doesn't finish. I have done debugging and the values are getting added to the array so I know that the adding to array and while loop work but I do not know why it freezes.

If I remove the line the while loop completes going through the 260k lines and then processes the rest of the file.

Here is my code:

$amountRecords = 0;
$totalValue = 0;
$valueArray = array();

// reads in csv file
$handle = fopen('Task1-DataForMeanMedianMode.csv', 'r');
// to skip the header names/values
fgetcsv($handle);

// creates array containing variables from csv file
while(($row = fgetcsv($handle, "\r")) != FALSE)
{

    /*
    echo "ROW CONTAINS: ";
    var_dump($row[1]);
    echo "<br />";
    */

    $valueArray[] = $row[1];

    /*
    echo "VALUEARRAY NOW CONTAINS: ";
    var_dump($valueArray);
    echo "<br />";
    */

    $totalValue = $totalValue + $row[1];
    $amountRecords++;

}

And sample of csv file:

ID,Value
1,243.00
2,243.00
3,243.00
4,243.00
5,123.11
6,243.00
7,180.00
8,55.00
9,243.00
10,55.00
share|improve this question
1  
Have you checked your error logs for the specific error? The obvious suspect is that you are exceeding your memory limit. Can you add your specific error to verify this? – ctrahey Aug 19 '12 at 16:27
@ctrahey no error is showing. Just completely stops and that's how. How can I get the error to appear? – Josh Pullin Aug 19 '12 at 16:37
You'll need to modify your php.ini file (on *nix systems usually at /etc/php.ini) and modify the display_errors line to be uncommented and have a value of 1. Note that you usually can enable this with ini_set('display_errors', 1);, however it may not work with fatal errors (such as out-of-memory). It is also possible that your server is already logging these errors, and you can find out where in your php.ini file. – ctrahey Aug 19 '12 at 18:20
@ctrahey Yeah just putting that line in worked. This is the error that got returned: Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 72 bytes) in /Applications/MAMP/htdocs/Kiddi Voucher Test/index.php on line 43 – Josh Pullin Aug 19 '12 at 19:00
add comment (requires an account with 50 reputation)

4 Answers

up vote 1 down vote

With an out-of-memory error, there are two general approaches. As usual with these choices, you can pick easy-but-wrong and hard-but-right. The easy-but-wrong solution is to increase your memory limit to an appropriate level: 

ini_set('memory_limit', '64M');

the better (although harder) solution is to re-engineer your algorithm to not need as much memory. This is clearly the more sustainable and robust approach. To do this properly, you will need to evaluate what you need to do with the array you are building. For instance, I have written similar scripts which were importing the rows to a database. Instead of building a huge array and then inserting, I did it in batches, where I built an array of 50-100 rows, then inserted those and cleared the array (freeing the memory for re-use).

Pseudo-code: 

for(each row in file) {
  $rows_cache[] = $row[1];
  if(count($rows_cache) >= 50) {
    insert_these($rows_cache);
    $rows_cache = array();
  }
}
share|improve this answer
Problem is I am not doing anything with databases on this file, just doing calculations on the data in the csv file. I see your point with doing the second approach though. Thanks for your help. – Josh Pullin Aug 19 '12 at 19:58
Yeah, the notion is the same. Can you do your calculations "in real-time"? i.e. if you are just doing total & average, you can simply keep two variables: $total += $row[1]; $num_rows++;. Just think about what you are doing with the array and find ways to do it "as you go" and not need to build the full array in-memory. – ctrahey Aug 19 '12 at 20:05
Well I am actually getting another error now as I have to sort the array to find the median. I cannot use any of the sort functions has to be done manually but it is now causing another error which is: Fatal error: Maximum execution time of 30 seconds exceeded in /Applications/MAMP/htdocs/Kiddi Voucher Test/index.php on line 71 How would I go about it sorting the array on the go like you said so I do not have to go back through the whole array again to sort it? – Josh Pullin Aug 19 '12 at 20:14
I have put set_time_limit(0); and ini_set('max_execution_time', 0); but still seems to be very slow and not doing anything? – Josh Pullin Aug 19 '12 at 20:15
add comment (requires an account with 50 reputation)
up vote 0 down vote

Your first row is string, maybe try adding

while(($row = fgetcsv($handle, "\r")) != FALSE)
{

    if(is_numeric($row[1]))
    { 
        $valueArray[] = $row[1];

        $totalValue = $totalValue + $row[1];
        $amountRecords++;
    }
}
share|improve this answer
I don't think that would freeze the code. It just puts a string in the array. When trying to add a string with a number, the string will parse to zero... – Dirk McQuickly Aug 19 '12 at 16:34
The line fgetcsv($handle); removes the first string so it isn't anything to do with that. And the totalValue and amountRecords are correct. – Josh Pullin Aug 19 '12 at 16:36
using maybe in answer is not a good idea – shiplu.mokadd.im Aug 19 '12 at 16:44
add comment (requires an account with 50 reputation)
up vote 0 down vote

Why not drop the line:

$totalValue = $totalValue + $row[1];

from inside your loop, and instead use:

$totalValue = array_sum($valueArray);

after completing your loop

share|improve this answer
For the task I cannot use any array sorting or numerical functions that PHP has. Has to be done all manually. – Josh Pullin Aug 19 '12 at 19:00
Is this homework then? Otherwise why the restriction? – Mark Baker Aug 19 '12 at 20:57
Yeah its. Got to work it out instead of use the functions. – Josh Pullin Aug 19 '12 at 21:34
add comment (requires an account with 50 reputation)
up vote 0 down vote

Not really the problem, but

while(($row = fgetcsv($handle, "\r")) != FALSE)

can be rewritten as

while($row = fgetcsv(...))

instead. There's no need for the explicit false check - if fgetcsv returns false, the while loop would terminate anyways. Plus this version is more legible, and not as risky. If you forget the () around the fget portion, you'll be doing the equivalent of $row = (fgetcsv() != false) and simply getting a boolean value.

share|improve this answer
add comment (requires an account with 50 reputation)

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.