This is a loop used in our script with curl. It's causing CPU usage to shoot up to 100%. A friend said "Your computer is looping so fast here it doesn't have time to process the request since it is constantly checking for a finish."... So my question is how can this loop be re-written to slow down? Thanks

$running = null;
do {
  curl_multi_exec($mh, $running);
} while($running > 0);
share|improve this question

6 Answers

Try this: http://php.net/manual/function.curl-multi-select.php

share|improve this answer
1  
Using this function is the VITAL step that is missing in the question and in all the other answers so far. – Daniel Stenberg Jun 21 '11 at 21:15

Adding a call to http://php.net/sleep or http://php.net/usleep in every iteration should reduce the CPU usage by allowing other running processes to be scheduled by the operating system.

share|improve this answer
sleep() isn't having an affect on the CPU usage, any other thoughts? – Anagio Jun 21 '11 at 18:54
If a value of 1 or 2 does not make any difference then you should investigate php.net/manual/en/function.curl-multi-exec.php. Maybe you have to close some handles or something similar. – webspy Jun 21 '11 at 19:00
the problem with sleep is that it will block even if the processes are finished. if curl finishes 0.1 second after calling curl_multi_exec you have to wait 0.9 or 1.9 seconds. – Karoly Horvath Jun 21 '11 at 19:18
Checking about 90 URL's with sleep(1) caused the script to slow down and take 140seconds to complete...... Currently it finishes in 9seconds were trying curl-multi-exec now – Anagio Jun 21 '11 at 19:24
Try usleep(1). On my system an infinite loop (that increments a variable) without any sleep calls causes 100% CPU usage, while one with a call to usleep(1) causes less than 20% CPU usage. I am aware that sleeping is not the best option, but it may help reduce CPU usage. – webspy Jun 21 '11 at 19:28
show 2 more comments

Unfortunately you didn't post whole code. I suppose you are doing something like

$mh = curl_multi_init();
for ($i = 0; $i < $desiredThreadsNumber; $i++) {
    $ch = curl_init();
    // set up $ch here
    curl_multi_add_handle($mh, $ch);
}

You should understand that you haven't run threads yet here. curl_multi_exec() runs all threads. But it can't run all $desiredThreadsNumber threads simultaneously. If you look on example on curl_multi_exec() php.net page, you will see that you must wait while curl_multi_exec() run all threads. In other words, you need next nested loop here:

$running = null;
do {
    do {
        $mrc = curl_multi_exec($mh, $running);
    } while ($mrc == CURLM_CALL_MULTI_PERFORM);
} while($running > 0);

At the end let me suggest you to read this article http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/ and use code snippet from there, I used it in 2 or 3 projects.

share|improve this answer

You could add "sleep 1", which sleeps for one second, into the loop.

share|improve this answer
Did you mean sleep(1)? – todofixthis Jun 22 '11 at 4:36

Try:

 $running = null;
    do {
        do {
            $mrc = curl_multi_exec($mh, $running);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM && curl_multi_select($mh) === 0 );
    } while($running > 0  && $mrc == CURLM_OK );
share|improve this answer

curl_multi_select (http://php.net/manual/function.curl-multi-select.php) is indeed the way to go, but there are a couple caveats.

First, if curl_multi_exec returns CURLM_CALL_MULTI_PERFORM, it has more data to be processed immediately, so should be run again. Also, it is important to check that curl_multi_exec did not fail immediately; in that case curl_multi_select could block forever.

This should work:

do {
    while (CURLM_CALL_MULTI_PERFORM === curl_multi_exec($mh, $running)) {};
    if (!$running) break;
    while (curl_multi_select($mh) === 0) {};
} while (true);

If anyone sees a good way to avoid the while(true) without duplicating code, please point it out.

share|improve this answer

Your Answer

 
or
required, but never shown
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.