How can I improve the performance of this concurrent http request loop?

I don't see what could be causing ever-decreasing performance since the only state outside the main loop is a counter, but adding calls to memory_get_usage as @jsanc623 recommends would help with that.

But you can definitely improve the performance by avoiding the loop over $results for every URL returned. Take advantage of PHP's array which is a hash table providing O(1) lookup.

First, make the URL the key for the results array.

foreach ($items as $item) {
    if ($item->img !== '') {
        $url = $item->img;
        $results[$url] = array(
            'id' => $item->id, 
            'url' => $url,
            'size' => null
        );
        $rollingCurl->get($url);
    }
}

Next, use isset instead of array_search and looping.

// foreach ($results as $key => $value) { ... } becomes

$url = $request->getURL();
if (isset($results[$url]) {
    $results[$url]['size'] = $length;
    File::append($filePath, $results[$url]['id'] . ',' . $length . "\r\n");
}

Since you're placing each URL into the array before making the GET request, the if isn't even necessary.

Separately, you can delay writing to the file until the end of the inner loop so you write one big chunk rather than one thousand tiny chunks. It may be that I/O is more costly, and perhaps you're getting killed by seeking to the end of an ever-growing file? Seems unlikely but maybe after a million writes it adds up.

General Review

There are a few things that could be cleaned up here as well.

• Consistent formatting: put your opening brace on the same line or the next, but do it consistently.
• Blank lines before the closing brace seem misplaced to me. Use blank lines to logically group related code, though functions are preferred to whitespace.

• Simplify testing if the image URL exists

  if ($item->img !== '')
  

  can be simplified to

  if ($item->img)
  

  to skip null values as well. Even if the field cannot be null, it's still easier to read.

Be sure to disable Laravel's query logging and filter all unwanted results directly with your SQL query. Also use a stream to write lots of small chunks of data, it's much more efficient (and what streams where made for).

Here's a possible solution, would love to see the benchmarking results for this one.

<?php

class CodeReview {

  /**
   * Keep track of the Laravel chunk we're processing.
   *
   * @var integer
   */
  protected $counter = 1;

  /**
   * File handle of the current CSV file (writing only).
   *
   * @var resource
   */
  protected $fh;

  /**
   * Array to map image URLs to IDs.
   *
   * @var array
   */
  protected $urlToId = array();

  /**
   * Used for caching only.
   *
   * @var string
   */
  protected $storagePath;

  /**
   * ...
   *
   * @return this
   */
  public function fire() {
    // Directly format current time and embed variable.
    $date = \DateTime::format("Y-m-d H:i:s");
    Log::info("Started: {$date}");

    // Get storage path once (why is this even a function?).
    $this->storagePath = storage_path();

    // Disable Laravel's query log, otherwise it will eat up our RAM.
    //
    // @link http://laravel.com/docs/database#query-logging
    DB::connection()->disableQueryLog();

    // Execute the query and filter all unwanted results right away. Don't use closures because they are slower than
    // native methods.
    Item2::whereNotNull("img_size")->where("img_size", "<>", "''")->chunk(1000, array($this, "chunkedWhereCallback"));

    return $this;
  }

  /**
   * Callback for the Laravel chunked WHERE query method.
   *
   * @internal Must be public in order to be accessible for PHP's call_user_func() function.
   * @param array $results
   *   The chunked results from the Laravel WHERE query.
   * @return this
   */
  public function chunkedWhereCallback($results) {
    // Open file handle to CSV file.
    $this->fh = fopen("{$this->storagePath}/imports/new/new_img_sizes_{$this->counter}.csv", "w");

    // Instantiate new rolling cURL and configure it.
    $rollingCurl = new \RollingCurl\RollingCurl();
    $rollingCurl->setOptions($this->curlOptions);

    // Micro benchmark
    $start = microtime(true);

    /* @var $result \Namespace\Of\Stub\Class\ClassName */
    foreach ($results as $result) {
      // Build hash array for easy lookup.
      $this->urlToId[$result->img] = $result->id;

      // Add this image to the cURL queue.
      $rollingCurl->get($result->img);
    }

    // That's it, we added all images to the array, let's fetch them. Again we use a native method and no closure.
    $rollingCurl->setCallback(array($this, "rollingCurlCallback"))->setSimultaneousLimit(10)->execute();

    // Finished handling chunk, close CSV handle and increase counter.
    fclose($this->fh);
    ++$this->counter;

    $end = microtime(true) - $start;
    Log::info("1000 records: {$end}");
    $last = end($this->urlToId);
    Log::info("Last URL was: {$last}");

    return $this;
  }

  /**
   * Write image size to CSV file.
   *
   * @param \RollingCurl\Request $request
   *   The rolling curl request.
   * @return this
   */
  public function rollingCurlCallback($request) {
    $size = $request->getResponseInfo()["download_content_length"];
    $url  = $request->getURL();

    if (fwrite($this->fh, "{$this->urlToId[$url]},{$size}\r\n") === false) {
      throw new \RuntimeException("Couldn't write to CSV file!");
    }

    return $this;
  }

}

• Format your code in a consistent fashion (stick to PSR-2 rules if you're unsure)
• Try to document your code where it makes sense for you, or for a developer that might have a look at it later
• Closures hurt readability, maintainability and performance (some may disagree with this, but in your case they definitely do because they are too long)
• Return $this by default if you have no return value (like in jQuery) and allow chaining of all your methods by default
• Forget about type checks if performance is all you want (they only add overhead); but don't forget that they are great for public APIs and other stuff (like you get it if you use a framework), it makes things easier for other developers

Give this a try and post the results - it integrates PHPBenchTime laps and get_memory_usage() with your script.

require('PHPBenchTime.php');
use PHPBenchTime\Timer as Timer;

public function fire(){
    $Benchmark = new Timer;
    $Benchmark->Start();

    $dt = new DateTime();
    Log::info("started: ".$dt->format('Y-m-d H:i:s'));
    $counter = 1;

    Item2::where('img_size', '=', NULL)->chunk(1000, function($items) use ( &$counter){
        $results = array();
        $filePath = storage_path().'/imports/new/new_img_sizes_'.$counter.'.csv';

        if (!File::exists($filePath)) {
            File::put($filePath, '');
        }
        $rollingCurl = new \RollingCurl\RollingCurl();
        $rollingCurl->setOptions($this->curlOptions);

        foreach ($items as $item){
            if ($item->img !== '') {
                $results[$item->id] = array('url' => $item->img, 'size' => null);
                $rollingCurl->get($item->img);
            }
        }

        //callback runs on each curl request
        $rollingCurl->setCallback(function(\RollingCurl\Request $request, \RollingCurl\RollingCurl $rollingCurl) use (&$results, $filePath) {
            $responseInfo = $request->getResponseInfo();
            //var_dump($responseInfo);exit;
            $length = $responseInfo['download_content_length'];

            foreach ($results as $key => $value) {
                if (array_search($request->getURL(),$value)) {
                    $idKey = $key;
                    $results[$idKey]['size'] = $length;
                    File::append($filePath,$idKey.','.$results[$idKey]['size']."\r\n");
                    break;
                }
            }
        })

            ->setSimultaneousLimit(10)
            ->execute();

        Log::info('Last url was: '.json_encode(end($results)));

        $Benchmark->Lap("Lap " . $counter . " mem " . memory_get_usage() . " bytes");

        $counter++;

        exit;
    }); // end item chunk


    $time = $Benchmark->End();
    print_r($time);
}