variable_get() using “default value” with no reason in Pressflow 6

If you look at the code for variable_init() used by Pressflow, you will notice the following:

• If the MAINTENANCE_MODE constant is defined, it tries to acquire the "variable_cache_regenerate" lock; this means the code will not be executed until lock_acquire() doesn't return TRUE.
• In the case lock_acquire() doesn't return TRUE, the function will call lock_wait('variable_cache_regenerate'), and then recursively call itself.
• After 49 recursive calls, the function set the $variables variable to an empty array; doing so, all Drupal variables are set to their default values.

• The comment in the function says:

  Try a limited number of times to avoid // infinite recursion if the database connection is invalid for
  // some reason, e.g., mysqld restart, loss of network, etc.

This behavior is not present in the function used by Drupal 6, where variable_init() doesn't use any lock.

Similar code is used from Drupal 7, even though there is a slight difference: Drupal 7 reads the values of the Drupal variables from the database, while Pressflow doesn't do that. Compare the following snippets; the first one is used in Drupal 7, while the second one is used in Pressflow 6.

// Cache miss. Avoid a stampede.
$name = 'variable_init';
if (!lock_acquire($name, 1)) {
  // Another request is building the variable cache.
  // Wait, then re-run this function.
  lock_wait($name);
  return variable_initialize($conf);
}
else {
  // Proceed with variable rebuild.
  $variables = array_map('unserialize', db_query('SELECT name, value FROM {variable}')->fetchAllKeyed());
  cache_set('variables', $variables, 'cache_bootstrap');
  lock_release($name);

  // Wait for another request that is already doing this work.
  lock_wait('variable_cache_regenerate');

  // Run the function again. Try a limited number of times to avoid 
  // infinite recursion if the database connection is invalid for  
  // some reason, e.g., mysqld restart, loss of network, etc.
  $recursion_depth++;
  if ($recursion_depth < 50) {
    return variable_init($conf, $regenerate, $recursion_depth);
  }

  $variables = array();

Hmmm... It's going to be a tough one to track down - unless someone here has some good insight or experience of this...

When I've seen similar behaviour on my sites it was when using the (still broken) file cache on the Cache Router module which had (has) big reliability issues.

Since then I've also seen white Panels pages on my Memcache-enabled sites, I believe this was caused when the cached panels pages got too big to fit in the 1Mb 'buckets' memcache uses by default.

So it's possible your variables table has got too big to safely store in a standard 1Mb bucket, so you might have to make the buckets bigger, or maybe try the Variable Cleanup module. Did this start after you installed a poorly behaving module, perhaps?

I reckon it could also be a number of other things too - here's some ideas to look into:

• Old/faulty Drupal Memcache module.
• Silent errors/failures when data is set to or got from the (mem)cache.
• code further up in execution erroneously writing a value to the variable.
• Strongarm or another module override of variable values
• Some other OS or debugging reason that means the variables table didn't get loaded or cached properly - what I don't know!

Good luck!