Tell me more ×
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It's 100% free, no registration required.
up vote 7 down vote favorite
1

I've been having problems designing an effective caching mechanism. The mechanism is to be used as part of a project developing scientific software.

The goal is to save computation time by loading results of identical computations done in the past.

The problem I've been encountering is how to know if a computation is identical to a previous one. Checking the input parameters can be sufficient in most cases, but sometimes the code changes, and then even though the input parameters can be identical, the results may differ.

The problem is that modifying the code now needs to be coupled with clearing the cache, in case that code creates cached results. This is a manual step which is error-prone. Failure can cause the results to stay the same, even after code changes have been made, leading to bugs or erroneous conclusions.

I was wondering if there is any mechanism for caching which can solve, or somehow help avoid this pitfall.

Thanks, Ofer

share|improve this question
Hashing the current source code - is a great idea, but how do you implement it? Discovering which files are used by a function seems difficult for most languages I can think of (I use Matlab). – Ofer Apr 28 at 8:25
Ofer, you can ask it at Stack Overflow... This needs some reflection capabilities in the language. – Deer Hunter Apr 28 at 16:19
@DeerHunter, I've actually found such a function in MATLAB called depfun – Ofer May 1 at 20:32

2 Answers

up vote 5 down vote accepted

The answer depends, in part, both on how robust of a solution you're looking for and on what kind of development workflow you use.

One particularly simple option is to tag each cache entry with the version number of the software. As long as every change to the code is accompanied by a change in version number, this will give you an easy way to check whether a cache file is still valid. However, version numbering is probably only appropriate if changes to the project are committed all at once.

A less robust option that is nonetheless more suited for continuous development would be to simply compare the last-modified dates for your [oldest] cache files and your source files. If the source files were last modified before all of the cache files, the latter should still be usable.

@DeerHunter pointed out a technique that is more robust than either of the above options and should work regardless of your development methodology.

When creating a cache file, include a hash of the current source code. At the start of program execution, hash the current source code and compare the result to the hash contained in each cache file. If they don't match, the file was produced by a different version of the software and should be removed.

A similar procedure can expedite comparisons between input parameters. Store a hash of the input parameters in each cache file, and then compare that hash to a hash of the current input parameters to check for a match. The only downside here is that you can no longer look for cache files with input parameters that are close to those of the current simulation.

share|improve this answer
1  
John Delong, you can consider editing your answer to include hashing the source code and the inputs and then comparing the two hashes with their previous versions. Much better in terms of conflict resolution, avoiding false alarms etc. – Deer Hunter Apr 27 at 18:07
@DeerHunter, you make a great point. I had considered mentioning hashing the source code, but I naively dismissed the idea as too computationally intensive. Of course, provided that one does not use something inappropriate like a cryptographic hash, this is an excellent idea. – John Delong Apr 27 at 18:18
1  
Compared to the time necessary to re-run a calculation, hashing is quite OK. Weak formerly-cryptographic hashes like MD5 are fast enough to compute and sport low probability of collisions. My impression is that 32 bit or 64 bit hashes are too small for many memoizing applications. – Deer Hunter Apr 27 at 18:27
4  
There are several projects in research reproducibility that are considering this question and broader issues. The list is quite long, so I'll point you to one, sumatra. It's Python-based, but hopefully it can help you find similar packages in the MATLAB space. – Aron Ahmadia Apr 28 at 21:51
I have an implementation of the second suggestion that you should checkout. I originally wanted to serialize all the params into a store of some variety, but I realized that would require keeping a whole metaData cache on the side which has its own headaches. I opted for md5 pureley because the output hex was small enough to fit cleanly as a file name. :P – meawoppl Apr 29 at 17:23
show 2 more comments
up vote 2 down vote

I have a Python implementation that satisfies the interest you specify. Take a look at it to see a simple way to do this in languages capable of introspection.

Any function that you decorate with it will be run once, and the results cached locally. Subsequent invocations with:

  1. the same arguments
  2. the same function code

will be loaded from disk.

It uses a different storage mechanism if the result is an array for faster saves and loads.

This would be a bunch harder to write in c/c++ and similar. Though a simple trick might be salting the hash with a pre-processor directive such as TIME or DATE or some such if you want to err on the side of tossing your cached results after a recompile.

share|improve this answer
1  
This is pretty fragile: it compares equality using str() (rendering functions as arguments useless, for example), names can collide, you can't return functions and other non-pickleable objects, and transitive dependencies are not versioned, to name a few issues. – Jed Brown Apr 30 at 1:21
That's pretty neat. I like the decorator idea. I might implement a Matlab one (if no one here happens to have a Matlab implementation by any chance?). – Ofer May 1 at 20:05
A fair critique @JedBrown. I will add the passed function hashing bit later today, but dealing with non-pickleable datatypes seems out of scope. If thats important, you can write a pickling routing for it etc. Functional returns would be really hard with this approach. I wrote this for a function that needed to generate something like 100 floats worth of data from each several GB datasets it was run on, not as a catch-all solution really... It works great for that sort of thing :) – meawoppl May 2 at 20:58
Also notably, it is posix specific in its temp file path, so it will probably not run correctly on windows. – meawoppl May 2 at 20:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.