proposal for remotable mutable caches like the 1.x jvm backend

[cosmicexplorer]

Remotable Mutable Caching

I'm looking to see whether we could support an optimization to avoid snapshotting process execution output files/directories that we already know about. This would be useful if we wanted to start snapshotting mutable caches to make them remotable, without requiring any specific support in the remexec API.

Background

digest_hint

It looks like we don't set the digest_hint in PathGlobsAndRoot anywhere:

pants/src/python/pants/engine/fs.py

Line 182 in 237e6a6

digest_hint: Optional[Digest] = None

It looks like we implemented that optimization in a one-off manner in several places in the 1.x jvm backend:

pants/src/python/pants/backend/jvm/tasks/resolve_shared.py

Lines 32 to 41 in 31cdbc8

	# Capture Snapshots for jars, using an optional adjacent digest. Create the digest afterward
	# if it does not exist.
	snapshots = self.context._scheduler.capture_snapshots(
	tuple(
	PathGlobsAndRoot(PathGlobs([jar]), get_buildroot(), Digest.load(jar),)
	for jar in jar_paths
	)
	)
	for snapshot, jar_path in zip(snapshots, jar_paths):
	snapshot.digest.dump(jar_path)

Symlink Optimization

See this comment on the mutable caches doc (https://docs.google.com/document/d/1n_MVVGjrkTKTPKHqRPlyfFzQyx2QioclMG_Q3DMUgYk/edit?disco=AAAAIva7gMw):

I didn't realize we weren't snapshotting these caches. It seems like one way to avoid having any upstream support for this would be to implement something like this PR (#8905), which maintains symlinks to read-only materialized files.

We wouldn't need to modify the Process struct to support this. Rather, we could implement the more general utility of "being able to provide symlinks to some snapshotted file or directory in local executions", then consume that in the exact way we already do, except that we would additionally then snapshot the caches at the end of the execution. This shouldn't introduce too much overhead, as we would only end up snapshotting any new entries. We can then introduce the same caches as part of the input digest to the remote execution request, and extract them from their known relative paths at the end of the request.

Luckily, we already do exactly this (write symlinks into the process execution dir) for local mutable caches. This proposal describes an extension of that idea which allows them to be remote-friendly without upstream support from BuildBarn/etc.

Proposal: Snapshot our Mutable Caches

Because we have already successfully employed this method in the 1.x jvm backend, it seems reasonable to expect it will work again, especially now that we have a specific mutable cache concept. The idea I'm thinking of is:

• to automatically write a digest_hint file after snapshotting any directory outside of the buildroot.
  • We can do this recursively for each subdirectory, assuming as before we are using a global append-only cache.
• to automatically check for a digest_hint file before expanding globs into any directory outside of the buildroot.
• after a local process execution, to snapshot every mutable cache directory, and take advantage of every digest_hint file to avoid expanding globs into any existing directory.
  • This would not change anything that happens before the local process execution, as we already write symlinks to mutable cache directories.
• make the caches remotable:
  • Before a remote process execution, merge the digests for each mutable cache directory into the input_digest field of the remote execution proto.
  • After a remote process execution:
    • extract the mutable cache directory digests from the output_digest field.
      • If any cache digests are different than before, recursively traverse the digest and materialize all the new cache entries into the corresponding mutable cache directory on the local disk.
    • subset the output_digest field to remove the mutable cache directories.

Possible Extensions

I could also see the above being useful outside of that context. In particular:

• Anywhere that pantsd needs to be restarted (e.g. in pantsd integration testing) we could write such digest hint files within source file directories to avoid having to trawl the entire pants repo again. This might reduce the time it takes to run such tests.
• As postulated in the same comment above, it seems possible to avoid the complex and lengthy workaround surrounding the exclusive_spawn argument for local process executions by expanding the number of things we can write symlinks for:

Looking again at the exclusive_spawn argument to CommandRunner::run_in_workdir(), it seems possible that the symlink optimization for materialized files could also be employed whenever we detect that argv[0] points inside the input_digest -- we could extract the file digest from the directory proto, materialize it into a canonical location (if this wasn't already done), then write it out as a symlink into the process execution sandbox before attempting to execute the process.

• MyPy uses non-append-only cache directories (see Improve MyPy performance #10864). We could extend the approach above to solve this by simply snapshotting the entire cache directory after each local process execution (instead of relying on digest_hint files).

[cosmicexplorer]

Hm. After reviewing other open issues, I think it would likely be much more generally useful to avoid dropping digest_hint files anywhere, and instead resort to e.g. polling or the notify crate as described in #10842 (comment).

The implementation that makes caches remote-friendly (merging them into the input_digest, and extracting from the output_digest) I think should still work 100% with the approach outlined in #10842.

[cosmicexplorer]

Ah! After thinking a bit further about it, I think the right description would be:

• For append-only caches (like coursier, as above), we can drop digest_hint files as described above to avoid re-traversing anything we've already snapshotted.
• For non-append-only caches (like MyPy, as in Improve MyPy performance #10864), we would use polling or the notify crate, as an extension of Add an intrinsic for capturing Snapshots from absolute paths #10842.

Either way, we would make these caches remote-friendly by merging them into the input_digest, and extracting from the output_digest when executing remotely, which does not require special support from the remexec backend.

• Eventually (as an optimization), if we can get the remexec API to agree that the remexec backend must recognize our digest_hint files, we can avoid that re-traversal on the remote end as well.

[cosmicexplorer]

As @stuhood noted in this comment on the mutable caches doc:

It's not as bad as it could be, sure. But a cold/first-time build on a user's machine will mean that they have to start by downloading lots-of/all-of-the artifacts.

If you assume an implementation like "Always+Only Run Resolver Processes Locally", then you could use the mutable caches locally, but all new artifacts would always need to be downloaded first. You would not need to re-upload them in most cases, because they would already exist on the cluster due to other clients, but.

I recall the performance issue there wasn't from uploading coursier jars (which remained relatively static), but in shipping compile output back and forth

You never need to download compile outputs if the client doesn't want them... I expect that if we were doing this is was because of the v1 APIs.

So if we wanted an "instantly responsive" experience, having to sync all of the mutable cache all of the time might be difficult (and might put a ton of bytes on the user's machine that they actually don't need).

One possible extension to the above would be to avoid downloading the output digest corresponding to the cache entries. This digest could still be extracted from the output_digest of the remexec protobuf, but when we reach the cache subdirectory in our recursive digest download process, we can just stop there (while recursing into the non-cache output paths). Then, we can provide that digest as a subdirectory to the next input_digest we use for the same request.

Basically, if we're already considering some hacky workaround to make use of the existing CAS API to sync our append-only caches, we might as well also consider another workaround to avoid actually recursing into that cache subdirectory.

This would allow us to have our instant responsiveness (not downloading the entire cache dir), while still being able to maintain a mutable cache concept in the remote cluster. It would also however mean we would start keeping digests around which don't actually exist, which is somewhat dicey.

[cosmicexplorer]

I think the above is actually clear enough to prototype, but it's very very complex. So I'm not sure I'll be looking at this quite yet. If we find it takes too long to wait for platform properties to support mutable caching, then it might be good to look back at this.

[stuhood]

#12716 added support for immutable inputs to processes, which are materialized once as readonly, and then reused. It has removed materialization from the hot path for the JVM, and is likely to be sufficient to support very large sandboxes in other cases as well (including for mypy).

Thanks a lot for having sketched this! While we might revisit portable+mutable caching in the future, I don't think that it is likely any time soon.