gw#597+gw#598: multi-writer CAS safety + digest-verified reuse for volume-backed roots#339
Merged
Merged
Conversation
th#850 moves the endpoint-volume design from a separate shared-read-tier (gw#277) to simply pointing the worker's existing CAS root (TENSORHUB_CACHE_DIR) at the mounted RunPod volume when one is attached. The first pod's ordinary R2 prefetch warms it for later same-endpoint pods -- no seed/publish/read-through split needed, so that plumbing is removed from cache_paths.py/download.py/cozy_snapshot.py. That collapse exposed a real bug: the primary blob-download temp path (_download_one_file_sync) staged to a fixed ".part" name, safe only because a pod-local CAS root has exactly one writer. Once the CAS root is a volume shared by concurrent same-endpoint pods, two pods racing on the same missing digest would collide on that name. Same gap in _materialize_snapshot's ".building" tmp dir. Both now stage to a writer-unique temp name + atomic replace/rename, matching the pattern the (now-removed) shared-tier helper already used correctly. Since a persistent volume (unlike ephemeral pod-local /tmp) keeps a crashed writer's orphaned temp forever, added a boot-time sweep (disk_gc.sweep_stale_writer_temp, wired into ModelStore.rescan_disk) and hardened sweep_orphan_blobs to skip dotfile temp artifacts instead of treating an in-flight writer's temp as an orphaned blob. Proof: 4-process spawn tests racing the real download and materialization paths against one shared destination (test_shared_cas_root_multiwriter.py). executor.py's existing gc_disk/_ensure_disk_headroom LRU disk-GC needed no changes -- it already operates on whatever cache_dir is mounted, satisfying th#850's per-volume size-cap design as-is. Full suite: 1518 passed, 35 skipped, 0 failed. ruff + mypy clean.
…t per boot The fresh-materialization path trusted bytes it did not download: an already-materialized snap_dir short-circuited with no check at all, and _blob_usable reused existing blobs on a size-only check. Safe when the CAS root's only writer was this pod; on a th#850 shared persistent volume those are another pod's bytes of any age. Fix lives in the CAS layer (cozy_snapshot), not the executor -- the reuse decision and its verification belong together, stubbed-download tests are untouched, and genuinely fresh downloads are never hashed twice. Mechanics: process-local trust sets keyed by (resolved root, digest/key). Reused blobs get a full BLAKE3 once per root+digest per process (_blob_trusted; corrupt -> unlink -> re-download); a reused materialized tree is verified once per root+key per process (_verify_materialized_tree: manifest size+blake3 + structural safetensors for reassembled originals; corrupt -> quarantine tree + bad blobs -> rebuild). A freshly built tree is trusted without a second pass since every input was verified this process. The executor's existing first-use-per-boot verify for residency-tracked refs is unchanged and now backed by the same guarantee underneath. Chosen tradeoff (recorded in gw#598): verify-on-first-use-per-process, NOT re-hash-every-boot -- warm-volume boot pays one hash pass over the model (the intended defense), cold boots pay nothing extra. Test: test_preseeded_unverified_cas_bytes_are_verified_before_trust -- corrupt right-size blob hardlinked into a materialized tree for a ref residency never saw; ensure_local must detect, quarantine, re-download, and repair the shared blob. test_disk_gc fixtures now declare truthful digests of their fake bytes. Full suite 1519 passed / 0 failed.
PaulFidika
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two tightly-coupled fixes for th#850's CAS-root-on-volume design (one wave, same files):
gw#597 — multi-writer safety + shared-tier removal
TENSORHUB_CACHE_DIR) directly at the mounted RunPod volume — removesENDPOINT_SHARED_CACHE_MOUNT/endpoint_shared_cas_dir()/shared_base_dirplumbing._download_one_file_sync's temp path and_materialize_snapshot's.buildingdir were not writer-unique (safe only with one writer per CAS root). Both now stage to writer-unique names + atomic replace/rename.sweep_orphan_blobshardened to skip in-flight dotfile temps.gw#598 — digest-verified reuse (BLAKE3 gap)
snap_dirshort-circuited with no check, and_blob_usablereused blobs on size alone. On a shared volume those are another pod's bytes of any age.Test plan
Cross-refs: th#850 + gw#597 + gw#598 in the trackers; tensorhub counterpart PR: https://github.com/cozy-creator/tensorhub/pull/489