gw#599: managed-tier fill-source hierarchy (R2 -> volume -> local NVMe)#342
Merged
Conversation
Paul's managed-tier ruling (th#850) supersedes the CAS-root-on-volume shape gw#597 shipped: the CAS root stays on local/pod-local storage as a managed, bounded LRU tier. An attached RunPod endpoint volume is FILL SOURCE #1 (checked before R2, FILL SOURCE #2); a fill from R2 writes through to the volume so the next same-endpoint pod finds it warm. - cache_paths.py: tensorhub_fill_source_dir() resolves TENSORHUB_FILL_SOURCE_DIR, ismount-guarded (same pattern as the removed endpoint_shared_cas_dir()). tensorhub_cache_dir()/tensorhub_cas_dir() stay local-only, always. - cozy_snapshot.py: per-digest fetch order in _ensure_blobs becomes local CAS -> fill source (BLAKE3-verified copy) -> R2 (existing _dl_locked); a fresh R2 fetch writes through to the fill source, best-effort. Threaded as an explicit fill_source_dir parameter through ensure_snapshot/ensure_snapshot_async (same shape as the removed shared_base_dir plumbing). NetworkBytesScope (a ContextVar, same idiom as provision.py's ArmingScope) captures bytes fetched from R2 for one ensure_local call with zero signature changes anywhere else. - executor.py: ModelStore resolves _fill_source_dir at construction (env by default, explicit override for tests); _materialize_local wraps its download with NetworkBytesScope and hands the result to _on_residency_event via a small pending-dict, so the ON_DISK ModelEvent Residency already emits for a fresh materialization carries network_bytes — no second event, no EventFn arity change (Residency has other direct callers). gc_disk's eviction ordering is extracted into a swappable _disk_eviction_order seam (default behavior unchanged, byte-for-byte); scheduler-intent pinning is an explicit follow-on, not built here. - proto/worker_scheduler.proto: additive ModelEvent.network_bytes = 20, version-gated (absent/zero on old workers = identity unknown, same convention as snapshot_digest/residency_generation). pb2 regenerated with the pinned grpcio-tools==1.82.1 (matches pyproject, not the stale venv default). - Companion tensorhub PR: cozy-creator/tensorhub#494 revises th PR #489 to set TENSORHUB_FILL_SOURCE_DIR (not TENSORHUB_CACHE_DIR) on volume-attached pod create. Tests (tests/test_managed_tier_fill_source.py): volume blob preferred over R2; R2 fetch writes through to volume; no fill source configured is byte-identical to pre-th#850 behavior; corrupt volume blob (right size, wrong content) falls through to R2 rather than being trusted; fill_source_dir ismount guard; network_bytes reaches the wire ON_DISK event for both a cold R2 fetch and a warm volume hit (0).
…ot just the terminal event Found while cross-checking against the sibling tensorhub th#850 lane's PR #493 (EndpointVolumeUsageStore / observeEndpointVolumeNetworkBytes): the hub reads network_bytes off the DOWNLOADING events' running value (WorkerModelDownloadState.NetworkBytes, populated the same way as BytesDownloaded/BytesTotal), not the terminal ON_DISK event. Two fixes: - NetworkBytesScope's sink now updates live inside _ensure_blobs's _on_bytes, chunk by chunk, instead of a single flush after the whole call finishes — the previous shape meant every mid-flight DOWNLOADING tick always read 0. - executor.py's _progress callback now includes network_bytes=net_scope.network_bytes on every DOWNLOADING event, reading net_scope (opened once, before _progress is defined, entered for the whole retry loop) live. New test: test_network_bytes_is_a_running_total_on_downloading_ticks — disables the progress debounce, streams a fake download in chunks, and asserts at least one DOWNLOADING event carries a PARTIAL (not 0, not final-total) network_bytes.
Follow-up accuracy pass: gw#599 made the CAS root local-only again (the volume is a fill source, not the root), so comments still framing it as "a persistent volume shared by several pods" by default were misleading.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cache_paths.py: newtensorhub_fill_source_dir()(TENSORHUB_FILL_SOURCE_DIR, ismount-guarded).tensorhub_cache_dir()/tensorhub_cas_dir()stay local-only, unchanged.cozy_snapshot.py:_ensure_blobsfetch order becomes local CAS -> fill source (BLAKE3-verified copy) -> R2; a fresh R2 fetch writes through to the fill source (best-effort).fill_source_dirthreaded explicitly throughensure_snapshot/ensure_snapshot_async(mirrors the removedshared_base_dirshape).NetworkBytesScope(aContextVar, same idiom asprovision.py'sArmingScope) captures R2-fetched bytes with zero signature changes elsewhere.executor.py:ModelStoreresolves_fill_source_dirat construction;_materialize_localwraps its download and handsnetwork_bytesto_on_residency_eventvia a small pending-dict so Residency's ownON_DISKevent carries it (no duplicate event, noEventFnarity change).gc_disk's eviction ordering is extracted into a swappable_disk_eviction_orderseam — default behavior byte-for-byte unchanged; scheduler-intent pinning is an explicit follow-on, not built here.proto/worker_scheduler.proto: additiveModelEvent.network_bytes = 20, version-gated. pb2 regenerated with pinnedgrpcio-tools==1.82.1.Test plan
tests/test_managed_tier_fill_source.py(new): volume blob preferred over R2; R2 fetch writes through to volume; no fill source configured is byte-identical to pre-th#850 behavior; corrupt volume blob (right size, wrong content) falls through to R2;tensorhub_fill_source_dir()ismount guard;network_bytesreaches the wireON_DISKevent for both a cold R2 fetch and a warm volume hit (0)tests/test_disk_gc.py,test_shared_cas_root_multiwriter.py,test_snapshot_corruption.py,test_snapshot_remint.py,test_residency.py,test_worker_grpc_e2e.py,test_executor_adopt.py,test_ram_admission.py,test_declarative_residency.py,test_config_parse_cross_era.py— all green--ignoreon CUDA-gated files that attempt real GPU work) — 1425+ passed, 0 failed attributable to this change (two intermittent full-suite-only timeouts reproduced as pre-existing flakes: both pass cleanly in isolation, unrelated files/code)ruff check src/clean;mypyclean on touched modulesNo cloud resources created or mutated. Behavior is unchanged whenever no fill source is configured (cozy-local, and every pod until tensorhub#494 deploys).