fix(model cache): make_room honors the admission grace; grace owned by its LoadedModel handle - #193
Merged
Merged
Conversation
make_room's eviction loop checked only is_locked, but multi-model invocations load their whole set before locking any of it (the H3 text encoder loads text_encoder, then tokenizer, then processor, then locks) — so a sibling's cold-load make_room runs with a just-admitted entry unlocked, and under shared-RAM-budget deficit it evicted the 27GB text encoder admitted one second earlier to make room for a 50MB tokenizer. Evicting such an entry frees NOTHING (the loader's handle keeps the model alive) while detaching the record from all cache accounting and turning every subsequent paced-lock pass into an issue-7513 diagnostic line (observed: ~25 identical INFO lines per text-encoder load on a dual-GPU rig). The asynchronous eviction paths (budget reconcile, peer eviction) already honored awaiting_first_use; the synchronous path now does too. The keep-alive timeout clear passes spare_awaiting_first_use=False: after an idle period a surviving grace is abandoned by definition (a healthy loader locks within seconds, and every lock resets the timer), so the timeout keeps clearing everything unlocked. Also demote continue_lock's detached-record diagnostic to DEBUG — lock() already reports it once at INFO, and a paced stream repeated the same line dozens of times. Two existing tests exercised make_room on entries that had never been locked; they now release the grace first, the way every real loader does. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
Adversarial review of the previous commit proved its guard incomplete:
put()'s stale-grace sweep cleared EVERY grace on each admission, so only
the most recently admitted sibling was ever protected, and only until the
next put() — both reproduced paths re-created the field failure (a
just-admitted 27GB model evicted by a sibling's cold load) with the guard
in place. The sweep's premise ('a grace surviving to the next admission is
stale') is false in exactly the window the grace exists for: multi-model
invocations load their whole set before locking any of it.
The grace's owner is the LoadedModel handle, which already releases it via
a finalizer on an unlocked drop. Record a weakref to the handle on the
CacheRecord (grace_holder) at handle construction; the sweep now clears
only provably orphaned graces — no handle was ever registered (the load
raised between put() and LoadedModel construction) or the handle died
without its finalizer running (lost deferred worker). Live holder = the
loader is coming back; lock() or the finalizer releases the grace.
Also per review: the clear-model-cache button (/empty_model_cache) passes
spare_awaiting_first_use=False through the now-parameterized public
make_room — a user-requested full clear outranks the grace (pre-existing
behavior restored) — and the tautological assertion in the new test file
is replaced by real coverage: grace survival across sibling admissions,
orphan sweeps both ways, finalizer release, and the explicit-clear path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
…ond review - The finalizer's release now carries the dying handle's weakref: when the record's grace has since been re-registered to a newer, still-live handle, the release leaves it alone (single-slot limitation documented — no current code path constructs two pre-lock handles for one record). - The deferred worker unpins the unpacked CacheRecord local before blocking on the next queue item (the existing anti-pinning test caught the tuple unpacking keeping the record alive). - The /empty_model_cache route tests now assert the spare_awaiting_first_use=False plumbing, not just tolerate the kwarg. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
lstein
requested review from
JPPhoto,
Pfannkuchensack and
blessedcoolant
as code owners
August 31, 2026 02:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
make_room's eviction loop checked onlyis_locked, but multi-model invocations load their whole set before locking any of it (the MiniMax H3 text encoder loads text_encoder → tokenizer → processor, then locks). Under shared-RAM-budget deficit on a dual-GPU rig, the tokenizer's cold-loadmake_roomevicted the 27 GB text encoder admitted one second earlier — freeing nothing (the loader's handle keeps the model alive) while detaching the record from all cache accounting and printing the issue-7513 diagnostic once per paced-lock pass (~25 identical INFO lines per text-encoder load, observed in the field).Fix (three commits, each shaped by an adversarial review of the previous)
make_roomspares entries inside their admission grace (awaiting_first_use). The async eviction paths (budget reconcile, peer eviction) already honored it; the synchronous path now does too. The keep-alive timeout clear and the/empty_model_cachebutton passspare_awaiting_first_use=False— an idle-expiry or user-requested full clear outranks the grace.LoadedModelhandle. Review of commit 1 provedput()'s stale-grace sweep defeated the guard: it cleared every grace on each admission, so only the newest sibling was protected, and only until the nextput()— the field failure reproduced with the guard in place. The handle registers a weakref (CacheRecord.grace_holder) alongside its existing release finalizer; the sweep now clears only provably orphaned graces (no handle ever registered, or the handle died without its finalizer running). Live holder ⇒lock()or the finalizer will release it.spare_awaiting_first_use=Falseplumbing.Also demotes
continue_lock's detached-record diagnostic to DEBUG —lock()reports it once at INFO; a paced stream repeated the identical line dozens of times.Verification
tests/backend/model_manager/load/and the router tests; new coverage: grace survival across sibling admissions (fails on commit 1 — verified), orphan sweeps both ways, finalizer release on handle drop, holder-identity release, explicit-clear override, plus the pre-existing eviction guards.🤖 Generated with Claude Code
https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A