Skip to content

fix(model cache): make_room honors the admission grace; grace owned by its LoadedModel handle - #193

Merged
lstein merged 4 commits into
mainfrom
fix/make-room-admission-grace
Aug 31, 2026
Merged

fix(model cache): make_room honors the admission grace; grace owned by its LoadedModel handle#193
lstein merged 4 commits into
mainfrom
fix/make-room-admission-grace

Conversation

@lstein

@lstein lstein commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Problem

make_room's eviction loop checked only is_locked, but multi-model invocations load their whole set before locking any of it (the MiniMax H3 text encoder loads text_encoder → tokenizer → processor, then locks). Under shared-RAM-budget deficit on a dual-GPU rig, the tokenizer's cold-load make_room evicted the 27 GB text encoder admitted one second earlier — freeing nothing (the loader's handle keeps the model alive) while detaching the record from all cache accounting and printing the issue-7513 diagnostic once per paced-lock pass (~25 identical INFO lines per text-encoder load, observed in the field).

Fix (three commits, each shaped by an adversarial review of the previous)

  1. make_room spares entries inside their admission grace (awaiting_first_use). The async eviction paths (budget reconcile, peer eviction) already honored it; the synchronous path now does too. The keep-alive timeout clear and the /empty_model_cache button pass spare_awaiting_first_use=False — an idle-expiry or user-requested full clear outranks the grace.
  2. The grace is now owned by its LoadedModel handle. Review of commit 1 proved put()'s stale-grace sweep defeated the guard: it cleared every grace on each admission, so only the newest sibling was protected, and only until the next put() — the field failure reproduced with the guard in place. The handle registers a weakref (CacheRecord.grace_holder) alongside its existing release finalizer; the sweep now clears only provably orphaned graces (no handle ever registered, or the handle died without its finalizer running). Live holder ⇒ lock() or the finalizer will release it.
  3. Holder-identity check in the release + worker unpinning. The finalizer's release carries the dying handle's ref and leaves a grace that has been re-registered to a newer live handle; the deferred worker unpins the unpacked record local before blocking (the existing anti-pinning test caught this); the route tests assert the spare_awaiting_first_use=False plumbing.

Also demotes continue_lock's detached-record diagnostic to DEBUG — lock() reports it once at INFO; a paced stream repeated the identical line dozens of times.

Verification

  • 607 tests pass across tests/backend/model_manager/load/ and the router tests; new coverage: grace survival across sibling admissions (fails on commit 1 — verified), orphan sweeps both ways, finalizer release on handle drop, holder-identity release, explicit-clear override, plus the pre-existing eviction guards.
  • Two rounds of adversarial fresh-context review. Round 1 reproduced the sweep hole (fixed in commit 2). Round 2 found no reachable breakage; its two hardening suggestions are commit 3. It also verified the scariest corner — a detached-but-alive record's shared CPU weights cannot be freed under a peer cache's feet (the canonical tensors are ordinary refcounted objects held by the live module; a later adoption misses and cold-loads).

🤖 Generated with Claude Code

https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A

lstein and others added 3 commits August 30, 2026 22:28
make_room's eviction loop checked only is_locked, but multi-model
invocations load their whole set before locking any of it (the H3 text
encoder loads text_encoder, then tokenizer, then processor, then locks) —
so a sibling's cold-load make_room runs with a just-admitted entry
unlocked, and under shared-RAM-budget deficit it evicted the 27GB text
encoder admitted one second earlier to make room for a 50MB tokenizer.
Evicting such an entry frees NOTHING (the loader's handle keeps the model
alive) while detaching the record from all cache accounting and turning
every subsequent paced-lock pass into an issue-7513 diagnostic line
(observed: ~25 identical INFO lines per text-encoder load on a dual-GPU
rig). The asynchronous eviction paths (budget reconcile, peer eviction)
already honored awaiting_first_use; the synchronous path now does too.

The keep-alive timeout clear passes spare_awaiting_first_use=False: after
an idle period a surviving grace is abandoned by definition (a healthy
loader locks within seconds, and every lock resets the timer), so the
timeout keeps clearing everything unlocked.

Also demote continue_lock's detached-record diagnostic to DEBUG — lock()
already reports it once at INFO, and a paced stream repeated the same
line dozens of times.

Two existing tests exercised make_room on entries that had never been
locked; they now release the grace first, the way every real loader does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
Adversarial review of the previous commit proved its guard incomplete:
put()'s stale-grace sweep cleared EVERY grace on each admission, so only
the most recently admitted sibling was ever protected, and only until the
next put() — both reproduced paths re-created the field failure (a
just-admitted 27GB model evicted by a sibling's cold load) with the guard
in place. The sweep's premise ('a grace surviving to the next admission is
stale') is false in exactly the window the grace exists for: multi-model
invocations load their whole set before locking any of it.

The grace's owner is the LoadedModel handle, which already releases it via
a finalizer on an unlocked drop. Record a weakref to the handle on the
CacheRecord (grace_holder) at handle construction; the sweep now clears
only provably orphaned graces — no handle was ever registered (the load
raised between put() and LoadedModel construction) or the handle died
without its finalizer running (lost deferred worker). Live holder = the
loader is coming back; lock() or the finalizer releases the grace.

Also per review: the clear-model-cache button (/empty_model_cache) passes
spare_awaiting_first_use=False through the now-parameterized public
make_room — a user-requested full clear outranks the grace (pre-existing
behavior restored) — and the tautological assertion in the new test file
is replaced by real coverage: grace survival across sibling admissions,
orphan sweeps both ways, finalizer release, and the explicit-clear path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
…ond review

- The finalizer's release now carries the dying handle's weakref: when the
  record's grace has since been re-registered to a newer, still-live
  handle, the release leaves it alone (single-slot limitation documented —
  no current code path constructs two pre-lock handles for one record).
- The deferred worker unpins the unpacked CacheRecord local before
  blocking on the next queue item (the existing anti-pinning test caught
  the tuple unpacking keeping the record alive).
- The /empty_model_cache route tests now assert the
  spare_awaiting_first_use=False plumbing, not just tolerate the kwarg.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
@lstein
lstein enabled auto-merge August 31, 2026 03:04
@lstein
lstein merged commit 9a12333 into main Aug 31, 2026
19 checks passed
@lstein
lstein deleted the fix/make-room-admission-grace branch August 31, 2026 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant