[Llama-Engine-P1] apply_t3_rebuild blocks update_slots during model reload — async reload + double-buffering needed

[ddvnguyen]

Finding (P1)

apply_t3_rebuild() in tools/server/server-context.cpp:4430 runs
synchronously inside update_slots(). The call site is:

// tools/server/server-context.cpp:4569-4577 (PR #42 head)
//
// NOTE: this runs synchronously inside update_slots(), blocking
// the serving loop for the duration of the GGUF load + VRAM
// alloc. The drain-timeout guards entry but not the reload
// itself — the engine appears hung to the Coordinator for the
// entire reload. A future improvement could offload this to a
// background thread and gate requests until the reload completes.
if (!load_model(swapped_params)) {

A full T3 model reload does:

1. unload_model(model_tgt) — frees the current ggml model + KV cache + tensor buffers
2. llama_model_load_from_file(swapped_params.model.path, ...) — reads the GGUF, allocates VRAM (multi-second for a 19 GB Q5_K_M model on a 16 GB GPU; tens of seconds for the full 35B-A3B Q3_K-mini on a 12 GB 3060 with CPU offload)
3. llama_new_context_with_model(model, cparams) — builds the KV cache
4. MTP / draft-model paths + slot rebuild
5. COMBINED teardown + reattach (the 02c7a9d99 fix)
6. ggml_backend_sched_reset + per-slot sampler re-init

All six steps run on the same thread as update_slots(). During the reload, no in-flight request on any slot can be scheduled, and the engine's HTTP response stream is silent — the Coordinator sees the engine as hung.

Reproduction

# 1. Start the engine on the 5060 Ti + 3060 pair (COMBINED-static, DENSE profile)
bash scripts/deploy-hydra-head.sh rtx
# 2. Confirm the engine is serving (curl /health → 200, /v1/chat/completions returns)
# 3. Run set-profile.sh moe (operator-initiated profile switch, fires a T3 CONFIGURE)
bash scripts/set-profile.sh moe
# 4. While the switch is in progress, fire a parallel chat completion:
time curl -X POST http://localhost:8080/v1/chat/completions \
     -d '{"model":"Q3_K-mini","messages":[{"role":"user","content":"hi"}]}' \
     -H 'Content-Type: application/json'
# Expected (today): the second request blocks for the full T3 reload duration
# before the first reload's drain returns. No progress is visible to the
# Coordinator — the engine reads as "hung" in the dashboard.
# Expected (after fix): the reload runs in a background thread; the second
# request is either served with the old config (Q1: old-config path, per
# the recommendation in llama.cpp#40) or rejected with 503 (Q2: block path).

Impact

• Multi-second engine hangs for the headline use case (set-profile.sh moe ↔ set-profile.sh dense, 35B-A3B Q3_K-mini). For the 12 GB 3060 with the full MoE, the reload can be 10-20s; the engine reads as a crash to the Coordinator.
• Coordinator's item.EngineConfigTier field is set at request time but the actual model serving the request may be a different model than the one the request specified. With Q1 (old-config path), the request is correctly served with the old config, but the response trace shows the new config — the operator's telemetry is misleading.
• Blocks any future feature that wants to hot-swap a model without operator intervention (e.g. context-length overflow detected at request time triggers a n_ctx change — T2 only, but the same blocking pattern applies if T2 also gets a larger refactor).
• Pre-existing, but newly relevant: PR fork: implement T2/T3 apply path — actually reload model+context at slot-free moment #42 made T3 actually work for the first time (previous PRs only staged the config). The reload is no longer theoretical.

Fix

8e0122e15 correctly identifies why the fix is non-trivial:

P3b (async T3 reload) is deferred to a follow-up — load_model() replaces
nearly every member variable, making background-thread reload require
double-buffering, which is a larger architectural change.

Sketch:

1. Two-context double-buffer. Add a server_context_impl::pending_ctx member — a second llama_context + llama_model + per-slot state being built on a background thread. The existing context keeps serving requests until the new one is ready.

2. Background-thread orchestration. apply_pending_hydra_config() returns immediately after spawning the reload (the slot-free check still gates entry). The background thread runs load_model(swapped_params) + COMBINED reattach.

3. Atomic swap on completion. When the background thread succeeds, the swap is a pointer exchange:

   • pending_ctx becomes the live ctx_tgt
   • the old context is torn down
   • per-slot pointers refresh
   • any in-flight requests (zero, by invariant — drain) see the new context on the next update_slots()

4. Cancellation / failure. If the background thread fails, the rollback runs in the same background thread (don't tie up update_slots). The result is reported via the next INFO call (deferred-keys cleared, error in the response).

5. T2 stays synchronous for now. T2 is llama_free + llama_new_context_with_model — single-digit ms on the RTX 5060 Ti, doesn't justify the double-buffer overhead. Revisit if T2 ever includes model-graph re-derivation that takes >100ms.

Acceptance:

• T3 apply no longer blocks update_slots(); the reload runs on a worker thread.
• The drain-timeout semantics are unchanged (entry still gated, drain still aborts on timeout).
• The Coordinator's item.EngineConfigTier is either honored (Q1: serve with old config) or the request is rejected (Q2: 503) — pick one per llama.cpp#40's Q2 and document in the design.
• E2E: tests/system/test_profile_switch.py (planned in ddvnguyen/hydra_vortex#397 Phase 5) measures the reload time and asserts it's < HYDRAD_PROFILE_SWITCH_RELOAD_DEADLINE_MS from the Coordinator's view.

Effort estimate: ~1-2 weeks of C++ work, plus ~3 days of C# coordination changes in ProfileSwitcher and the new EngineConfigApplier if the swap semantic is Q1 (old-config serve) vs Q2 (reject).

Tracking

• Found on: ddvnguyen/llama.cpp#42 re-review (Jul 12, 2026)
• Code lives in: tools/server/server-context.cpp (hydra-fork branch only)
• Sub-mode of: Llama-Engine
• Explicitly deferred by: 8e0122e15 commit message ("P3b async T3 reload is deferred to a follow-up")

Cross-repo

• Hydra parent: ddvnguyen/hydra_vortex#397 (parent tracker for v4 design, Phase 2b / 4 / 5)
• Fork PR (where the issue was found): ddvnguyen/llama.cpp#42 (T2/T3 apply path, the new commits verified)
• Design question: ddvnguyen/llama.cpp#40 Q2 ("T3 race during decode") — the swap semantic must align with the chosen answer (Q1: old-config, Q2: 503 reject).

Sub-tasks (none yet)

To be split once the design is firmed up. Likely sub-tasks:

1. Double-buffer primitives in server_context_impl (slot, ctx_tgt, model_tgt, spec pointer — all need swap-friendly representations)
2. Background-thread orchestration in apply_pending_hydra_config
3. Atomic swap protocol (pointer exchange under the slot-free mutex)
4. C# ProfileSwitcher change for Q1 vs Q2 semantics
5. tests/system/test_profile_switch.py (lives in ddvnguyen/hydra_vortex#397 Phase 5)