[fork/phase-2b] runtime 0x40 EngineConfigure: T1/T2/T3 common_params delta + deferred T2/T3

[ddvnguyen]

Summary

Extend the 0x40 CONFIGURE engine control opcode to accept a common_params JSON
delta and apply it at runtime. The current implementation (per the
server-context.cpp:2932 task handler) only handles state_chunk_size; the
parent-side hydra_vortex#397 plan (v4 design, Phase 2b) requires the full T1/T2/T3
tier model so the C# Coordinator can drive profile switches (MoE ↔ DENSE) and
per-request param overrides (sampling, n_predict) without restarting the engine.

Critical path: without T3, ModelRegistry (introduced in
hydra_vortex#402) is dead code in the runtime path — the per-model config
(override_tensor, split_mode, tensor_split, model path) cannot reach the
engine after startup.

T1/T2/T3 tier model

Tier	Rebuild cost	Engine primitive	Examples
T1	None (cparams field write + sampler re-init; graph rebuild only)	llama_sampler_init + field writes	sampling.temp, sampling.top_p, sampling.top_k, sampling.min_p, sampling.penalty_repeat, seed, n_predict, n_keep, antiprompt
T2	Context reload (unload + reload KV)	llama_new_context_with_model	n_ctx, cache_type_k, cache_type_v, rope_freq_base, yarn_*, rope_scaling_type
T3	Model reload (unload + reload model)	llama_model_load_from_file	n_gpu_layers, n_cpu_moe, tensor_buft_overrides (for override_tensor), split_mode, tensor_split, model.path

Proposed wire schema (tracked in the parent-side docs PR)

Request (UTF-8 JSON)

{
  "sampling": {
    "temp": 0.5,
    "top_p": 0.9,
    "top_k": 40,
    "min_p": 0.05,
    "penalty_repeat": 1.1
  },
  "seed": 42,
  "n_predict": 500,
  "antiprompt": ["\nUser:"],
  "state_chunk_size": 2097152,
  "n_ctx": 65536,
  "cache_type_k": "q8_0",
  "cache_type_v": "q8_0",
  "n_gpu_layers": 65,
  "split_mode": "layer",
  "tensor_split": [25.0, 40.0]
}

Response (UTF-8 JSON)

{
  "success": true,
  "tier": "T1",
  "params_applied": {
    "sampling.temp": 0.5,
    "seed": 42,
    "n_predict": 500
  },
  "deferred_keys": []
}

tier is the highest tier in the request that the engine could apply (or
defer). deferred_keys lists the keys in T2/T3 that were deferred to the
next slot-free moment (not applied yet). params_applied echoes back the
actual values after any clamping — same echo pattern as the existing
state_chunk_size_applied.

Implementation outline (fork side)

Step 1: extend server_task_result_hydra_engine

src/llama-cpp/tools/server/server-task.h:679-735 — add fields:

• std::map<std::string, llama_json_value> params_applied;
• std::string tier;
• std::vector<std::string> deferred_keys;

Step 2: extend the CONFIGURE task handler

src/llama-cpp/tools/server/server-context.cpp:2932 — parse the JSON,
classify each key as T1/T2/T3, apply T1 in-place, defer T2/T3. The C++ side
needs a slot-free hook for deferred rebuilds.

Step 3: add runtime mutators

src/llama-cpp/src/llama-hydra.cpp — add llama_hydra_* runtime mutators
for T3 keys (currently no runtime mutator exists for model reload, split
mode, override_tensor):

• llama_hydra_set_override_tensor(ctx, pattern) — must invalidate the
  graph cache; the next compute rebuilds with the new pattern
• llama_hydra_set_split_mode(ctx, mode, tensor_split) — requires model
  reload
• llama_hydra_reload_model(ctx, params.model) — the heavy T3 rebuild

Step 4: extend hydra_handle_configure

src/llama-cpp/tools/server/server-context.cpp:6735-6775 — write the
params_applied / tier / deferred_keys into the response meta.

Backward compat

The existing state_chunk_size-only payload (sent by
WorkerSchedulerService.cs:2842 at startup) MUST keep working unchanged.
The new handler treats the legacy single-key payload as a T1 request with
params_applied: {"state_chunk_size": N} and tier: "T1".

Open design questions (need resolution before PR 2 lands)

Tracked in hydra_vortex#397's Phase 2b sub-tasks.

Q1: T3 deferred trigger

"Next slot-free moment" needs a precise definition. Options:

• (a) All slots fully released — safest, but a long decode can stall a
  profile switch for minutes
• (b) Any slot in idle state — faster, but the active slot could see
  a config delta mid-decode (need a graph-cache invalidation strategy)
• (c) Operator-triggered drain — the C# scheduler can force-slot-release
  when it issues the T3 request

Recommendation: (a) for the first cut, with a
HYDRA_COORD_PROFILE_SWITCH_DRAIN_TIMEOUT env var for the operator to
bound the wait.

Q2: T3 race during decode

If a model reload is pending and a new request comes in, what config does
the new request see?

• (a) Old config — dispatch immediately, log a warning
• (b) Block at the scheduler — wait for the rebuild to complete
• (c) Reject with HTTP 503 — caller retries after a few seconds

Recommendation: (a) — matches the existing SET_EXPERT_MODE fall-back
pattern. The C# scheduler's item.EngineConfigTier field carries the
"what config this request actually used" for the response trace.

Q3: Backward compat strategy

The existing startup state_chunk_size call is from
WorkerSchedulerService.cs:2842 (one-shot). Options:

• (a) Keep the single-key legacy payload working — the C++ handler
  detects the absence of tier field and treats it as T1 with just
  state_chunk_size
• (b) Migrate the startup call to the new schema

Recommendation: (a) for the first cut. The startup call is one place;
no need to change it as part of this PR stack.

Q4: EngineConfigApplier (PR 4) timing

When does the C# push the EngineConfig from ModelRegistry to the
engine?

• (a) On startup (one-shot, ensures the engine is in the expected state)
• (b) On every profile switch (e.g. bash scripts/set-profile.sh moe)
• (c) On first request after a model change (lazy)

Recommendation: (a) + (b) — startup is the safety net, profile switch
is the explicit trigger. The ProfileSwitcher service (already in the
v4 design at hydra_vortex#397 Phase 4) calls (b).

Cross-references

• ddvnguyen/hydra_vortex#36 (v4 design handoff — Phase 2)
• ddvnguyen/hydra_vortex#397 (parent tracker)
• ddvnguyen/hydra_vortex#398 (parent Phase 1 tracker)
• ddvnguyen/hydra_vortex#402 (parent Phase 2a — EngineConfig +
  ModelRegistry are the input to this work; without T3 they are dead
  code in the runtime path)

Workload estimate

~550 LOC C++ across two fork PRs (the C++ is the critical path; the
parent-side C# is in 3 separate parent PRs that depend on this work landing
first).

Tracking

• Fork PR 2: fork/phase-2b-configure-handler — extends the CONFIGURE
  task handler for T1+T2+T3 classification and the params_applied /
  tier / deferred_keys echo
• Fork PR 3: fork/phase-2b-t3-mutators — adds the llama_hydra_*
  runtime mutators for T3 keys (model reload, override_tensor, split_mode,
  tensor_split)

Both PRs land on hydra-fork first, then the parent bumps the submodule
in hydra_vortex#406 (TBD).

[ddvnguyen]

Cross-ref: the parent-side wire schema is in ddvnguyen/hydra_vortex#406. The 4 open design questions (Q1 deferred trigger, Q2 race-during-decode, Q3 backward-compat, Q4 EngineConfigApplier timing) have recommended defaults documented inline in that PR. The fork-side work (PRs 2 + 3 in the Phase 2b stack) can start once Q1 and Q2 are resolved — both have a clear default ("all slots fully released" and "old config wins" respectively) so implementation can proceed in parallel with the design discussion.