llama-engine: per-GPU llama-engines + merged Hydra RPC + no-weight-transfer COMBINE

[ddvnguyen]

Implementation handoff — v4 design: per-GPU llama-engines + merged Hydra RPC + no-weight-transfer COMBINE

TL;DR

Replace the current 3-server design (HTTP + Hydra control RPC + ggml-RPC) with a single merged Hydra RPC server in a new fork-isolated module tools/llama-engine/hydra_rpc/. The merged server dispatches by first byte (ggml-RPC 0x0E vs Hydra 0x30–0x46) and uses a bounded thread pool (size 2). COMBINE is layer-split only, with each engine mmap'ing the same model from disk and the head binding to the peer's mmap-resident tensors via the existing RPC_CMD_RESOLVE_TENSOR (PR #20) — no weight data on the wire. Per-process model loading: A tells B which layers to load via POST /control/load_model with a keep_layer_range field.

Phase 1 is MERGED via PR #37 (merged Hydra RPC server in tools/llama-engine/hydra_rpc/, bounded thread pool, MSG_PEEK dispatch). This issue is the implementation handoff for Phase 2 onward. Phase 2 is the Unified COMBINE refactor: collapse the Hydra-side COMBINED-OT and COMBINED-static abstractions in Hydra.Core into a single unified COMBINE concept, accept common_params at runtime via 0x40 EngineConfigure, and remove the Hydra-specific CLI flag names (--rpc-engine, --combined-ot-pattern, --combined-split-mode, --combined-tensor-split, --peer-only, --ggml-rpc-port). The dual-load machinery, the 0x44 SET_EXPERT_MODE runtime toggle, the RPC_CMD_RESOLVE_TENSOR no-weight-transfer path, and the no-model peer path all stay — they are the features, not the abstractions. Engines boot model-less in the lazy-load design; Hydra.Core tells each engine which model + mode (SOLO or COMBINE) to run via 0x40.

Why this matters

• DENSE 27B at Q4 doesn't fit on a single 5060 Ti (16 GB) → layer-split COMBINE is the only path to serve it.
• MoE 35B-A3B at Q3_K-mini fits on the 3060 alone → COMBINE is optional; SOLO on the 5060 Ti is faster (~200 vs ~48 tok/s for COMBINED layer-split). The MoE COMBINED win is resource-sharing (free the 5060 Ti), not throughput.
• The current pre-PR-fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31 design (3 separate servers) has a known crash on the dense profile (PR fix: COMBINED layer-split startup crash + performance improvements #34 root cause: the ggml scheduler follows data dependencies, not weight placement; see "Scheduler feasibility" below).
• The target architecture lets the operator configure one port per engine (--rpc-port) instead of two (--rpc-port + --ggml-rpc-port).

Scheduler feasibility (CRITICAL — do this before Phase 2)

PR #34 documented that the upstream ggml scheduler assigns compute to the device that holds the inputs, not the device that holds the weights. This breaks layer-split for DENSE models: a layer's compute goes to the peer (where the previous layer's outputs live), but the peer's RPC server can't access the local CUDA's weights.

Before Phase 2 starts, run a 1B model (e.g. Qwen2-1.5B) layer-split 50/50 across CUDA0 + RPC0, decode 50 tokens, confirm no GET_TENSOR FAILED errors. If the test fails:

• Fallback A: use upstream's -ts --rpc path (option 2 in PR fix: COMBINED layer-split startup crash + performance improvements #34). The fork's per-process model loading (Phase 3) still works; it just drives the upstream CLI instead of --rpc-engine.
• Fallback B: drop DENSE 27B from G1; only pursue MoE 35B COMBINED.

The feasibility check is a precondition for G1 (DENSE 27B ≥ 50 tok/s). If G1 is dropped, update the design doc (#35) accordingly.

Goals & acceptance criteria

Each goal is measurable. The acceptance test for each is the listed condition.

#	Goal	Target	Acceptance test
G1	DENSE 27B layer-split COMBINED decode	≥ 50 tok/s on 5060+3060	tests/system/test_combine_dense.py passes; the scheduler feasibility check above passes first
G2	MoE 35B-A3B layer-split COMBINED decode	≥ 45 tok/s on 5060+3060	tests/system/test_combine_moe.py passes
G3	SOLO 5060 Ti decode (no regression)	≥ 190 tok/s (≤ 5% from 200)	tests/system/test_solo_rtx.py measures ≥ 190 tok/s on the existing Q3_K-mini 35B
G4	Per-engine model load (cold, NVMe)	≤ 25 s for 27B Q4	tests/system/test_engine_load.py --model DENSE-27B-Q4 --cold measures ≤ 25 s
G5	Per-engine model load (warm, page cache)	≤ 5 s for 27B Q4	tests/system/test_engine_load.py --model DENSE-27B-Q4 --warm measures ≤ 5 s
G6	Runtime SOLO ↔ COMBINED switch (same model, MoE 35B)	< 100 ms (no reload)	tests/system/test_mode_switch.py measures < 100 ms between SOLO and COMBINED for the same model
G7	Multi-model profile switch (DENSE ↔ MoE)	< 60 s end-to-end	tests/system/test_profile_switch.py measures end-to-end reload + RPC rebind ≤ 60 s
G8	P100 cross-host P/D split	no regression (28 tok/s)	tests/system/test_p100_pd.py measures ≥ 28 tok/s on the existing P100 P/D path
G9	Upstream-touched LOC	≤ 4 functions / 75 LOC	git diff upstream/master -- 'ggml/src/ggml-rpc/*' 'ggml/src/ggml-cuda/*' 'ggml/src/ggml-backend.cpp' shows ≤ 75 insertions across ≤ 4 functions

Hard constraints

• C1: minimal upstream diff. All new code in tools/llama-engine/hydra_rpc/ (fork-only) or one of four small upstream-touched functions: ggml_backend_rpc_handle_client (~20 LOC), ggml_backend_rpc_remove_server (~30 LOC), set_keepalive (~20 LOC), MSG_NOSIGNAL (1 LOC). Total upstream-touched: ~71 LOC.
• C2: no weight transfer over the wire. Peer loads tensors from its own mmap; RPC_CMD_RESOLVE_TENSOR (PR fork: zero-copy COMBINED expert tensors (name-based remote binding, replaces network copy) #20) sends the tensor name only. This is the explicit mechanism for "no weight transfer" — not "no RPC at all". Issue Unified RPC Server: one port, one binary, no extra flags #29's design doc had this wrong (it said "no RPC_CMD_RESOLVE_TENSOR" which is contradictory; see "Clarifications" below).
• C3: SOLO mode must not regress. No new global locks, no mandatory setup, no per-op overhead vs upstream.
• C4: cross-host (P100) unchanged. All Hydra-specific behavior is same-host only. The P100 uses upstream rpc-server binary, not the merged engine.
• C5: each engine can load any model from a known path on demand. No per-process model lock-in.
• C6: fail-soft wrapping is allowed only in teardown paths (buffer_free during model unload, scheduler teardown). Normal ops keep RPC_STATUS_ASSERT / GGML_ABORT on failure. This is why PR fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31's broad fail-soft replacement is rejected — it would mask data corruption in the inference path (peer dies mid-graph, head returns GGML_STATUS_FAILED, scheduler continues with uninitialized tensors, user gets garbage tokens).
• C7: protocol dispatch uses one byte (ggml-RPC uses 0x00–0x11, Hydra uses 0x30–0x46; ranges don't overlap, dispatch is unambiguous). No magic bytes, no per-conn detach(), bounded thread pool.

Clarifications (vs the issue #29 design doc)

Issue #29 was the original design doc for the unified server (now PR #31). Two clarifications to read carefully:

1. "No RPC_CMD_RESOLVE_TENSOR" in issue Unified RPC Server: one port, one binary, no extra flags #29 is wrong. The design wants "no weight transfer", but the only mechanism for "no weight transfer" with the standard upstream buffer model is RPC_CMD_RESOLVE_TENSOR. The v4 design uses RPC_CMD_RESOLVE_TENSOR explicitly. If you see "no RESOLVE_TENSOR" anywhere in issue Unified RPC Server: one port, one binary, no extra flags #29, treat it as a typo for "the RESOLVE_TENSOR path replaces the SET_TENSOR weight-transfer path".

2. PR fix: COMBINED layer-split startup crash + performance improvements #34's 7 fixes don't address the actual crash. The author's own analysis says "The GET_TENSOR crash persists" and the root cause is the upstream ggml scheduler. The 7 fixes are mitigations. Do not cherry-pick from PR fix: COMBINED layer-split startup crash + performance improvements #34. The crash is real, but the fix is upstream or a different split mechanism (Fallback A in the "Scheduler feasibility" section).

Architecture

                  Hydra.Core (C#)
                       │
        HTTP: {"mode":"combine","peer":"localhost:9504","model":"DENSE-27B-Q4","split":"21/44"}
                       │
        ┌──────────────▼─────────────────────────────┐
        │   llama-engine A (host: GPU 0 = 5060 Ti)    │
        │   role: COMBINE-head                         │
        │   ports: HTTP=8080, RPC=9504                │
        │                                              │
        │   ┌─────────────────────────────────────┐   │
        │   │  HTTP server (upstream)              │   │
        │   │  --port                              │   │
        │   │  /health, /v1/chat/completions,      │   │
        │   │  /control/load_model (A→B),         │   │
        │   │  /control/set_expert_mode, etc.      │   │
        │   └─────────────────────────────────────┘   │
        │   ┌─────────────────────────────────────┐   │
        │   │  Hydra RPC (tools/llama-engine/     │   │
        │   │  hydra_rpc/) --rpc-port              │   │
        │   │  dispatch: 1-byte peek               │   │
        │   │   0x0E → ggml_backend_rpc_handle_   │   │
        │   │          client (ggml-RPC compute)  │   │
        │   │   else  → hydra_handle_connection   │   │
        │   │          (Hydra control)            │   │
        │   │  bounded thread pool (size 2)       │   │
        │   └─────────────────────────────────────┘   │
        │   mmap: DENSE-27B-Q4.gguf                  │
        │   tensor_split: layers 0–20 on local GPU    │
        │                  layers 21–43 on RPC peer   │
        └──────────────┬──────────────────────────────┘
                       │  ggml-RPC over loopback
                       │  RPC_CMD_GRAPH_COMPUTE (per token)
                       │  RPC_CMD_SET_TENSOR (peer loads from mmap)
                       │  RPC_CMD_RESOLVE_TENSOR (zero-copy binding)
                       │  NO weight data on the wire
        ┌──────────────▼──────────────────────────────┐
        │   llama-engine B (host: GPU 1 = 3060)       │
        │   role: COMBINE-peer                        │
        │   ports: HTTP=8081, RPC=9505                │
        │                                              │
        │   ┌─────────────────────────────────────┐   │
        │   │  HTTP server (upstream)              │   │
        │   │  --port                              │   │
        │   └─────────────────────────────────────┘   │
        │   ┌─────────────────────────────────────┐   │
        │   │  Hydra RPC (tools/llama-engine/     │   │
        │   │  hydra_rpc/) --rpc-port              │   │
        │   │  dispatches ggml-RPC ops to the      │   │
        │   │  local GPU; rejects Hydra control    │   │
        │   │  (peer is passive)                   │   │
        │   └─────────────────────────────────────┘   │
        │   mmap: DENSE-27B-Q4.gguf                  │
        │   exposes GPU 1 as ggml-RPC backend        │
        │   layers 21–43 resident on GPU 1            │
        └─────────────────────────────────────────────┘

Inside the merged Hydra RPC server

┌─────────────────────────────────────────────────────────────┐
│  hydra_rpc::start({port, backends, hydra_ctx, pool_size})  │
│                                                              │
│  bind+listen(server_fd, port)                                │
│      │                                                       │
│      ▼                                                       │
│  accept_thread (single, blocked in accept())                │
│      │                                                       │
│      ├── accept() returns conn_fd                            │
│      │                                                       │
│      ├── pool.try_enqueue([conn_fd]{                         │
│      │       recv(conn_fd, &b, 1, MSG_PEEK)                 │
│      │       ├── b == 0x0E && !backends.empty()             │
│      │       │     → ggml_backend_rpc_handle_client(...)    │
│      │       ├── b != 0x0E && hydra_ctx != nullptr          │
│      │       │     → hydra_handle_connection(...)          │
│      │       └── else → ::close(conn_fd)                    │
│      │   })                                                  │
│      │   (drops connection if pool queue is full)           │
│      │                                                       │
│      └── loop until stop_requested                          │
│                                                              │
│  pool: bounded_thread_pool<size=2, max_queue=64>            │
│  shutdown: ::shutdown(server_fd, SHUT_RDWR) + atomic flag   │
└─────────────────────────────────────────────────────────────┘

Mode-to-port mapping

Mode	--port	--rpc-port	Backends exposed	Hydra control served
SOLO (no control)	8080	0 (off)	—	—
SOLO with control	8080	9504	none	yes
COMBINE head	8080	9504	none (head's GPU is consumed locally)	yes
COMBINE peer (model)	8080	9505	local GPU	no (passive)
COMBINE peer (no model)	8080	9505	local GPU	no (passive)

Wire protocol summary

The wire format is upstream ggml-RPC (RPC_CMD_*) + the Hydra protocol (HYDRA_OP_*), served from the same listener. The dispatch is by first byte:

• 0x0E (RPC_CMD_HELLO) → ggml_backend_rpc_handle_client(conn_fd, ...) serves RPC_CMD_GRAPH_COMPUTE, RPC_CMD_SET_TENSOR, RPC_CMD_INIT_TENSOR, RPC_CMD_RESOLVE_TENSOR, etc.
• 0x30–0x46 (HYDRA_OP_*) → hydra_handle_connection(conn_fd, ctx) serves HYDRA_OP_SET_EXPERT_MODE, HYDRA_OP_SWAP_QUANT, HYDRA_OP_STATE_GET, HYDRA_OP_STATE_PUT.

The peer also supports RPC_CMD_RESOLVE_TENSOR for the no-weight-transfer path: the head calls RPC_CMD_RESOLVE_TENSOR("blk.21.attn_q.weight") on the peer, the peer looks up the tensor in its mmap, returns a remote_ptr. The head uses that remote_ptr in the rpc_tensor.buffer field of RPC_CMD_GRAPH_COMPUTE. No weight data crosses the wire.

For A→B partial load (Phase 3), a new HTTP route POST /control/load_model on B's HTTP server:

• Body: {"model": "<path>", "keep_layer_range": {"lo": <int>, "hi": <int>}}
• B unloads current model, loads with the range filter, returns {"loaded": true, "n_tensors": <int>, "size_mb": <int>}.
• Trust-gated by --control-allow-from (default 127.0.0.1).

Phased implementation plan

Decisions made (resolved during Phase 2 design)

The following design questions were open at Phase 1 handoff and have been resolved as part of the Phase 2 (unified COMBINE) design. They are recorded here for traceability.

Q1 — Runtime EngineConfigure (0x40) reconfigure path: three-tier classification

Every common_params field is classified into one of three tiers based on what it costs to change at runtime. The 0x40 payload diffs against the last-applied state and triggers the minimum rebuild needed.

Tier	Cost	What it means
T1 — no teardown	per-request	Field is applied per-request (sampling, n_predict, per-slot settings, server hot-swaps). No rebuild.
T2 — context teardown	llama_context_free + llama_new_context_with_model	Field is consumed at context init (n_ctx, kv cache types, n_threads, RoPE/YaRN, flash-attn, etc.).
T3 — model teardown	llama_model_free + llama_model_load_from_file + llama_new_context_with_model	Field is consumed at model load (model path, n_gpu_layers, tensor_split, split_mode, rpc_servers, override_tensors, kv_overrides, tensor_buft_overrides, mmproj, lora_adapters, control_vectors, n_cpu_moe, devices, use_mmap/use_mlock/use_direct_io, check_tensors, no_alloc, speculative.model).
T4 — process restart	(empty for common_params)	Only start-only items outside common_params: bind ports (--port, --rpc-port, --rpc-server-port), --hostname, --n-threads-http, log config, TLS cert paths, CUDA device selection. Forbidden to change at runtime; 0x40 rejects these fields.

T1 fields include: n_predict, sampling.*, n_keep, n_chunks, n_print, antiprompt, prompt_cache_*, cache_prompt, cache_idle_slots, n_cache_reuse, n_ctx_checkpoints, checkpoint_min_step, cache_ram_mib, slot_prompt_similarity, single_turn, cont_batching, ctx_shift, input_prefix/suffix/bos, prompt, system_prompt, prompt_file, api_keys, chat_template, use_jinja, enable_chat_template, force_pure_content_parser, default_template_kwargs, reasoning_format, enable_reasoning, prefill_assistant, embedding, embd_normalize, embd_out, embd_sep, cls_sep, image, image_min_tokens, image_max_tokens, timeout_read, timeout_write, sse_ping_interval, sleep_idle_seconds, server TTY-related flags, logits_file, logits_output_dir, save_logits, tensor_filter, out_file, load_progress_callback[_user_data].

T2 fields include: n_ctx, n_batch, n_ubatch, n_parallel, n_sequences, n_outputs_max, grp_attn_n, grp_attn_w, cpuparams, cpuparams_batch, cb_eval, cb_eval_user_data, numa, rope_freq_base/scale, yarn_*, rope_scaling_type, pooling_type, attention_type, flash_attn_type, cache_type_k, cache_type_v, kv_unified, swa_full, no_perf, no_kv_offload, no_op_offload, no_extra_bufts, no_host, speculative (lookup-ngram only — the model-draft field is T3).

T3 fields include: model, mmproj, n_gpu_layers, main_gpu, tensor_split, split_mode, devices, use_mmap, use_direct_io, use_mlock, check_tensors, no_alloc, kv_overrides, tensor_buft_overrides, lora_init_without_apply, lora_adapters, control_vectors, the rpc_servers list (stock --rpc-server), n_cpu_moe, speculative.model.

Field map reference: src/llama-cpp/common/common.h:426-706 is the source of truth. common_model_params_to_llama (common.h:884) consumes the T3 fields. common_context_params_to_llama (common.h:885) consumes the T2 fields. Sub-structs: common_params_sampling at common.h:210-289, common_params_speculative at common.h:347-370, common_params_model at common.h:292-301.

Q2 — 0x44 SET_EXPERT_MODE stays a separate opcode

0x44 does not fold into 0x40. The mode is cparams.hydra_expert_mode and is part of the graph-reuse key, so flipping it forces a graph rebuild but not a model reload. Keeping 0x44 separate from 0x40 means per-request SOLO↔COMBINED flips are cheap (graph rebuild only) when the dual-load is already set up at T3 load time. The cost is one extra opcode in the Hydra protocol; the win is no model reload on every mode flip.

Q3 — RPC_CMD_RESOLVE_TENSOR (PR #20) is a stable fork API

The fork commits to maintaining RPC_CMD_RESOLVE_TENSOR and RPC_CMD_RESOLVE_TENSES as extensions to ggml-RPC. Stock llama.cpp does not have an equivalent; this is what the dual-load zero-copy bind uses (ggml_backend_rpc_bind_remote_tensor resolves tensor handles by name from the peer's resident-tensor registry). Future work that needs the no-weight-transfer path (e.g. dense 27B layer-split with mmap-share) is built on top of PR #20.

Q4 — Lazy model load; no-model peer path is in scope; engines boot model-less

The default boot state is model-less. The engine binds the HTTP port, the Hydra RPC port, the local-CUDA-exposure port (if any), starts the servers, and waits for the first 0x40 EngineConfigure. The first 0x40 triggers a T3 model load.

This is the "3 GPU worker pool" pattern in production: each of the 5060 Ti, the 3060, and the P100 runs a llama-engine with no model loaded; Hydra.Core sends 0x40 to whichever engine should serve a given (model, mode) request. The benefit is fast startup (no model-load barrier at boot, <1 second to "engine ready" / waiting for 0x40) and operator simplicity (3 identical bare engines; Hydra.Core routes work).

The no-model peer path (the 3060 exposing its CUDA on --rpc-server-port without loading a model) is the no-weight-transfer support for the dense 27B layer-split. The 5060 Ti loads the full model and binds to the 3060's CUDA backend via stock --rpc-server localhost:9506.

Three-layer startup config (CLI + optional YAML + 0x40 payload)

The three layers, in order of precedence:

highest priority  →  Layer 3: 0x40 EngineConfigure payload  (per-request)
                    Layer 2: YAML config (or CLI)         (per-startup defaults)
                    Layer 1: llama.cpp built-in defaults  (common_params {})
lowest priority

• Layer 1: common_params built-in defaults at common.h:426-706. Always present. Fork does not change these.
• Layer 2: startup config. Two sources merged: CLI flags (the T4 items) and an optional YAML config file (the non-T4 defaults the operator wants to bake in: n_ctx, cache_type_k/v, n_gpu_layers, flash_attn, n_parallel, RoPE/YaRN, etc.). Without --config, only CLI flags are applied; the engine is a true bare compute slot.
• Layer 3: 0x40 EngineConfigure payload, per-request. JSON shape mirrors common_params. Engine diffs against the last-applied state and triggers T1/T2/T3 rebuild as needed.

In production with YAML defaults, Hydra.Core can send a minimal 0x40 payload (e.g. { "model": "moe-35b", "mode": "combine", "peer": "localhost:9506" }) and the engine applies the layered defaults for everything else. In the bare-pool case (no YAML), Hydra.Core sends the full common_params-shaped payload and the engine applies it directly.

"Unified COMBINE" resolver (Hydra.Core C# side)

The C# MultiEngineRouter mode resolver replaces the OT-vs-static branch. Given (model_alias, combined: bool), it returns a 0x40 payload delta:

(model, combined)	0x40 payload shape (delta from defaults)
MoE + SOLO	{ model, n_gpu_layers: 99, n_cpu_moe: 8, override_tensors: ["token_embd=CPU", "output=CPU", "output_norm=CPU"] }
MoE + COMBINE	as SOLO, plus override_tensors: [..., "blk.*.ffn_*_exps.weight=RPC0"] and rpc_servers: ["localhost:9506"] (the peer)
DENSE + SOLO	{ model, n_gpu_layers: 99, split_mode: "none" }
DENSE + COMBINE	{ model, n_gpu_layers: 99, split_mode: "layer", tensor_split: [21, 44], rpc_servers: ["localhost:9506"] }

The resolver does not name "OT" or "static" — those words do not appear in the C# code. The four rows are just stock-params combinations, looked up by (model_arch, mode). The C# still emits 0x44 SET_EXPERT_MODE for MoE COMBINE plans (translator layer under the hood) so the live MoE dual-load path keeps working through the transition.

---

Phase 0: Scheduler feasibility (DO THIS BEFORE PHASE 1)

Goal: confirm the layer-split scheduler can place compute on the right device for our models.

Tasks:

1. Build the existing hydra-fork fat binary: cmake --build build_sm86_sm120 --target llama-engine
2. Load Qwen2-1.5B-Instruct-Q4_K_M with tensor_split "20/40" (50/50) on CUDA0 + RPC0 (loopback, ggml-rpc-server upstream binary).
3. Decode 50 tokens. Log per-token device placement. Look for GET_TENSOR FAILED or RPC_CMD_* failed for ... on ... errors.
4. If the test passes: proceed to Phase 1. If it fails: file a parent issue documenting the scheduler limitation; coordinate with upstream llama.cpp; pick Fallback A (use -ts --rpc) or Fallback B (drop G1).

Acceptance: test runs cleanly, no errors. The output layer (output.weight or l_out-N) is placed on the device that holds its weights, not the device that holds its inputs.

Time: 1 day.

Commit message: infra: scheduler-feasibility check (1B layer-split sanity test) (in the parent hydra_vortex repo, not the fork).

---

Phase 1: Cherry-pick + new module (FIRST MERGE)

Goal: the merged Hydra RPC server runs in place of the current 3 servers. No new functionality, no behavior change, just the architecture swap.

Files to create (under src/llama-cpp/tools/llama-engine/hydra_rpc/):

File	LOC	Purpose
hydra_rpc.h	70	Public API: start(), stop(), is_running(), settings struct
hydra_rpc.cpp	180	Accept loop, 1-byte dispatch, lifecycle
bounded_thread_pool.h	50	bounded_thread_pool<N> template + enqueue() + try_enqueue()
bounded_thread_pool.cpp	90	Implementation

Total new fork-isolated: ~390 LOC.

Files to modify:

File	LOC	Change
tools/llama-engine/llama-engine.cpp	+30 / −90	Collapse two start_*_rpc_server calls into one hydra_rpc::start call. Delete start_shared_backend_rpc_server (~60 LOC) and start_backend_rpc_peer_server (~30 LOC). Update CLI to remove --ggml-rpc-port.
tools/server/server-context.cpp	−40	Delete server_context::start_rpc_server (replaced). Delete the per-conn std::thread::detach() at line 7187.
ggml/src/ggml-rpc/ggml-rpc.cpp	+50	Add ggml_backend_rpc_handle_client (public API for serving a pre-accepted fd; ~20 LOC) + ggml_backend_rpc_remove_server (cleanup; ~30 LOC). Register in ggml_backend_rpc_get_proc_address.
ggml/src/ggml-rpc/transport.cpp	+25	Add set_keepalive (~20 LOC) + MSG_NOSIGNAL on send (1 LOC).
ggml/include/ggml-rpc.h	+5	Export the two new functions.
ggml/src/ggml-rpc/ggml-rpc.cpp	−16	Revert PR #31's broad RPC_STATUS_ASSERT → fail-soft replacement. Restore upstream's GGML_ABORT behavior in normal ops. Only keep the null-guard (`if (!ctx

Total upstream-touched: ~75 LOC (4 functions, 2 files). Within G9 budget.

CLI flag changes:

• Remove: --ggml-rpc-port (use --rpc-port only).
• Add: --control-allow-from <ip> (default 127.0.0.1) for the new /control/load_model route in Phase 3. Default allows Phase 1 to work without changes.

Config file changes (in ddvnguyen/hydra_vortex parent repo):

• infra/hydra-core/config/workers.json: remove ggml_rpc_port field from each worker.
• infra/hydra-head/config/node-rtx.yaml: remove ggml_rpc_port field.
• infra/hydra-head/config/node-rtx3060.yaml: remove ggml_rpc_port field.

Test plan (added in this phase):

Test	What it verifies
tests/test-hydra-rpc-accept-loop.cpp	MSG_PEEK dispatch sends 0x0E to ggml-RPC handler and 0x30 to Hydra handler. Pool full → connection dropped. Shutdown unblocks accept thread.
tests/test-hydra-bounded-thread-pool.cpp	enqueue blocks when full, try_enqueue returns false, workers join on stop().
tests/test-hydra-solo-regression.cpp	SOLO mode (no --rpc-port) starts and serves /health on --port. No accept thread spawned. Same as before Phase 1.
tests/system/test_solo_rtx.py	G3 acceptance: SOLO 5060 Ti tok/s ≥ 190 (≤ 5% from 200).

Acceptance:

• G3: SOLO 5060 Ti decode ≥ 190 tok/s (no regression).
• G9: upstream diff ≤ 75 LOC across ≤ 4 functions in 2 files.
• The fat binary builds: cmake --build build_sm86_sm120 --target llama-engine.
• All existing SOLO tests pass unchanged.

Cherry-pick source: from PR #31, only:

• ggml_backend_rpc_handle_client (ggml-rpc.cpp, ~20 LOC)
• ggml_backend_rpc_remove_server (ggml-rpc.cpp, ~30 LOC)
• set_keepalive (transport.cpp, ~20 LOC)
• MSG_NOSIGNAL (transport.cpp, 1 LOC)
• socket_t::from_fd (transport.{h,cpp}, ~10 LOC) — needed for ggml_backend_rpc_handle_client to wrap the fd

NOT cherry-picked (rejected per C6 / C7 / G9 / dead-path cleanup):

• The RPC_STATUS_ASSERT fail-soft replacement (16+ places)
• The response = {} zero-inits (3 places)
• The ggml-cuda/fattn.cu GGML_ABORT removal
• The per-conn std::thread::detach() (replaced by bounded pool)
• The start_rpc_accept_loop in tools/server/ (replaced by tools/llama-engine/hydra_rpc/)
• The unconditional llama_hydra_enable_shared_backend_compute_lock() (the call site in start_shared_backend_rpc_server is being deleted)
• llama_hydra_load_combined_experts, _rebind_*, s_hydra_combined_bindings, ffn_*_exps_rpc — KEPT, not cherry-picked from; these are the dual-load machinery that survives into Phase 2
• llama_hydra_validate_quant_parity (Phase C) + SWAP_QUANT validation block (split to separate PR per hydra_vortex#394)
• startup_stage and the staged init (we keep the simple is_ready for now; staged init can be added in Phase 2 if needed)
• wait_for_peer_ready + --peer-health-url (added in Phase 3 when A→B load needs it)

Time: 1 week. PR title: fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/. PR body references this issue, #35, and the existing hydra_vortex#392 epic.

Rollback plan: revert the PR. The previous 3-server design is preserved on hydra-fork. No production deploys are gated on this; the merged server is feature-equivalent to the 3-server design when --rpc-port=0 (no SOLO regression). Revert = git revert <merge-sha> and redeploy.

---

Phase 2: Unified COMBINE refactor — stock params + runtime reconfigure

Goal: collapse the Hydra-side COMBINED-OT and COMBINED-static abstractions in Hydra.Core into a single unified COMBINE concept. The llama-engine accepts common_params at runtime via 0x40 EngineConfigure. The dual-load machinery, the 0x44 SET_EXPERT_MODE runtime toggle, the RPC_CMD_RESOLVE_TENSOR no-weight-transfer path, and the no-model peer path all stay — they are the features, not the abstractions. What dies is the Hydra-side labelling (Plan.OtSplit, Plan.run_type, OT-vs-static branches in MultiEngineRouter) and the Hydra-specific CLI flag names (--rpc-engine, --combined-ot-pattern, --combined-split-mode, --combined-tensor-split, --peer-only, --ggml-rpc-port).

Why this matters

The current state has two redundant "mode" abstractions:

• Hydra-side: Plan.OtSplit, Plan.CombinedOtSplit, run_type: "combined-static-peer", combined_ot_split config field, OT-vs-static branching in MultiEngineRouter.cs.
• Engine-side CLI: --rpc-engine, --combined-ot-pattern, --combined-split-mode, --combined-tensor-split, --peer-only, --ggml-rpc-port (the last five stripped by the PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37 deprecation shim).

The engine-side flags are shorthand for stock common_params (--rpc-server, --override-tensor, --split-mode, --tensor-split, and the stock arg for exposing local CUDA). The Hydra-side labels are shorthand for "set these stock params for this (model, combined) combination". Both shorthand layers hide what's actually happening and prevent the system from supporting any common_params field beyond the few the fork chose to expose. Removing them is a code-reduction win, a clarity win, and a "support all llama.cpp params" win.

The mechanisms that stay (per the corrected design):

• src/llama-hydra.cpp — llama_hydra_load_combined_experts, llama_hydra_rebind_combined_experts, s_hydra_combined_bindings, llama_hydra_set_expert_mode, llama_hydra_register_local_tensors_for_rpc, llama_hydra_lock_compute / unlock_compute / llama_hydra_enable_shared_backend_compute_lock / llama_hydra_force_sync_if_shared, llama_hydra_clear_combined_bindings, llama_hydra_get_compute_backends, llama_hydra_preload_rpc_device. The dual-load is the feature: it provides the per-request SOLO↔COMBINED toggle (0x44) that stock llama.cpp's load-time-fixed --override-tensor cannot.
• src/llama-model.h:320-322 — ffn_gate_exps_rpc, ffn_down_exps_rpc, ffn_up_exps_rpc fields on llama_layer. Stay.
• src/llama-context.cpp:423-432 — llama_context::hydra_add_combined_rpc_backend. Stays (required for the scheduler to know about the peer's backend).
• src/llama-context.cpp:2433-2441 — RAII compute lock. Stays (required for the same-host dual-role case: the 3060 serves its own SOLO and the head's RPC traffic on the same backend).
• src/models/qwen35moe.cpp:507-513 — per-layer ternary selecting between ffn_*_exps and ffn_*_exps_rpc. Stays.
• tools/llama-engine/hydra_rpc/ module (added by PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37). Stays.
• ggml/src/ggml-rpc/ RPC_CMD_RESOLVE_TENSOR (PR fork: zero-copy COMBINED expert tensors (name-based remote binding, replaces network copy) #20). Stays — the dual-load zero-copy bind uses it.

Sub-phase 2a — Hydra.Core C# refactor (parent repo only, no fork change)

Files (in ddvnguyen/hydra_vortex parent):

• src/coordinator/MultiEngineRouter.cs — drop the OT-vs-static branch. The router emits a single Plan shape whose payload is a stock common_params JSON (or delta).
• src/coordinator/WorkerSchedulerService.cs — model registry added. Keys by alias. Each entry holds { model_path, default_ctx_size, default_kv_quant, default_n_cpu_moe, default_overrides[] }.
• src/coordinator/Plan.cs (wherever Plan is defined) — Plan.OtSplit and Plan.CombinedOtSplit removed. Replaced by Plan.CommonParams (the full stock-params JSON) or by fields Plan.OverrideTensors, Plan.TensorSplit, Plan.SplitMode, Plan.RpcServers.
• src/core/Hydra.Shared/RpcClient.cs:499 — the ot_split JSON key replaced with a stock-params payload.
• src/core/Hydra.Shared/Protocol.cs:55-58 — EnginePipelineAttach payload (ot_split field) updated to the new stock-params shape (or removed if 0x46 is also retired in favour of 0x40; see Q46 below).
• infra/hydra-core/config/workers.json and workers-27b.json — run_type: "combined-static-peer" and combined_ot_split fields removed. Replaced with combined_peer: bool + stock args in the worker config.
• infra/hydra-head/config/node-rtx.yaml, node-rtx-27b.yaml, node-rtx3060.yaml, node-rtx3060-27b.yaml — replace the Hydra-specific args with stock llama.cpp args passed through directly.

C# still emits 0x44 SET_EXPERT_MODE for MoE COMBINE plans (translator layer under the hood), so the live MoE dual-load path keeps working. This sub-phase ships in production alongside the current fork — no engine-side change, no production risk.

Sub-phase 2b — Fork runtime reconfigure + flag cleanup

Files (in fork):

• tools/llama-engine/llama-engine.cpp:

  • Delete the 5-flag deprecation shim at tools/llama-engine/llama-engine.cpp:201-214 (added by PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37, now obsolete). The flags themselves are also removed from argparse.
  • Delete the old start_shared_backend_rpc_server and start_backend_rpc_peer_server paths in the no-model-peer block. PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37 already replaced them with hydra_rpc::start; verify no callers remain and remove.
  • Add a runtime reconfigure path in the main loop: receive 0x40 EngineConfigure with a full common_params JSON payload, diff against the last-applied snapshot, decide which lifecycle point is affected (model reload / context rebuild / per-request setting), tear down + rebuild as needed, return the actual state. The diff walks the common_params struct field by field; T3 fields trigger model teardown, T2 fields trigger context teardown, T1 fields apply per-request. T4 fields are rejected.
  • The 0x40 payload schema is common_params-shaped: { model, n_ctx, n_batch, n_ubatch, split_mode, tensor_split, override_tensors: [string], rpc_servers: [string], n_gpu_layers, n_cpu_moe, cache_type_k, cache_type_v, flash_attn, rope_*, yarn_*, ... }. The engine applies via a common_params_parse-equivalent on the new params.
  • The 0x44 SET_EXPERT_MODE opcode stays (separate from 0x40). The mode is cparams.hydra_expert_mode; flipping it forces a graph rebuild (the mode is in the graph-reuse key) but not a model reload.
  • The EngineConfigure (0x40) handler in tools/server/server-context.cpp is extended to apply the stock-params JSON (replacing the current per-mode branch).

• tools/llama-engine/llama-engine.cpp CLI parser:

  • Delete: --rpc-engine, --combined-ot-pattern, --combined-split-mode, --combined-tensor-split, --peer-only, --ggml-rpc-port.
  • Add (stock llama.cpp args, passed through unchanged): --rpc-server <host:port> (adds a remote device — replaces --rpc-engine), --rpc-server-port <port> (exposes local CUDA — replaces --ggml-rpc-port), --override-tensor <pat=backend> (replaces --combined-ot-pattern; the =CPU / =PEER / =RPC0 suffix is stock llama.cpp syntax already), --split-mode <none|layer|row>, --tensor-split <a,b,...>.
  • The --model flag stays (still required for SOLO + dense + MoE loads). For the no-model peer case, omit --model and pass --rpc-server-port <port> only.
  • The --config <yaml> flag is added (Layer 2 startup config; see "Three-layer startup config" in Decisions made).

• src/llama-hydra.cpp:178-192 (llama_hydra_clear_combined_bindings): no change. Still used by the dual-load rebind path for the 0x44 handler's per-rebind cleanup.

Files that stay unchanged:

• src/llama-hydra.h (full file, all 152 lines) — every API stays.
• src/llama-hydra.cpp lines 1-435 — every function stays.
• src/llama-model.h:320-322 — *_rpc fields stay.
• src/llama-context.cpp:418-432 (hydra_add_combined_rpc_backend) — stays.
• src/llama-context.cpp:2433-2498 (RAII compute lock) — stays.
• src/models/qwen35moe.cpp:507-513 — per-layer ternary stays.
• tools/llama-engine/hydra_rpc/ (PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37 module) — stays.
• ggml/src/ggml-rpc/ RPC_CMD_RESOLVE_TENSOR (PR fork: zero-copy COMBINED expert tensors (name-based remote binding, replaces network copy) #20) — stays.
• tools/server/server-context.cpp:3536-3666 (SET_EXPERT_MODE handler) — stays (the 0x44 opcode is still served; the new 0x40 semantics don't replace it).
• specs/rpc-protocol.md:219-227 (0x44 SET_EXPERT_MODE description) — stays.

Test plan:

Test	Sub-phase	What it verifies
tests/test-hydra-rpc-accept-loop.cpp (from Phase 1)	regression	MSG_PEEK dispatch still works after the flag cleanup
tests/test-hydra-bounded-thread-pool.cpp (from Phase 1)	regression	Pool still bounded
tests/test-hydra-solo-regression.cpp (from Phase 1)	regression	SOLO mode unchanged
tests/test-hydra-runtime-reconfigure.cpp (NEW)	2b	The reconfigure path correctly diffs common_params payloads, rebuilds model/context as needed, returns the new state. Covers: ctx-only changes, model-load changes, rpc-server list changes, override-tensor changes, no-op diffs.
tests/test-hydra-common-params-passthrough.cpp (NEW)	2b	Every common_params field Hydra.Core might emit (split_mode, tensor_split, override_tensors, n_cpu_moe, n_gpu_layers, ctx_size, kv cache types, etc.) is applied correctly
tests/test-hydra-combine-mode-toggle.cpp (NEW)	2b	0x44 SET_EXPERT_MODE still toggles between SOLO and COMBINED per request without a reload (G6 acceptance)
tests/test-hydra-no-model-peer.cpp (NEW)	2b	llama-engine boots with no --model and --rpc-server-port <port>, exposes local CUDA, accepts 0x40 EngineConfigure with a model field (loads it) or without (stays bare)
tests/test-hydra-tier-classification.cpp (NEW)	2b	Each common_params field is classified into the right tier; the reconfigure path applies the right rebuild for each tier transition
tests/test-hydra-lazy-load.cpp (NEW)	2b	Engine boots model-less, returns "no model" on first request without 0x40, loads on first 0x40, second 0x40 with same params is a no-op
tests/test-hydra-three-layer-config.cpp (NEW)	2b	Layer 1 defaults + Layer 2 YAML + Layer 3 0x40 payload compose correctly; precedence is right
tests/coordinator/test_multi_engine_router.py (extended)	2a	The mode resolver returns the right stock-params for each (model, mode) combination — covers all 4 cells of the unified COMBINE table
tests/coordinator/test_plan_schema.py (NEW)	2a	The new Plan shape validates against a JSON schema; OtSplit field rejected if present (translator compatibility check)
tests/system/test_solo_rtx.py	2b regression	G3: SOLO 5060 Ti decode ≥ 190 tok/s (no regression)
tests/system/test_combine_moe.py	2b	G2: MoE 35B COMBINED decode ≥ 45 tok/s, dual-load still works end-to-end via 0x40 + 0x44
tests/system/test_combine_dense.py	2b	G1: DENSE 27B COMBINED decode via 0x40 with stock --tensor-split 21/44 --rpc-server works (≥ 24 tok/s — see G1 note)
tests/system/test_mode_switch.py	2b	G6: SOLO ↔ COMBINED < 100 ms round-trip via 0x44, no model reload
tests/system/test_lazy_load.py (NEW)	2b	Engine boot time < 1s; first 0x40 triggers model load; subsequent 0x40 with same params is a no-op

Acceptance:

• All Phase 1 tests still pass (no regression from the flag cleanup).
• The 5-flag deprecation shim is removed; the 6 flags themselves are removed from llama-engine.cpp argparse.
• llama-engine accepts --rpc-server, --rpc-server-port, --override-tensor, --split-mode, --tensor-split (stock llama.cpp args) and applies them at runtime via 0x40.
• The dual-load machinery (llama_hydra_load_combined_experts, *_rpc fields, 0x44 SET_EXPERT_MODE) still works; existing MoE dual-load tests pass.
• The no-model peer path still works; the dense 27B profile loads via 0x40 with stock --tensor-split 21/44 --rpc-server localhost:9506.
• The engine boots model-less by default; first 0x40 triggers a T3 model load; subsequent 0x40 with the same model + no T2/T3 changes is a no-op.
• The three-layer config (defaults + YAML + 0x40) composes correctly; precedence is right; T4 fields are rejected with a clear error.
• Hydra.Core emits a single Plan shape with stock-params payload; no OT-vs-static branching.
• workers.json schema simplified: combined_peer: bool instead of run_type strings; no combined_ot_split field.

NOT in this phase (deferred to Phase 3 or as separate issues):

• The tensor_buft_overrides extension for per-process layer ranges (Phase 3) — same as the existing plan; required for A→B partial load.
• The A→B POST /control/load_model route (Phase 3) — same.
• The startup_stage state machine and --peer-health-url (existing Q10) — same.

G1 number revision: the original G1 (DENSE 27B ≥ 50 tok/s on 5060 Ti + 3060 with 25/40 split) is aspirational; live Phase 1 measurement on this hardware is 24.70 tok/s (the 3060 is the bottleneck). Update G1 to "≥ 24 tok/s" or document the 3060 as the binding constraint. The 50 tok/s target was set before the v4 design's actual perf profile was measured; the v4 keeps the goal of layer-split working end-to-end, not the specific tok/s number. File a follow-up issue for the G1 number revision before Phase 2 starts; update the design doc (#35) accordingly.

Cross-links:

• PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37: Phase 1 (merged). The 5-flag deprecation shim added by PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37 is the cleanup target for sub-phase 2b.
• PR fork: zero-copy COMBINED expert tensors (name-based remote binding, replaces network copy) #20: zero-copy RPC_CMD_RESOLVE_TENSOR. Used by the dual-load machinery; not modified in this phase.
• hydra_vortex#392: parent epic in the parent repo. Phase 2 work in the parent repo is sub-phase 2a.
• hydra_vortex#398: per-task tracker in the parent repo.
• hydra_vortex#394: Phase C / quant-parity split, now separate.

Time: 1-2 weeks (2a is C# only, ~1 week; 2b is fork, ~1 week including the runtime reconfigure path).

Rollback plan: revert 2b PR if the reconfigure path has bugs. The 5-flag deprecation shim was a transitional aid; reverting the cleanup just leaves the shim back in place — Phase 1's behavior is preserved. 2a is independently rollbackable: the old Plan.OtSplit field can stay (deprecated) and the new stock-params path can be a parallel implementation until verified.

Open questions for this phase:

• Q46: Does 0x46 PIPELINE_ATTACH stay, get retired in favour of 0x40 stock --tensor-split, or get renamed? The v4 design's PIPELINE_ATTACH was the prima.cpp-style attach — that's now expressible as 0x40 EngineConfigure with stock params. Recommend retire 0x46 and fold into 0x40.
• Q47: When the 0x40 payload includes a model field that differs from the currently-loaded model, the engine must unload the current model first. Confirm this is acceptable (it is — llama_model_free is straightforward) and that the reconfigure path handles it.
• Q48: When the 0x40 payload includes rpc_servers with a peer that's unreachable, should the reconfigure fail (refuse to apply) or fall back to SOLO (apply, log a warning, return partial success)? Recommend fail — operator should know the peer is gone, not silently degrade.
• Q49: For the no-model peer path, when 0x40 EngineConfigure arrives with a model field, should the peer load the model (becoming a SOLO+COMBINE-peer dual-role engine, per CLAUDE.md:266) or refuse (peer is a pure RPC backend)? Recommend "load it" — the peer is dual-role by architecture principle, just initially bare-metal. The "peer (no model)" and "peer (model)" rows in the existing Mode-to-port mapping collapse into one row.

---

Phase 3: Per-engine model loading (A tells B)

Goal: A and B both have the model loaded from a known path on demand. A tells B which layers to load via HTTP. B's mmap covers only the kept layers. Memory savings: ~50% on B's VRAM + host mmap.

Files to modify:

File	LOC	Change
src/llama-model-loader.cpp	+30	Extend tensor_buft_overrides to match by layer range. Pattern layer:N-M matches tensors with tn.bid ∈ [N, M]. The existing pattern matching code at src/llama-model-loader.cpp:1154 is the hook point.
src/llama-hydra.cpp	+20	Add llama_hydra_set_keep_layer_range(ctx, lo, hi) setter. Called by the loader when the override matches a layer range.
tools/llama-engine/llama-engine.cpp	+80	Add POST /control/load_model route. Add extract_keep_layer_range_from_split(split, &lo, &hi) helper. A's start_combined_with_peer orchestrator parses the split, computes B's range, sends the load command via HTTP.
tools/llama-engine/llama-engine.cpp	+10	Add load_model_unload_old(ctx) helper: unload current model (if any), call llama_model_load_from_file with the new keep_layer_range.

Test plan:

Test	What it verifies
tests/test-hydra-partial-load.cpp	The tensor_buft_overrides parser matches layer:N-M correctly. The loader skips tensors outside the range. The n_tensors count is consistent with the range.
tests/test-hydra-keep-layer-range.cpp	llama_hydra_set_keep_layer_range correctly filters model loading. B's mmap is half the size of the full model.
tests/system/test_combine_partial_load.py	G4 acceptance: cold load (NVMe) ≤ 25 s for 27B Q4. G5 acceptance: warm load (page cache) ≤ 5 s. G6 acceptance: A→B parallel load → COMBINED ready in ≤ 25 s wall clock.
tests/system/test_mode_switch.py	G6 acceptance: SOLO → COMBINED → SOLO round trip < 100 ms (no model reload).
tests/system/test_profile_switch.py	G7 acceptance: DENSE 27B → MoE 35B → DENSE 27B end-to-end ≤ 60 s.

Acceptance:

• G4, G5, G6, G7 met.
• B's mmap is half the size of the full model for a 50/50 split.

Time: 1 week. PR title: fork: Phase 3 — A→B partial load via keep_layer_range.

Rollback plan: revert Phase 3 PR. Phase 1 + 2 stay. The A→B orchestration falls back to the full-model load (B mmaps the full GGUF; A's RPC_CMD_RESOLVE_TENSOR returns nullptr for out-of-range tensors; A's scheduler falls back to local CUDA for those). Performance degrades but correctness holds.

---

Phase 4: Hydra.Core orchestration (parent repo)

Goal: Hydra.Core sends the right config to the right engine at the right time. Profile switch (DENSE ↔ MoE) is bounded by G7.

Files to modify (in ddvnguyen/hydra_vortex parent repo):

File	LOC	Change
src/coordinator/MultiEngineRouter.cs	+30	Extended Select to emit a per-engine config block: {mode, peer, model, split, keep_layer_range}.
src/coordinator/WorkerSchedulerService.cs	+50	New SendEngineConfig(worker, config) method. Called when the head engine is selected.
src/coordinator/WorkerInfo.cs	+10	Add PeerEndpoint, CurrentModel, KeepLayerRange fields.
src/coordinator/HydraEngineClient.cs	+20	Add SetEngineConfigAsync(config) that POSTs to the engine's /control/set_expert_mode route.
infra/hydra-core/config/workers.json	+20	Add peer_endpoint, default_model, default_split per worker.
src/coordinator/WorkerConfig.cs	+40	Add ReloadAsync(newModel) method that calls WorkerConfig.Reload() with the new env vars (for profile switch).
src/coordinator/ProfileSwitcher.cs	+30 (NEW)	New service that coordinates the DENSE ↔ MoE switch: reload both engines in parallel, wait for capability advertisement, resume routing.

Test plan:

Test	What it verifies
tests/coordinator/test_multi_engine_router.py	Select emits the right config for each model + mode + GPU pair.
tests/coordinator/test_worker_config_reload.py	WorkerConfig.Reload updates env vars and restarts the engine.
tests/system/test_profile_switch.py (extended)	G7 acceptance end-to-end.
tests/coordinator/test_hydra_engine_client.py	SetEngineConfigAsync POSTs the right payload; engine acknowledges.

Acceptance:

• G7 met end-to-end.
• workers.json schema is documented and validated.

Time: 1 week. PR title: hydra: Phase 4 — coordinator orchestration + profile switch.

Rollback plan: revert Phase 4 PR. The new routes (/control/load_model, /control/set_expert_mode) are unused; engines continue to work as before. The profile switch UX is lost (manual restart required).

---

Phase 5: Profile + decide on the wire (deferred, opt-in)

Goal: measure per-token time on the 3060+5060 Ti pair in COMBINED layer-split mode. Decide whether to invest in AF_UNIX, SHM, or CUDA IPC.

Tasks:

1. Run nsys profile --stats=true ./llama-engine --model DENSE-27B-Q4 --port 8080 --rpc-port 9504 --combined-tensor-split 21/44 --rpc-engine localhost:9505 on a 200-token decode.
2. Capture kernel timeline, RPC dispatch time, scheduler time, lock wait time.
3. Identify the per-token time breakdown. If RPC_CMD_GRAPH_COMPUTE dispatch is > 20% of per-token time, proceed to Phase 6 (AF_UNIX). Otherwise, stop here.

Output: docs/perf-baseline-2026-07.md with timeline screenshots.

Time: 2 days. No code change.

---

Phase 6 (optional): AF_UNIX wrapper

Only if Phase 5 shows the wire is the bottleneck. Fork-isolated in tools/llama-engine/hydra_rpc/af_unix.{h,cpp}. Wraps ggml_backend_rpc_* to use AF_UNIX sockets (/var/run/hydra-<node>.sock) instead of TCP loopback. Wire format unchanged. ~2× lower dispatch latency on same-host. Cross-host stays on TCP.

Time: 1 week. PR title: fork: Phase 6 — AF_UNIX wrapper for same-host dispatch.

---

Phase 7+ (deferred): shared memory, CUDA IPC

Only if Phase 6 leaves the per-token path on the critical path. Skipped for the current hardware.

Non-goals (do NOT implement)

• A new ggml-RPC opcode set. We use upstream's opcodes + the Hydra protocol unchanged. The new thing is the merged server, not a new wire.
• COMBINED for MoE 35B as a throughput win over SOLO 5060 Ti (~48 vs 200 tok/s, by math). The MoE COMBINED win is resource-sharing (free the 5060 Ti), not throughput.
• Simultaneous DENSE 27B + MoE 35B residency (28 GB VRAM, not enough for both at usable quants).
• A custom CUDA shared-context multi-GPU (would require rewriting ggml's backend scheduler; out of scope).
• "Seamless" < 100 ms model swap between DENSE and MoE. The model reload dominates (~25 s); we don't keep both resident.
• PR fix: COMBINED layer-split startup crash + performance improvements #34's 7 fixes (CUDA context bind, PDL guards, warmup skip, etc.). These are mitigations; the root cause is upstream. Do NOT cherry-pick.
• PR engine: report COMBINED-static layer-split capability truthfully #33's capability reporting in this issue's scope. That's a separate PR; coordinate after Phase 1 lands.

Test strategy

Unit tests (in tests/)

File	Phase	What it verifies
test-hydra-rpc-accept-loop.cpp	1	MSG_PEEK dispatch; pool full → drop; shutdown unblocks accept
test-hydra-bounded-thread-pool.cpp	1	enqueue/try_enqueue/stop semantics
test-hydra-solo-regression.cpp	1	SOLO mode is unchanged
test-hydra-runtime-reconfigure.cpp	2b	The reconfigure path correctly diffs common_params, rebuilds model/context as needed
test-hydra-common-params-passthrough.cpp	2b	Every common_params field Hydra.Core might emit is applied correctly
test-hydra-combine-mode-toggle.cpp	2b	0x44 SET_EXPERT_MODE still toggles SOLO/COMBINED per request without reload
test-hydra-no-model-peer.cpp	2b	Engine boots with no --model and --rpc-server-port, exposes local CUDA
test-hydra-tier-classification.cpp	2b	Each common_params field is classified into the right tier
test-hydra-lazy-load.cpp	2b	Engine boots model-less, first 0x40 triggers model load
test-hydra-three-layer-config.cpp	2b	Defaults + YAML + 0x40 compose correctly; T4 fields rejected
test-hydra-resolve-mode.cpp	2b (regression)	RPC_CMD_RESOLVE_TENSOR round-trip + epoch check (still works after fork refactor)
test-hydra-partial-load.cpp	3	tensor_buft_overrides layer-range pattern
test-hydra-keep-layer-range.cpp	3	llama_hydra_set_keep_layer_range filter
test-hydra-orchestrator.cpp	3	A's start_combined_with_peer parses split correctly

E2E tests (in tests/system/)

File	Phase	Acceptance
test_solo_rtx.py	1	G3: SOLO 5060 Ti ≥ 190 tok/s
test_combine_dense.py	2b	G1: DENSE 27B COMBINED ≥ 24 tok/s (revised from 50; see Phase 2 G1 note), works via stock --tensor-split 21/44 --rpc-server
test_combine_moe.py	2b	G2: MoE 35B COMBINED ≥ 45 tok/s via 0x40 + dual-load
test_p100_pd.py	2b	G8: P100 P/D ≥ 28 tok/s (regression — unchanged)
test_lazy_load.py	2b	Engine boot < 1s; first 0x40 triggers T3 model load; same-params 0x40 is no-op
test_combine_partial_load.py	3	G4, G5, G6: load times + mode switch
test_mode_switch.py	3	G6: SOLO ↔ COMBINED < 100 ms
test_profile_switch.py	4	G7: DENSE ↔ MoE < 60 s end-to-end

CI gates (per phase)

Phase	Required green before merge
1	test-hydra-rpc-accept-loop, test-hydra-bounded-thread-pool, test-hydra-solo-regression, test_solo_rtx.py
2a	(parent repo) test_multi_engine_router.py (extended), test_plan_schema.py. No fork change.
2b	All of phase 1 + test-hydra-runtime-reconfigure, test-hydra-common-params-passthrough, test-hydra-combine-mode-toggle, test-hydra-no-model-peer, test-hydra-tier-classification, test-hydra-lazy-load, test-hydra-three-layer-config, test-hydra-resolve-mode (regression), test_combine_moe.py, test_p100_pd.py, test_lazy_load.py. test_combine_dense.py is gated on Phase 0 result.
3	All of phase 2 + test-hydra-partial-load, test-hydra-keep-layer-range, test_combine_partial_load.py, test_mode_switch.py
4	All of phase 3 + test_profile_switch.py

Configuration migration

Phase 1 (DONE via PR #37) removed the --ggml-rpc-port flag and consolidated to --rpc-port. The Phase 2 (unified COMBINE) refactor additionally:

• Removes the Hydra-specific flags --rpc-engine, --combined-ot-pattern, --combined-split-mode, --combined-tensor-split, --peer-only (the last five are no longer needed; PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37's deprecation shim is deleted in sub-phase 2b).
• Adds the stock llama.cpp flags --rpc-server <host:port> (replaces --rpc-engine), --rpc-server-port <port> (replaces --ggml-rpc-port for peer exposure), --override-tensor <pat=backend> (replaces --combined-ot-pattern; the =CPU / =PEER / =RPC0 suffix is stock llama.cpp syntax), --split-mode <none|layer|row>, --tensor-split <a,b,...>.
• Adds the --config <yaml> flag for the Layer 2 startup config (non-T4 defaults).

Config files to update:

• ddvnguyen/hydra_vortex/infra/hydra-core/config/workers.json — remove ggml_rpc_port, combined_ot_split, run_type from each worker. Add combined_peer: bool and the Layer 2 default defaults: block.
• ddvnguyen/hydra_vortex/infra/hydra-head/config/node-rtx.yaml — replace the --rpc-engine, --combined-ot-pattern, --combined-split-mode, --combined-tensor-split, --peer-only, --ggml-rpc-port lines with stock llama.cpp args (--rpc-server localhost:9506, --rpc-server-port <port>, --override-tensor <...>, --split-mode <...>, --tensor-split <...>).
• ddvnguyen/hydra_vortex/infra/hydra-head/config/node-rtx3060.yaml — same.
• ddvnguyen/hydra_vortex/infra/hydra-head/config/node-rtx-27b.yaml — same.
• ddvnguyen/hydra_vortex/infra/hydra-head/config/node-rtx3060-27b.yaml — same.

In Phase 3, add a peer_endpoint field per worker (e.g., peer_endpoint: "localhost:9505").

In Phase 4, add a default_model and default_split per worker (e.g., default_model: "/models/DENSE-27B-Q4.gguf", default_split: "21/44").

Communication plan

When each phase lands:

1. Phase 1: comment on this issue with the PR link + the build/test results. Update the parent hydra_vortex#392 epic with status. File a monitoring issue if any alert fires.
2. Phase 2 (unified COMBINE): open sub-phase 2a PR against ddvnguyen/hydra_vortex (parent repo) with the C# refactor; cross-link this issue. Open sub-phase 2b PR against ddvnguyen/llama.cpp (this repo) with the runtime reconfigure path + flag cleanup. Comment on this issue with both PR links + the test results. If G1 number is revised (50 → 24 tok/s), file a follow-up issue documenting the 3060 as the binding constraint. Update PR fix: COMBINED layer-split startup crash + performance improvements #34 with a "superseded by Phase 2" notice (the scheduler feasibility finding is still valid; the crash is addressed by the v4 layer-split design).
3. Phase 3: comment on this issue. Update the parent hydra_vortex#392 epic.
4. Phase 4: comment on this issue. Update hydra_vortex#392. The profile switch is now usable end-to-end.

When each phase breaks (rolls back):

1. Comment on this issue with the root cause.
2. File a review-finding issue per hydra_vortex's ## Coding Agent Rules with the [Llama-Engine-P1-...] label.
3. Do not merge the next phase until the rollback is understood.

Open questions

(Updated after Phase 2 design pass. Q1–Q4 are RESOLVED — see "Decisions made" at the top of the Phased implementation plan. Remaining open questions are Q5–Q10 from the v4 design doc plus Q46–Q49 from the Phase 2 refactor.)

1. Model storage location. Each engine mmap's the same GGUF. Same path on every engine? Shared mount? Different paths? (Affects G4 cold-load time.) — still open
2. Engine startup model config. RESOLVED in Phase 2: engines boot model-less by default; first 0x40 EngineConfigure triggers a T3 model load. The --config <yaml> flag is the optional Layer 2 startup defaults.
3. Per-request payload to engine A. RESOLVED in Phase 2: the payload is a common_params JSON delta. Anything in common_params is supported (subject to tier classification). Peer engines do not need per-request config; they're configured once at startup and accept inbound 0x40/0x44 from the head.
4. Where does the split ratio come from? RESOLVED in Phase 2: operator-configured in the YAML defaults: block and overridable per-request via the 0x40 tensor_split field. The unified COMBINE resolver emits the right split per (model, mode) combination.
5. Per-request peer switch. If A is currently in COMBINED with B and a new request asks for COMBINED with C, do we tear down B and bring up C inline (blocks the request), or queue and tear down async? (Affects request latency tail.) — still open
6. tensor_buft_overrides extension syntax. Is the layer:N-M pattern sufficient, or do we need a more expressive syntax (e.g., layer:N-M.*.ffn_.*=CPU for subset of tensors within a layer range)? — still open (Phase 3 dependency)
7. B's /control/load_model auth. Per hydra_vortex#393, the per-request peer field needs a trust gate. The /control/load_model endpoint has the same surface. Default: only accept from loopback (--control-allow-from 127.0.0.1). Confirm the gate is sufficient. — still open (Phase 3 dependency)
8. PR engine: report COMBINED-static layer-split capability truthfully #33 capability reporting is a separate concern. After Phase 1 lands, PR engine: report COMBINED-static layer-split capability truthfully #33 should be re-reviewed to ensure it still works with the merged server (the set_hydra_capabilities call site may have moved). — still open (post-Phase 2)
9. PR fix: COMBINED layer-split startup crash + performance improvements #34's CUDA context bind is a possible real fix for one symptom of the crash. Should it be extracted into a minimal standalone PR and tested in isolation, rather than left in the broken PR fix: COMBINED layer-split startup crash + performance improvements #34? — still open (post-Phase 2)
10. The startup_stage state machine in PR fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31 was useful for the staged /health. We deferred it from Phase 1. If a phase needs it (e.g., Phase 3's A→B readiness gate), it's a small Phase 1.5 add. — still open
11. Phase 2 — 0x46 PIPELINE_ATTACH retirement. Fold into 0x40 stock --tensor-split or keep as a separate opcode? Recommend retire.
12. Phase 2 — T3 model swap on 0x40. Confirm llama_model_free + llama_model_load_from_file is the right unload-then-reload path; the reconfigure path handles it.
13. Phase 2 — unreachable peer in rpc_servers. Fail the reconfigure (operator must know the peer is gone) or fall back to SOLO with a warning? Recommend fail.
14. Phase 2 — no-model peer receives 0x40 with a model field. Load the model (becoming a dual-role engine per CLAUDE.md:266) or refuse? Recommend "load it" — the peer is dual-role by architecture principle, just initially bare-metal.

References

• Design: PR [design] Hydra llama-engine architecture: per-GPU engines + merged RPC + no-weight-transfer COMBINE #35 (this issue's source) and docs/hydra-llama-engine-architecture.md in that PR.
• Supersedes: PR fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31 (left as draft, with supersession comment), issue Unified RPC Server: one port, one binary, no extra flags #29 (design doc, has the RESOLVE_TENSOR contradiction).
• Related:
  • PR engine: report COMBINED-static layer-split capability truthfully #33: capability reporting fix (separate, post-Phase 1)
  • PR fix: COMBINED layer-split startup crash + performance improvements #34: scheduler limitation finding (root cause; not a fix)
  • hydra_vortex#376: startup crash (closed by Phase 2)
  • hydra_vortex#392: Unified RPC Server epic (parent)
  • hydra_vortex#394: split Phase C out of fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31 (independent)
  • hydra_vortex#393: per-request peer trust gate (Phase 3 dependency)
  • hydra_vortex#353: COMBINED first-PREFILL crash (closed by Phase 2)
  • hydra_vortex#395: graph_compute degrade-to-solo (Step 4 — separate)
• Pre-existing fork pieces (already in the submodule, NOT to be re-implemented):
  • RPC_CMD_RESOLVE_TENSOR and RPC_CMD_RESOLVE_TENSES (added by PR fork: zero-copy COMBINED expert tensors (name-based remote binding, replaces network copy) #20) — stable fork API, committed to maintaining
  • llama_hydra_register_local_tensors_for_rpc (added by PR fork: zero-copy COMBINED expert tensors (name-based remote binding, replaces network copy) #20)
  • ggml_backend_rpc_bind_remote_tensor (PR fork: zero-copy COMBINED expert tensors (name-based remote binding, replaces network copy) #20)
  • The epoch + re-callable registration plumbing (fix typo in chatLLaMa ggml-org/llama.cpp#368)
  • start_shared_backend_rpc_server (current ggml-RPC server in llama-engine.cpp) — DELETED in Phase 1, not reused
  • tools/llama-engine/hydra_rpc/ (PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37, Phase 1) — merged Hydra RPC server, KEPT in Phase 2
  • The dual-load machinery (KEPT in Phase 2, not deleted):
    • llama_hydra_load_combined_experts (src/llama-hydra.cpp:123-156)
    • llama_hydra_rebind_combined_experts (src/llama-hydra.cpp:209-362)
    • s_hydra_combined_bindings (src/llama-hydra.cpp:169)
    • ffn_*_exps_rpc fields on llama_layer (src/llama-model.h:320-322)
    • llama_hydra_set_expert_mode (src/llama-hydra.cpp:364-370)
    • llama_hydra_lock_compute / unlock_compute / enable_shared_backend_compute_lock (src/llama-hydra.cpp:402-429)
    • llama_hydra_register_local_tensors_for_rpc (PR fork: zero-copy COMBINED expert tensors (name-based remote binding, replaces network copy) #20)
    • The per-layer ternary in src/models/qwen35moe.cpp:507-513

How to use this issue

1. Read the design doc (docs/hydra-llama-engine-architecture.md in PR [design] Hydra llama-engine architecture: per-GPU engines + merged RPC + no-weight-transfer COMBINE #35) for the full context.
2. Check the open questions above. Resolve any blocking ones before starting the corresponding phase.
3. Run Phase 0 (scheduler feasibility). If it fails, file the upstream issue and pick Fallback A or B.
4. Implement Phase 1 (the merged server). This is the only phase that has no upstream-scheduler dependency. ~1 week.
5. Implement Phases 2–4 sequentially, each gated on the previous phase's acceptance tests.
6. Implement Phase 5 (profiling). Decide on Phase 6 (AF_UNIX) based on the result.
7. File the Phase 1 PR referencing this issue. Cross-link in the PR body.

Acceptance for closing this issue

This issue is closed when all of the following are true:

• Phase 1 is merged (DONE via PR fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37).
• Phase 2 (unified COMBINE refactor) is merged: sub-phase 2a (Hydra.Core C#) and sub-phase 2b (fork runtime reconfigure + flag cleanup).
• Phases 3–4 are merged (per-process model loading + coordinator orchestration), or superseded with rationale.
• G1–G9 are all met (or G1 is explicitly dropped with documented rationale — the G1 number is likely revised from 50 tok/s to ≥ 24 tok/s based on the 3060 bottleneck; the file-a-follow-up-issue step is in the Phase 2 acceptance).
• The merged server is in production for the 5060 Ti + 3060 same-host pair.
• The cross-host P100 path is unchanged (G8 met).
• The dual-load machinery (llama_hydra_*, *_rpc fields, 0x44 SET_EXPERT_MODE) is verified working in the new runtime-reconfigure flow.
• The PR fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31 supersession comment + this issue's design doc are the canonical references.

[ddvnguyen]

Design review

Overall the direction is sound — one merged server, one port per engine, layer-split-only, killing the dead COMBINED-OT expert paths, per-phase rollback, and the C3 no-SOLO-regression discipline are all clear improvements over the 3-server design. C6's rejection of PR #31's blanket fail-soft is right. But the plan's central risk assumption is inverted, and its de-risk step tests the wrong path.

Top issue: the "upstream scheduler limitation" (PR #34) is a misattribution

The plan gates the whole DENSE path (G1) on PR #34's conclusion that the ggml scheduler "assigns compute by inputs, not weights, and this breaks layer-split for DENSE," and makes Phase 0 the feasibility gate. The evidence says otherwise:

• Spike hydra_vortex#381 already ran standard --rpc -ts layer-split on DENSE 27B end-to-end and it worked — 620 prefill / 20.3 decode, coherent output. Same scheduler, same data-dependency placement of l_out-N. It did not crash, because the peer (rpc-server) had the layer weights (pushed via SET_TENSOR at load).
• PR fix: COMBINED layer-split startup crash + performance improvements #34 crashes because its peer is --peer-only — no model, no weights — and the RPC_CMD_RESOLVE_TENSOR binding that's meant to give the peer its weights isn't wired yet. So when the scheduler legitimately places layer-25 compute on RPC0, the peer can't reach layer-25's weights. PR fix: COMBINED layer-split startup crash + performance improvements #34's own primary root cause ("peer RPC server thread has no CUDA context bound") is a second, fork-side threading bug.
• So this is not an upstream scheduler limitation. The scheduler behaves normally; the fork half-removed weight transfer without completing the RESOLVE replacement.

Consequence for the plan: Phase 0 tests -ts + the upstream ggml-rpc-server (peer receives weights via SET_TENSOR) — i.e. the already-proven path — so it will pass and tell you nothing about Phase 2's actual mechanism (--rpc-engine + RESOLVE + peer-only). Phase 0 gives false confidence. Two ways out:

• (a) make Phase 0 exercise the real RESOLVE / peer-only path so it actually de-risks Phase 2, or
• (b) adopt the proven --rpc -ts path (spike Doesn't compile due to missing headers for memcpy and assert ggml-org/llama.cpp#381) as the primary and treat no-weight-transfer RESOLVE as an optional later optimization, not a gate on G1.

Other concrete risks

1. C6 (keep GGML_ABORT on normal ops) vs degrade-to-solo. With C6, a peer blip mid-inference aborts the head process (which owns CUDA0). For a "resource-sharing" COMBINE that's a robustness regression vs SOLO. The design correctly rejects PR fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31's per-op fail-soft (silent corruption) but names no alternative. State where recovery lives — the engine/Core lifecycle layer (drive only a verified-ready peer; on peer loss, tear the peer out and rebuild solo) — or explicitly accept "peer loss = head crash, Core restarts."
2. Thread pool size 2 + "drop on full." Dropping an active ggml-RPC compute connection manufactures exactly the peer-closes-connection → RPC_STATUS_ASSERT → abort failure that plagued PR fork: resolve #376 — make COMBINE mode RPC work (gate-then-degrade) #31, and it contradicts C6 (an abort trigger becomes a normal-load event). Size the pool to the max concurrent connections per COMBINE session; never drop an active compute connection.
3. G9 (≤75 LOC upstream) undercounts. Its budget scopes only ggml-rpc / ggml-cuda / ggml-backend, but Phase 2 edits src/llama-model-loader.cpp and src/llama-context.cpp (core, effectively upstream) and those aren't counted. ggml_backend_rpc_handle_client (serving a pre-accepted fd) also likely exceeds the claimed ~20 LOC once the accept loop is refactored out.
4. ROI of Phases 2–3 (the RESOLVE path). Vs -ts --rpc, VRAM is identical; the win is load-time bandwidth + B's host mmap halved. G4/G5 (25s/5s) are disk/mmap-bound, not RPC-bandwidth-bound. Measure -ts --rpc cold-load + peer RAM at 27B first; if it's already ≤25s, RESOLVE is a pure memory optimization that can sit behind a working baseline rather than gating it.
5. PR engine: report COMBINED-static layer-split capability truthfully #33 should fold into Phase 1. Phase 1 rewrites the llama-engine.cpp server-start code where PR engine: report COMBINED-static layer-split capability truthfully #33's set_hydra_capabilities call lives (open Q#8). Land it in Phase 1 or rebase immediately rather than leaving it dangling.

Smaller notes

• The 1-byte dispatch is valid: RPC_CMD_HELLO == 14 == 0x0E (there's already a static_assert), and Hydra 0x30–0x46 don't overlap ggml 0x00–0x11. Caveat: it relies on little-endian cmd serialization (fine for the x86-only fork) — worth a one-line comment where the peek happens.
• Of the open questions, the ones that actually block coding are feat: COMBINED expert-split two-engine mode (hydra_vortex issue #287/#260) #2 (model fixed-at-startup vs per-request — drives G6), [fork/hydra-316] checkpoint creation gate misses recurrent/hybrid models without rpc_port #4 (split-ratio source), and fix: include recurrent/hybrid models in checkpoint-creation gate (hydra_vortex #316) #5 (per-request peer-switch teardown — drives latency tail). Merge upstream b9501 + fix: context-checkpoint reuse for recurrent/hybrid (qwen35moe) #1/fork: make StateGet chunk size runtime-configurable, default 2MB #7 are ops/security and can trail.

What's good (keep)

Merged server + single port; layer-split-only; deleting the COMBINED-OT dead paths; the C3 no-regression discipline; per-phase rollback plans; and the non-goals are well-reasoned (MoE COMBINE = resource-sharing, not a throughput win). C6's rejection of blanket fail-soft is the right call.

---

Grounding: hydra_vortex#381 (working --rpc -ts DENSE 27B), PR #34 (the crash trace + --peer-only context), #31/#33 (the fail-soft and capability-reporting history).

---

Generated by Claude Code

[hydra-z]

Hydrazine — Phase 1 picked up

Cross-repo coordination (per ddvnguyen/hydra_vortex docs/workflow/08-llama-fork.md):

Link	What
Hydra parent epic	ddvnguyen/hydra_vortex#392 (already open)
Hydra per-task tracker	ddvnguyen/hydra_vortex#398 (just filed, milestone #6 "Llama-Engine - P/D split mix-quant")
Fork branch	ddvnguyen/llama.cpp:fork/hydra-392-phase-1-merged-rpc (just created at 84e65f1 = PR #30 merge)

Reality-check before Phase 1 lands

PR #30 (Unified RPC server — one port, one binary, no extra flags) merged 2026-07-05, two days before this issue was filed. The bulk of "Phase 1" as written here is already on hydra-fork:

• ✅ socket_t::from_fd (transport.h/cpp) — in fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30
• ✅ ggml_backend_rpc_handle_client (ggml-rpc.cpp) — in fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30
• ✅ ggml_backend_rpc_remove_server (ggml-rpc.cpp, with registry cleanup) — in fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30
• ✅ First-byte MSG_PEEK protocol detection — in fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30
• ✅ Removed --peer-only, --ggml-rpc-port, --rpc-port, --combined-ot-pattern, --combined-split-mode, --combined-tensor-split — in fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30
• ✅ --model optional, RPC port auto-derived — in fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30
• ✅ llama_hydra_clear_combined_bindings for runtime peer switch — in fork: Unified RPC server — one port, one binary, no extra flags (closes #29) #30

What remains for Phase 1 per this issue's plan: the refactor that moves the merged server from tools/server/server-context.cpp (where #30 put it) into a new fork-isolated tools/llama-engine/hydra_rpc/ module, plus a bounded thread pool (size 2).

Open-question resolutions (will be on the Phase 1 PR)

• Q8 (PR engine: report COMBINED-static layer-split capability truthfully #33 capability reporting): defer review until after Phase 1 lands; ask PR engine: report COMBINED-static layer-split capability truthfully #33 author to rebase + re-verify set_hydra_capabilities call site.
• Q9 (PR fix: COMBINED layer-split startup crash + performance improvements #34 CUDA context bind): do not extract now. Per the design, PR fix: COMBINED layer-split startup crash + performance improvements #34's 7 fixes do not address the actual crash (root cause is upstream). Defer unless a future phase surfaces it as needed.
• Q1–Q7, Q10: defer to later phases (not Phase 0/1 blockers).

needs-decision label

I am proceeding under the resolved defaults above. If you wanted a different call on Q8 or Q9, flag now.

Phase 0 (first deliverable, ~1 day)

I am about to build the existing hydra-fork fat binary (build_sm86_sm120 covering sm_86 + sm_120 with CUDA 13.2) and run a layer-split sanity test against the upstream ggml-rpc-server peer. The design calls for Qwen2-1.5B-Instruct-Q4_K_M but the user is substituting /mnt/SSD/Qwen3.5-9B-Q8_0.gguf for the quick test (real test with /mnt/SSD/Qwopus3.6-27B-Coder-Compat-MTP-Q5_K_M.gguf). Phase 0 result will be reported in hydra_vortex#398.

cc ddvnguyen/hydra_vortex#398 ddvnguyen/hydra_vortex#392