server_context task queue is single-threaded: STATE_META starves behind blocking DECODE/PREFILL

[ddvnguyen]

Summary

STATE_META (used by Hydra Core's GetStateMetaAsync, added in ggml-org#465/#55 for slot-progress visibility) cannot be serviced while the slot it's asking about is actually busy with a long-running DECODE or atomic PREFILL. The query reliably times out for exactly the case it's meant to help diagnose.

Root cause

server_context's task queue (server-queue.cpp, server_queue::start_loop()) is strictly single-threaded: it pops one task from queue_tasks and calls process_single_task() synchronously, only proceeding to the next task once that call returns.

Hydra's DECODE handler (server-context.cpp, SERVER_TASK_TYPE_HYDRA_ENGINE_DECODE) runs its entire generation in a blocking while (n_decoded < n_predict) { llama_decode(...); ... } loop inside a single process_single_task() call, with no yield back to the queue. Same shape for atomic-mode PREFILL. Both get a 180s client-side timeout (hydra_handle_decode / hydra_handle_prefill), which is consistent with the engine expecting these calls to legitimately hold the thread for up to 3 minutes.

STATE_META goes through the same queue but only gets a 5s client-side timeout (hydra_handle_state_meta). So while a slot is genuinely busy — the case Hydra Core actually wants progress for — the META task sits behind the blocking DECODE/PREFILL task in the FIFO and cannot be processed until it finishes. The 5s client wait fires first, and GetStateMetaAsync returns null/error.

Given this deployment's decode rates (P100 @ 28 tok/s, RTX 3060 @ 60 tok/s), any generation over roughly 150-300 tokens guarantees the timeout. In practice, progressMeta will be null for most real busy periods, WorkerSchedulerService.PrefillAsync's stuck/slow classification falls back to the workload-aware EstimatedTokens timeout, and the "distinguish stuck from slow via live progress" value the feature was built for mostly doesn't materialize.

This is not already solved by the existing thread pool

tools/llama-engine/hydra_rpc/ (landed via #37 per the #36 design) added a bounded_thread_pool (2 → 8 workers, see hydra_rpc.cpp) to the merged RPC accept loop. That pool parallelizes TCP connection acceptance and first-byte (ggml-RPC vs Hydra) dispatch — its own comment says it exists so "the head's compute graph doesn't stall on the peer's dispatch path" for COMBINED layer-split ggml-RPC traffic. Once a connection is identified as Hydra, it's handed to hydra_handle_connection, which — per hydra_rpc.cpp's own comment — "still lives in server-context.cpp." Every Hydra opcode handler there (hydra_handle_state_meta, hydra_handle_decode, etc.), regardless of which pool thread accepted the connection, still does queue_tasks->post() + queue_results->recv_with_timeout() against the same single server_context::queue_tasks. The 8-way pool buys connection-level concurrency, not task-execution concurrency — the actual GPU-serialized inference loop is still (correctly, since one llama_context isn't safe for concurrent decode) single-threaded, and that's the layer this issue is about.

Related but distinct: #21 (closed) found a ~52s stall/crash in the ggml-rpc compute-lock (llama_hydra_lock_compute/unlock_compute) under concurrent COMBINED load — a different subsystem (cross-GPU graph-compute serialization), not server_context's task queue.

Impact

• Hydra Core cannot get real-time progress for a slot that's genuinely busy — the primary use case for Can it support avx cpu's older than 10 years old? ggml-org/llama.cpp#451.
• Stuck-vs-slow classification degrades to timeout-only heuristics during real load.
• Any future consumer of /slots/:id/state/meta (dashboards, other coordinators) hits the same wall.

Suggested direction (not scoped/committed)

Progress needs to be readable without going through queue_tasks. E.g.: DECODE/PREFILL write a small atomic/lock-free per-slot progress struct (tokens processed, start time) on each iteration; hydra_handle_state_meta reads that struct directly instead of posting a task. Needs a real design pass — not a quick patch — hence filing as tracked design work rather than bundling into ggml-org#451.

Refs

• Found during review of hydra_vortex#465 / llama.cpp#55 (slot progress visibility, Can it support avx cpu's older than 10 years old? ggml-org/llama.cpp#451)
• Related connection-pool work: llama-engine: per-GPU llama-engines + merged Hydra RPC + no-weight-transfer COMBINE #36 (design), fork: Phase 1 — merged Hydra RPC server in tools/llama-engine/hydra_rpc/ #37 (merged Hydra RPC + bounded thread pool)
• Related concurrency incident: fork: PREFILL latency under sustained concurrent local+remote load (~70s vs ~0.3s sequential) #21 (compute-lock stall under concurrent COMBINED load)