Bug: brv query returns "daemon is unresponsive" when concurrent brv curate runs on slow LLM backend

[Teasuti]

Bug Report: brv query returns "daemon is unresponsive" when concurrent brv curate runs on slow LLM backend

Description

When brv query and brv curate run concurrently in the same project, with a slow local LLM backend (e.g., qwen36_27b_heretic_gguf via lmstudio), the brv query command returns:

Error: the daemon is unresponsive in this task

This occurs even though both configurable timeout settings exist and have been verified to work independently. The issue specifically manifests when:

1. A long-running brv curate process is active (taking >30s due to slow LLM inference)
2. A concurrent brv query request arrives for the same project
3. The single agent per project misses tool-result events from the interrupted curate task
4. After 30 seconds without heartbeat activity, the client reports "daemon is unresponsive"

Steps to Reproduce

1. Set slow LLM backend (e.g., qwen36_27b_heretic_gguf via lmstudio) with extended timeout settings:

   brv settings set llm.requestTimeoutMs 600000  # 10 minutes
   brv settings set agentPool.maxConcurrentTasksPerProject 5
   brv restart

2. Start a long-running brv curate on the same project (takes >30s due to slow LLM):

   brv curate context "Your lengthy knowledge content here" &

3. While curate is still running, immediately issue a concurrent query:

   brv query "What do we know about X?"

4. Observe the error in the query output:

   Error: the daemon is unresponsive in this task
   

Expected Behavior

• Both tasks should execute concurrently without one blocking the other
• The query should return a valid synthesized answer from the context tree
• No "daemon is unresponsive" errors should occur when concurrent operations are within configured timeout limits

Actual Behavior

The brv query command returns an error immediately or after 30 seconds:

the daemon is unresponsive in this task

This happens because the single agent process for the project cannot handle both tasks simultaneously — when interrupted by a new task, it misses tool-result events from the previous task and reports as "unresponsive" due to missing heartbeat activity.

Environment Details

• ByteRover version: 3.16.1 (and earlier)
• OS: Linux x64 (Debian 13)
• Node.js: v22.22.3
• LLM backend: qwen36_27b_heretic_gguf via lmstudio (slow local inference, ~8+ minutes for complex tasks)

Root Cause Analysis

The real root cause is a client-side misperception: the transport heartbeat mechanism doesn't fire during a stalled event loop. When a brv curate task runs on a slow LLM backend, the agent process's event loop gets blocked processing the long-running LLM call. During this time:

1. The daemon process IS alive and working — it hasn't crashed or hung
2. The transport heartbeat (which checks if the daemon is responsive) relies on periodic activity in the agent loop
3. When the event loop is stalled waiting for a slow LLM response, no heartbeats are sent
4. After 30 seconds without heartbeat activity (_IDLE_HEARTBEAT_TIMEOUT_MS from constants.js), the client assumes the daemon is dead and reports "unresponsive"

This is NOT:

• Tool-result event loss (the events aren't dropped)
• A connection pool leak (related to brv curate hangs on multi-iteration LLM calls due to connection pool leak #441 but distinct)
• An LLM timeout issue (those are configurable and work independently)

The daemon is actually working — the client simply can't distinguish between "agent processing a slow LLM call" and "agent/process dead."

Why Existing Configurable Settings Don't Fix This Layer

Setting	What it controls	Why it doesn't fix this bug
llm.requestTimeoutMs	How long the client waits for an LLM response before timing out	Controls model-side timeout, not heartbeat detection. The 30s idle-heartbeat fires BEFORE the LLM timeout ever triggers.
agentPool.maxConcurrentTasksPerProject	Max concurrent tasks per project (default=5)	Even with concurrency=5, a single agent process's event loop can still be blocked by one slow task, starving heartbeat activity.

Relevant Constants from constants.js

• _IDLE_HEARTBEAT_TIMEOUT_MS = 30000 (30 seconds — the threshold that triggers "unresponsive")
• These are hardcoded client-side constants with no config override available

Proposed Fix Directions

1. Keep-alive heartbeats from daemon process: The daemon could emit heartbeat signals independently of agent loop activity, proving it's alive even when an event loop is stalled on a slow LLM call.

2. Separate query path for read-only queries: brv query (read-only synthesis) shouldn't need to go through the same agent pool as long-running curate tasks. A direct index search path could bypass the agent entirely.

3. Stall-aware client timeout: The client should detect when a task is in-flight but slow, rather than assuming silence = death. Could use in-flight task tracking + expected max-duration heuristics to differentiate "working slowly" from "dead."

Related Issues

• brv curate hangs on multi-iteration LLM calls due to connection pool leak #441 — CLOSE_WAIT connection pool leak
• brv curate hangs indefinitely on v3.8.3 #537 — Agent loop event handling issues
• Curation completes but no operations/knowledge files created - v3.10.1 #597 — Daemon process stability
• Feature request: Concurrency Limit for curating #641 — Heartbeat mechanism improvements
• brv mcp client stuck in infinite exception loop consuming 75–90% CPU for hours #660 — Concurrency and timeout configuration

Why This Issue is Distinct from Previous Reports

Previous issues (#441, #537, #597, #641, #660) addressed various aspects of daemon stability, connection pooling, agent loop handling, and timeout configuration. While these are real bugs that need fixing, they don't address the specific scenario where:

• The daemon is fully alive and working on a slow LLM inference task
• The client falsely reports it as dead due to missing heartbeats during event loop stall
• This happens specifically with slow local LLM backends (not cloud APIs)
• It occurs even when concurrency limits are increased above the default of 1

The fix requires changes at the heartbeat/health-check layer, not at the task scheduling or connection pooling layers.

[Teasuti]

Related: #441, #537, #597, #641, #660