Repository navigation
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Create a thread on Grok and grow the native session large (several MB of history, many tool calls).
- Switch the thread to another provider (e.g. Codex), run one turn, switch back to Grok so a
delta_since_target_last_seenhandoff is pending. - Send a message. Turn start fails with
ContextHandoffBackgroundWorkErroron every retry (4 attempts), and on every later message. - Wait 30 min:
idle-release-deferredis logged for the provider session although nothing runs.
Expected behavior
Replayed history (_meta.isReplay: true) during session/load must not create wake/background state; recovery into a fresh native session should proceed. Also, for providers that do not report usage, a zero budget should not be derived from raw history bytes alone.
Actual behavior
A long-lived thread was switched Grok -> Codex -> Grok. Every new turn on Grok now fails before any prompt is sent with:
ProviderAdapterTurnStartError ... [cause]: ContextHandoffBackgroundWorkError: The target provider still has background work running.
Only initialize + session/load are sent to Grok; no session/prompt. The thread is unusable on Grok; provider-session.detach or a restart does not help because the next session/load replays the same history again.
Diagnosis (from reading the 0.1.9 bundle):
- Handoff budget is 0. Grok does not report context usage, so
contextUsageForHandofffalls back to estimating the native context from history size (~6.36 MB) which exceeds the 256k window. The pending delta handoff (~116 KB) is rejected withContextHandoffBudgetError. loadFromProviderthen takes the recovery branch (fresh native session + recovery handoff), but first checkssession.hasPendingBackgroundWork, which istrue->ContextHandoffBackgroundWorkError.- Why it is true on a freshly started runtime:
AcpSessionRuntimefilters replay (sessionUpdateIsReplay(...) -> return) only inside its ownacp.handleSessionUpdatecallback. But the runtime also exposeshandleSessionUpdate: acp.handleSessionUpdate, and the V2 adapter subscribesprojectAcpRuntimeSessionUpdatethrough it (wireAcpRuntimeHandlers->targetRuntime.handleSessionUpdate(...)). That path has noisReplaycheck; the only guard isflavor.ignoreSessionUpdate, which the Grok flavor does not define. WithenablePostSettleContinuation(Grok), replayedtool_call/agent_message_chunkupdates reachbufferPostSettleWake/trackRunningBackgroundToolswhenactiveTurn === null. For a large session the tail of the replay is processed afteractiveSessionIdis set, sowakeBuffer/runningBackgroundTaskIdsbecome non-empty and stay that way. - Evidence that the state is stuck (not in-flight): 30 min after the failed turn, with no turns running, the idle reaper logs
orchestration-v2.driver-session.idle-release-deferred(pinnedForMs: 0) for that provider session, i.e.hasPendingBackgroundWorkis still true. Overnight the same session was deferred every 30 min untilidle-release-pin-expiredafter 4h. Grok itself reports nothing running: all replayed_x.ai/session/updateevents carry_meta.isReplay: true, everytask_backgroundedhas a matchingtask_completed, and the final livebackground_taskssnapshot is[].
Small sessions do not reproduce because the replay finishes before session/load returns.
Impact
Blocks work completely (the thread cannot start any turn on Grok)
Version or commit
T3 Code 0.1.9 (server, orchestrator V2), self-hosted on Linux. Observed on 0.1.9 (runtime logs + bundle). Not re-run on 0.1.10/0.1.11. Source of tag v0.1.11 was read (not executed): the same path is unchanged. projectAcpRuntimeSessionUpdateEffect in packages/provider-acp/src/server/adapter.ts still has only the flavor.ignoreSessionUpdate guard (no isReplay check) and is attached via targetRuntime.handleSessionUpdate(...); AcpSessionRuntime filters replay only in its own acp.handleSessionUpdate callback; ProviderTurnStartService still throws ContextHandoffBackgroundWorkError when hasPendingBackgroundWork is true after the fresh-session budget check. hasPendingBackgroundWork additionally checks midTurnUnreportedCompletedTaskIds. Release notes 0.1.10/0.1.11 do not mention this.
Environment
Linux x86_64; Grok CLI 1.0.92 (grok agent --always-approve stdio), model grok-4.7-build-fast, effort xhigh. Native Grok session ~6.4 MB of history (~7k replayed notifications on session/load).
Logs or stack traces
ProviderAdapterTurnStartError ... [cause]: ContextHandoffBackgroundWorkError: The target provider still has background work running.
Only initialize + session/load are sent to Grok; no session/prompt. After 30 min idle with no turns running: orchestration-v2.driver-session.idle-release-deferred (pinnedForMs: 0). Overnight the same session was deferred every 30 min until idle-release-pin-expired after 4h.
Workaround
Detach the provider session (provider-session.detach) and clear the provider thread's nativeThreadRef so T3 creates a new native Grok session (loses native history; context must be re-supplied manually).
Suggested fix: skip sessionUpdateIsReplay(...) notifications in projectAcpRuntimeSessionUpdateEffect (or default ignoreSessionUpdate to isReplay for all ACP flavors), and/or ignore replay-derived wake state when evaluating hasPendingBackgroundWork for the handoff-recovery gate.
Related but different: #3560 (Grok resume replay floods thread, closed), #29211 (Copilot CLI session/load replay leak, closed), #25610, #20180, #29359, #32296 (send refused while background work runs), #38257 (Grok wait_all leaves subagents running).
(Report drafted with an AI agent; diagnosis from the 0.1.9 bundle and server logs.)
Activity