fix(buzz-acp): distinguish idle vs hard-cap timeout, dead-letter hard kills immediately#1844
Open
wpfleger96 wants to merge 3 commits into
Open
fix(buzz-acp): distinguish idle vs hard-cap timeout, dead-letter hard kills immediately#1844wpfleger96 wants to merge 3 commits into
wpfleger96 wants to merge 3 commits into
Conversation
… kills immediately
Hard-cap timeouts (wall-clock limit) were mislabeled as idle-timeout inactivity
and then retried up to 10× — each cycle re-running the same >1h task from a fresh
session before dying again, producing ~10h of silent churn.
Three coupled fixes:
- Plumb `TimeoutKind { Idle, Hard }` through `PromptOutcome::Timeout` so every
downstream site can tell which clock fired. Split the combined cancel-drain
`IdleTimeout | HardTimeout` arm to carry the correct kind through.
- Dead-letter hard-cap kills immediately in `handle_prompt_result` without calling
`queue.requeue()`. Hard-cap failure is deterministic; a fresh session will
reproduce the same death. Idle timeout keeps its current retry behavior (may be
a transient provider stall). Observer `outcome_label` now emits `"idle_timeout"`
or `"hard_timeout"` instead of `"timeout"` for observability.
- Raise `max_turn_duration` default from 3600s to 7200s so genuinely long turns
(heavy implementation work) have headroom. Idle stays at 900s, preserving the
`idle < max_turn` invariant and the real hang detector.
Adds unit tests verifying hard timeout is NOT requeued and idle IS, and that
`outcome_label` differs between the two kinds.
The queue's IN_FLIGHT_DEADLINE_SECS (3700) was hardcoded to the old 3600s max_turn_duration + 100s buffer. With the cap raised to 7200s, a legitimate 2h turn hits the 3700s backstop at ~62 min and gets force-released while still running — enabling duplicate same-channel processing. Replace the static constant with a configurable `in_flight_deadline` field on EventQueue, derived from max_turn_duration_secs + 100s buffer at the prod construction site. Tests keep the default (7300s) via EventQueue::new(); prod calls .with_in_flight_deadline(). Also: harden the hard-timeout requeue test with a real event batch instead of an empty FlushBatch, and add invariant tests asserting in_flight_deadline > max_turn_duration.
Reject max_turn_duration above 604800s (7 days) at the config boundary so the unchecked u64 addition in with_in_flight_deadline cannot overflow. Make the requeue test assert queued event counts (not just channel keys) to verify event preservation on idle timeout and event drop on hard timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hard-cap timeouts (wall-clock limit) were mislabeled as idle-timeout "inactivity" and retried up to 10× — each cycle re-running the same >1h task from a fresh session before dying again, burning ~10h of compute on a single message. Additionally, the queue's in-flight backstop was hardcoded to the old 3600s cap + 100s buffer, meaning a legitimate 2h turn would get force-released at ~62 min while still running.
Five coupled fixes:
Plumb
TimeoutKind { Idle, Hard }throughPromptOutcome::Timeoutso every downstream site can distinguish which clock fired. Split the combinedIdleTimeout | HardTimeoutcancel-drain arm inpool.rsto carry the correct kind. Observeroutcome_labelnow emits"idle_timeout"or"hard_timeout"instead of the ambiguous"timeout".Dead-letter hard-cap kills immediately in
handle_prompt_resultwithout callingqueue.requeue(). Hard-cap failure is deterministic — a fresh session reproduces the same death. Idle timeout keeps its current retry behavior (may be a transient provider stall). The user-facing message now reads"Agent turn exceeded the maximum duration ({N}s)"instead of the misleading"Agent session timed out due to inactivity".Raise
max_turn_durationdefault 3600s → 7200s so genuinely long turns (heavy implementation work) have headroom. Idle stays at 900s, preserving theidle < max_turninvariant and the real hang detector.Derive in-flight deadline from
max_turn_durationinstead of a hardcodedIN_FLIGHT_DEADLINE_SECS = 3700. The queue'sEventQueuenow carries a configurablein_flight_deadlinefield set tomax_turn_duration + 100sat the prod construction site. Tests keep the default (7300s) viaEventQueue::new(); prod calls.with_in_flight_deadline(). AddsDEFAULT_MAX_TURN_DURATION_SECSconstant toconfig.rsand wires it into all test fixtures and the clapdefault_value_t.Validate
max_turn_durationupper bound at the config boundary (MAX_TURN_DURATION_CEILING_SECS = 604_800, 7 days). Rejects values above the ceiling with aConfigError, preventing arithmetic overflow in the in-flight deadline derivation (max_turn + 100). The timeout requeue test (hard_timeout_not_requeued_idle_timeout_is_requeued) now asserts queued event counts viaqueued_event_count(), not just channel-key presence, verifying that idle timeout preserves events and hard timeout drops them.Files changed
crates/buzz-acp/src/pool.rs—TimeoutKindenum, split cancel-drain arm,outcome_labelbranchingcrates/buzz-acp/src/lib.rs—handle_prompt_resulthard-cap divert,death_message/outcome_label/ dead-letterreasonsplit, prodEventQueueconstruction with.with_in_flight_deadline(), hardened timeout requeue test with real event batch and event-count assertionscrates/buzz-acp/src/config.rs—DEFAULT_MAX_TURN_DURATION_SECSandMAX_TURN_DURATION_CEILING_SECSconstants, upper-bound validation withConfigError,default_value_ton clap arg, boundary testscrates/buzz-acp/src/queue.rs—in_flight_deadlinefield onEventQueue,with_in_flight_deadline()builder,queued_event_count()test accessor, replaced allIN_FLIGHT_DEADLINE_SECSusage with instance field, invariant tests