Skip to content

fix(buzz-acp): distinguish idle vs hard-cap timeout, dead-letter hard kills immediately#1844

Open
wpfleger96 wants to merge 3 commits into
mainfrom
duncan/timeout-hardcap-honesty
Open

fix(buzz-acp): distinguish idle vs hard-cap timeout, dead-letter hard kills immediately#1844
wpfleger96 wants to merge 3 commits into
mainfrom
duncan/timeout-hardcap-honesty

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Hard-cap timeouts (wall-clock limit) were mislabeled as idle-timeout "inactivity" and retried up to 10× — each cycle re-running the same >1h task from a fresh session before dying again, burning ~10h of compute on a single message. Additionally, the queue's in-flight backstop was hardcoded to the old 3600s cap + 100s buffer, meaning a legitimate 2h turn would get force-released at ~62 min while still running.

Five coupled fixes:

  • Plumb TimeoutKind { Idle, Hard } through PromptOutcome::Timeout so every downstream site can distinguish which clock fired. Split the combined IdleTimeout | HardTimeout cancel-drain arm in pool.rs to carry the correct kind. Observer outcome_label now emits "idle_timeout" or "hard_timeout" instead of the ambiguous "timeout".

  • Dead-letter hard-cap kills immediately in handle_prompt_result without calling queue.requeue(). Hard-cap failure is deterministic — a fresh session reproduces the same death. Idle timeout keeps its current retry behavior (may be a transient provider stall). The user-facing message now reads "Agent turn exceeded the maximum duration ({N}s)" instead of the misleading "Agent session timed out due to inactivity".

  • Raise max_turn_duration default 3600s → 7200s so genuinely long turns (heavy implementation work) have headroom. Idle stays at 900s, preserving the idle < max_turn invariant and the real hang detector.

  • Derive in-flight deadline from max_turn_duration instead of a hardcoded IN_FLIGHT_DEADLINE_SECS = 3700. The queue's EventQueue now carries a configurable in_flight_deadline field set to max_turn_duration + 100s at the prod construction site. Tests keep the default (7300s) via EventQueue::new(); prod calls .with_in_flight_deadline(). Adds DEFAULT_MAX_TURN_DURATION_SECS constant to config.rs and wires it into all test fixtures and the clap default_value_t.

  • Validate max_turn_duration upper bound at the config boundary (MAX_TURN_DURATION_CEILING_SECS = 604_800, 7 days). Rejects values above the ceiling with a ConfigError, preventing arithmetic overflow in the in-flight deadline derivation (max_turn + 100). The timeout requeue test (hard_timeout_not_requeued_idle_timeout_is_requeued) now asserts queued event counts via queued_event_count(), not just channel-key presence, verifying that idle timeout preserves events and hard timeout drops them.

Files changed

  • crates/buzz-acp/src/pool.rsTimeoutKind enum, split cancel-drain arm, outcome_label branching
  • crates/buzz-acp/src/lib.rshandle_prompt_result hard-cap divert, death_message / outcome_label / dead-letter reason split, prod EventQueue construction with .with_in_flight_deadline(), hardened timeout requeue test with real event batch and event-count assertions
  • crates/buzz-acp/src/config.rsDEFAULT_MAX_TURN_DURATION_SECS and MAX_TURN_DURATION_CEILING_SECS constants, upper-bound validation with ConfigError, default_value_t on clap arg, boundary tests
  • crates/buzz-acp/src/queue.rsin_flight_deadline field on EventQueue, with_in_flight_deadline() builder, queued_event_count() test accessor, replaced all IN_FLIGHT_DEADLINE_SECS usage with instance field, invariant tests

… kills immediately

Hard-cap timeouts (wall-clock limit) were mislabeled as idle-timeout inactivity
and then retried up to 10× — each cycle re-running the same >1h task from a fresh
session before dying again, producing ~10h of silent churn.

Three coupled fixes:

- Plumb `TimeoutKind { Idle, Hard }` through `PromptOutcome::Timeout` so every
  downstream site can tell which clock fired. Split the combined cancel-drain
  `IdleTimeout | HardTimeout` arm to carry the correct kind through.
- Dead-letter hard-cap kills immediately in `handle_prompt_result` without calling
  `queue.requeue()`. Hard-cap failure is deterministic; a fresh session will
  reproduce the same death. Idle timeout keeps its current retry behavior (may be
  a transient provider stall). Observer `outcome_label` now emits `"idle_timeout"`
  or `"hard_timeout"` instead of `"timeout"` for observability.
- Raise `max_turn_duration` default from 3600s to 7200s so genuinely long turns
  (heavy implementation work) have headroom. Idle stays at 900s, preserving the
  `idle < max_turn` invariant and the real hang detector.

Adds unit tests verifying hard timeout is NOT requeued and idle IS, and that
`outcome_label` differs between the two kinds.
@wpfleger96 wpfleger96 requested a review from a team as a code owner July 14, 2026 02:43
The queue's IN_FLIGHT_DEADLINE_SECS (3700) was hardcoded to the old
3600s max_turn_duration + 100s buffer. With the cap raised to 7200s,
a legitimate 2h turn hits the 3700s backstop at ~62 min and gets
force-released while still running — enabling duplicate same-channel
processing.

Replace the static constant with a configurable `in_flight_deadline`
field on EventQueue, derived from max_turn_duration_secs + 100s buffer
at the prod construction site. Tests keep the default (7300s) via
EventQueue::new(); prod calls .with_in_flight_deadline().

Also: harden the hard-timeout requeue test with a real event batch
instead of an empty FlushBatch, and add invariant tests asserting
in_flight_deadline > max_turn_duration.
Reject max_turn_duration above 604800s (7 days) at the config boundary
so the unchecked u64 addition in with_in_flight_deadline cannot overflow.
Make the requeue test assert queued event counts (not just channel keys)
to verify event preservation on idle timeout and event drop on hard timeout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant