Skip to content

fix(#1200): drain async delegations before cron session teardown#1202

Merged
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1200-delegate-task-race
Jul 21, 2026
Merged

fix(#1200): drain async delegations before cron session teardown#1202
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1200-delegate-task-race

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Problem

Closes #1200 — Async delegate_task race in evolution-hydra: parent cron dispatches subagents but does not await completion before read-back, causing missing artifacts.

Root Cause

In cron/scheduler.py _run_job_impl, the inactivity timeout loop (lines 4195-4214) terminates as soon as the agent stops calling tools. The finally: block then calls _cron_pool.shutdown(wait=False, cancel_futures=True)without waiting for background subagents dispatched via delegate_task(background=true) that are still running on the daemon executor.

This orphans completion events and loses artifacts — the subagents' work vanishes when the cron session tears down before they finish.

Additionally, the inactivity timeout was hardcoded to 600s, which is too short for subagent-heavy cron jobs (the issue requests 900-1200s).

Fix

Three parts:

1. tools/async_delegation.py — drain helper

New wait_for_session_delegations(parent_session_id, deadline_seconds, poll_interval, on_interrupt) function:

  • Polls _running_for_session(parent_session_id) every poll_interval
  • Waits up to deadline_seconds for delegations matching the session to complete naturally
  • Force-interrupts stragglers via interrupt_for_session(parent_session_id=...) when deadline expires
  • Returns {waited, completed, interrupted, timed_out}

2. hermes_cli/config.py — config keys

Two new cron config keys with defaults:

  • delegation_drain_seconds: 1200 — drain deadline (matches heartbeat stale ceiling of 40×30s)
  • inactivity_timeout_seconds: 900 — up from hardcoded 600s

3. cron/scheduler.py — wire the drain

  • Inactivity timeout now reads cron.inactivity_timeout_seconds (default 900s) instead of hardcoded 600.0. HERMES_CRON_TIMEOUT env var still overrides.
  • finally: block calls wait_for_session_delegations() before _cron_pool.shutdown(). Configurable via cron.delegation_drain_seconds (default 1200s) or HERMES_CRON_DELEGATION_DRAIN_SECONDS env var.
  • Drain exceptions are caught and logged at debug — never block teardown.

Tests

tests/tools/test_async_delegation_drain.py (10 tests):

  • Empty session → immediate return
  • Natural completion before deadline → waited, not interrupted
  • Deadline expiry → force-interrupt remaining
  • Session scoping → only matches own delegations
  • Completed records not counted
  • on_interrupt callback invoked with count
  • on_interrupt exception swallowed (never propagates)
  • Mixed: some complete, some timeout

All 821 existing cron + delegation lifecycle tests still pass.

Verification

python -m pytest tests/tools/test_async_delegation_drain.py -v
# 10 passed in 2.56s

python -m pytest tests/tui_gateway/test_delegation_session_lifecycle.py tests/cron/ -v
# 821 passed in 36.23s

Footprint

No new core tools. The drain helper is a plain function in an existing module. The config keys are additive (deep-merge handles them). The scheduler change adds a drain step to an existing finally: block — no structural change to the loop.

Race condition: cron's _run_job_impl dispatched subagents via
delegate_task(background=true) but never waited for their completion
before tearing down the cron thread pool. The inactivity timeout loop
(lines 4195-4214) terminated as soon as the agent stopped calling tools,
and _cron_pool.shutdown(wait=False, cancel_futures=True) orphaned any
subagents still running on the daemon executor. This lost artifacts and
orphaned completion events.

Fix in three parts:

1. tools/async_delegation.py — new wait_for_session_delegations()
   helper. Polls _running_for_session() every poll_interval, waits up
   to deadline_seconds for delegations matching parent_session_id to
   complete naturally, then force-interrupts stragglers via
   interrupt_for_session(). Returns a dict with waited/completed/
   interrupted/timed_out counts.

2. hermes_cli/config.py — two new cron config keys:
     delegation_drain_seconds: 1200  (drain deadline, matches heartbeat
                                      stale ceiling of 40*30s)
     inactivity_timeout_seconds: 900  (up from hardcoded 600s)

3. cron/scheduler.py — three changes:
   a) Inactivity timeout now reads cron.inactivity_timeout_seconds from
      config (default 900s) instead of a hardcoded 600.0. HERMES_CRON_TIMEOUT
      env var still overrides.
   b) finally: block now calls wait_for_session_delegations() before
      _cron_pool.shutdown(), giving background subagents a chance to
      complete. Configurable via cron.delegation_drain_seconds (default
      1200s) or HERMES_CRON_DELEGATION_DRAIN_SECONDS env var.
   c) Drain exceptions are caught and logged at debug level — never
      block teardown.

Tests: tests/tools/test_async_delegation_drain.py (10 tests) covers
empty-session, natural-completion, deadline-interrupt, session-scoping,
completed-record filtering, on_interrupt callback (incl. exception
swallowing), and mixed completion+timeout. All 821 existing cron +
delegation tests still pass.
@Lexus2016
Lexus2016 merged commit 6028451 into main Jul 21, 2026
35 checks passed
@Lexus2016
Lexus2016 deleted the evolution/issue-1200-delegate-task-race branch July 21, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant