fix(#1200): drain async delegations before cron session teardown#1202
Merged
Conversation
Race condition: cron's _run_job_impl dispatched subagents via
delegate_task(background=true) but never waited for their completion
before tearing down the cron thread pool. The inactivity timeout loop
(lines 4195-4214) terminated as soon as the agent stopped calling tools,
and _cron_pool.shutdown(wait=False, cancel_futures=True) orphaned any
subagents still running on the daemon executor. This lost artifacts and
orphaned completion events.
Fix in three parts:
1. tools/async_delegation.py — new wait_for_session_delegations()
helper. Polls _running_for_session() every poll_interval, waits up
to deadline_seconds for delegations matching parent_session_id to
complete naturally, then force-interrupts stragglers via
interrupt_for_session(). Returns a dict with waited/completed/
interrupted/timed_out counts.
2. hermes_cli/config.py — two new cron config keys:
delegation_drain_seconds: 1200 (drain deadline, matches heartbeat
stale ceiling of 40*30s)
inactivity_timeout_seconds: 900 (up from hardcoded 600s)
3. cron/scheduler.py — three changes:
a) Inactivity timeout now reads cron.inactivity_timeout_seconds from
config (default 900s) instead of a hardcoded 600.0. HERMES_CRON_TIMEOUT
env var still overrides.
b) finally: block now calls wait_for_session_delegations() before
_cron_pool.shutdown(), giving background subagents a chance to
complete. Configurable via cron.delegation_drain_seconds (default
1200s) or HERMES_CRON_DELEGATION_DRAIN_SECONDS env var.
c) Drain exceptions are caught and logged at debug level — never
block teardown.
Tests: tests/tools/test_async_delegation_drain.py (10 tests) covers
empty-session, natural-completion, deadline-interrupt, session-scoping,
completed-record filtering, on_interrupt callback (incl. exception
swallowing), and mixed completion+timeout. All 821 existing cron +
delegation tests still pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Closes #1200 — Async delegate_task race in evolution-hydra: parent cron dispatches subagents but does not await completion before read-back, causing missing artifacts.
Root Cause
In
cron/scheduler.py_run_job_impl, the inactivity timeout loop (lines 4195-4214) terminates as soon as the agent stops calling tools. Thefinally:block then calls_cron_pool.shutdown(wait=False, cancel_futures=True)— without waiting for background subagents dispatched viadelegate_task(background=true)that are still running on the daemon executor.This orphans completion events and loses artifacts — the subagents' work vanishes when the cron session tears down before they finish.
Additionally, the inactivity timeout was hardcoded to 600s, which is too short for subagent-heavy cron jobs (the issue requests 900-1200s).
Fix
Three parts:
1.
tools/async_delegation.py— drain helperNew
wait_for_session_delegations(parent_session_id, deadline_seconds, poll_interval, on_interrupt)function:_running_for_session(parent_session_id)everypoll_intervaldeadline_secondsfor delegations matching the session to complete naturallyinterrupt_for_session(parent_session_id=...)when deadline expires{waited, completed, interrupted, timed_out}2.
hermes_cli/config.py— config keysTwo new cron config keys with defaults:
delegation_drain_seconds: 1200— drain deadline (matches heartbeat stale ceiling of 40×30s)inactivity_timeout_seconds: 900— up from hardcoded 600s3.
cron/scheduler.py— wire the draincron.inactivity_timeout_seconds(default 900s) instead of hardcoded 600.0.HERMES_CRON_TIMEOUTenv var still overrides.finally:block callswait_for_session_delegations()before_cron_pool.shutdown(). Configurable viacron.delegation_drain_seconds(default 1200s) orHERMES_CRON_DELEGATION_DRAIN_SECONDSenv var.Tests
tests/tools/test_async_delegation_drain.py(10 tests):on_interruptcallback invoked with counton_interruptexception swallowed (never propagates)All 821 existing cron + delegation lifecycle tests still pass.
Verification
Footprint
No new core tools. The drain helper is a plain function in an existing module. The config keys are additive (deep-merge handles them). The scheduler change adds a drain step to an existing
finally:block — no structural change to the loop.