fix(guardrails): cap deferred-tool + memory spirals (#1185, #1186, #1187)#1189
Merged
Merged
Conversation
) Add tool_call, tool_describe, and memory to _SPIRAL_PRONE_TOOLS and _TOOL_FALLBACK_DIRECTIVE so the existing always-on per-tool failure cap covers the whole deferred-tool loading chain (search→describe→call) plus memory operations. Trace-miner evidence: - #1185: tool_call — 168 failures / 21 sessions, 13-deep spirals - #1186: memory — 94 failures / 21 sessions, 11-deep (regressed 10x from #1135/#1136) - #1187: tool_describe — 59 failures (search→describe→call middle step) These had no circuit breaker — the agent blind-retried the same failing call up to 13 consecutive times with no fallback directive. Extending the existing, tested spiral-cap mechanism covers them consistently with the core tools (terminal, execute_code, read_file, process, search_files). Closes #1185 Closes #1186 Closes #1187 Co-Authored-By: Hermes Evolution <evolution@hermes.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the existing always-on spiral-prone-tool failure cap to cover the whole deferred-tool loading chain (
tool_search→tool_describe→tool_call) plusmemoryoperations. These three tools had no circuit breaker — the agent blind-retried the same failing call up to 13 consecutive times with no fallback directive, while the core tools (terminal,execute_code,read_file,process,search_files) were already capped.This is the established, tested mechanism (
_SPIRAL_PRONE_TOOLS+_TOOL_FALLBACK_DIRECTIVEinagent/tool_guardrails.py): consecutive same-tool failures hit thespiral_failure_cap(default 5) and halt the turn with a concrete fallback directive, regardless ofhard_stop_enabled. Adding these three tool names is a 2-line-per-tool change that takes effect through the existingbefore_call/after_callguards — no new dispatch-site code, no new surface.Evidence (trace-miner, aggregated/anonymized, 7-day window)
tool_call: 168 failures / 21 sessions, 13-deep consecutive spirals. The agent loads a deferred tool via search/describe/call, the call fails (wrong args schema, tool unavailable, provider rejection), and it retries the identical call repeatedly instead of switching strategy.memory: 94 failures / 21 sessions, 11-deep consecutive retries. Regressed ~9x from the 10x that triggered [FIX]memorytool failures recur (10x in 7d) #1135/[FIX] memory: memory results look like failures 10x in 7d #1136 (those fixes closed completed but the signal climbed back). A single corrupted/locked store causes a long spiral that never recovers.tool_describe: 59 failures, the middle step of the search→describe→call chain. When it fails (tool not in catalog, schema fetch error, stale name), the agent has no schema to invoke the deferred tool and either re-describes the same name or falls back to re-runningtool_search(which itself spirals).Changes
agent/tool_guardrails.py(2 edits):_SPIRAL_PRONE_TOOLS— addedtool_call,tool_describe,memoryalongside the existing five, with a comment citing [FIX] tool_call failures recur — 168 failures / 21 sessions with 13-deep consecutive spirals #1185/[FIX] memory tool failures regress — 94 failures / 21 sessions (up from 10x that triggered #1135/#1136) #1186/[FIX] tool_describe failures recur — 59 failures in the deferred-tool loading chain #1187._TOOL_FALLBACK_DIRECTIVE— added actionable directives for each:tool_call→ runtool_searchfor an alternative, usetool_describeto validate schema, or invoke a core tool instead of blind-retrying.tool_describe→ re-runtool_searchto refresh the catalog, verify the exact name, don't keep describing the same failing name.memory→ distinguish busy/locked (transient, retry once) from genuine failure (skip-and-continue, log the unmet persistence need).tests/agent/test_tool_guardrails.py(+8 tests, mirroring the existingtest_cross_turn_process_spiral_cap_fires/test_cross_turn_search_files_spiral_cap_firespattern):action == "block",code == "spiral_prone_tool_failure_cap", and a non-empty fallback directive.>= 8tools, extending the existing>= 5invariant).Validation
scripts/evolution_pre_pr_test_runner.py): PASS (1.2s)ruff check: All checks passedWhy this approach (not error-message enrichment)
The issues also proposed enriching
tool_describe/tool_callfailure messages. On inspection, those dispatch paths (_dispatch_tool_describe_innerintools/tool_search.py, the bridge unwrap inmodel_tools.py) already produce structured, actionable errors: "not a deferrable tool, did you mean X", "not currently available, re-run tool_search to refresh", "'' is not available in this session. Use tool_search...". The missing piece was the circuit breaker — the cap is what stops the spiral; the error messages were already guiding the agent, it just had no reason to listen while it could keep retrying. This PR closes that gap with the smallest, most-tested surface.Closes #1185
Closes #1186
Closes #1187