fix(#1188): browser_navigate spiral cap regression — 15-deep spirals#1199
Merged
Conversation
Two root causes for the browser_navigate cap=3 not enforcing: 1. bot_detection_warning misclassified as success (agent/display.py) browser_navigate returns success=True with bot_detection_warning when it lands on a Cloudflare/captcha page. _detect_tool_failure saw success=True and returned failed=False, so the guardrail counter never incremented. The model retried 15× against a blocked page that never loads. Fix: classify bot_detection_warning results as failures in both _detect_tool_failure and classify_tool_failure (the fallback classifier). 2. Intermittent success resets cross-turn streak (agent/tool_guardrails.py) A single successful navigation (e.g., to a different, fast-loading page between failing attempts) fully cleared _cross_turn_tool_failure_counts via pop(), resetting the streak to 0. The cap never accumulated. Fix: decay the streak by 1 on success instead of clearing it. Multiple consecutive successes still drain to 0 (genuinely recovered backend is not penalized), but a fail/succeed/fail pattern now accumulates toward the cap. Tests: 4 new tests + 2 updated tests in test_tool_guardrails.py, all 65 pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1188 —
browser_navigatefailures regress with 15-deep consecutive spirals across 30 sessions, despite the existing cap=3.Root Cause
Two independent root causes prevented the cap from enforcing:
1.
bot_detection_warningmisclassified as success (agent/display.py)browser_navigatereturns{"success": true, "bot_detection_warning": "..."}when it successfully navigates to a bot-detection/Cloudflare/captcha page (the browser-level navigation succeeded, but the page is a blocked page). The failure classifier_detect_tool_failuresawsuccess: trueand returnedfailed=False— so the guardrail counter never incremented. The model retried 15× against a blocked page that will never load properly.Fix: Added a
bot_detection_warningcheck to both_detect_tool_failure(production path) andclassify_tool_failure(fallback classifier) — whensuccess=trueANDbot_detection_warningis present, classify as[bot detection]failure.2. Intermittent success resets cross-turn streak (
agent/tool_guardrails.py)A single successful navigation (e.g., to a different, fast-loading page between failing attempts) fully cleared
_cross_turn_tool_failure_countsviapop(), resetting the streak to 0. The cap never accumulated because the patternfail, succeed, fail, succeed, fail...kept the counter below the threshold.Fix: Decay the cross-turn streak by 1 on success instead of clearing it. Multiple consecutive successes still drain to 0 (a genuinely recovered backend is not penalized), but a fail/succeed/fail pattern now accumulates toward the cap and eventually halts.
Verification
scripts/run_tests.sh tests/agent/test_tool_guardrails.py— 65/65 tests passscripts/run_tests.sh tests/agent/test_display.py tests/tools/test_browser_navigate_fallback.py tests/agent/test_tool_result_classification.py— 75/75 passFiles Changed
agent/display.py—_detect_tool_failure: classifybot_detection_warningas failureagent/tool_guardrails.py—classify_tool_failure: mirror the bot_detection check;after_call: decay cross-turn streak by 1 on success instead of clearingtests/agent/test_tool_guardrails.py— 4 new + 2 updated tests