Skip to content

fix: Cap fix retries and classify failures instead of retrying forever - #13

Open
Senna46 wants to merge 2 commits into
mainfrom
fix/retry-policy
Open

fix: Cap fix retries and classify failures instead of retrying forever#13
Senna46 wants to merge 2 commits into
mainfrom
fix/retry-policy

Conversation

@Senna46

@Senna46 Senna46 commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Problem

A bug whose fix could not be pushed was re-marked FAILED every cycle and retried indefinitely. Each retry re-ran a full claude -p fix generation (2–6 minutes) and threw the result away at the push step.

This is not hypothetical — on 2026-07-22 a PR touching .github/workflows/ failed to push because the PAT lacked the workflow scope, and Fixooly burned six full generations at ~2.5 minute intervals before the token was fixed by hand. Nothing in the code would have stopped it. With effortLevel: max each wasted generation now costs more.

A secondary effect: computeSinceForRepo dropped the since filter entirely whenever a repo had any FAILED bug, paginating every review comment in the repo each cycle (measured: 6 pages/cycle on cauchye/ai-talent-mapping, which has 572 review comments) for as long as the failure persisted.

Changes

Failure classification (src/failureClassifier.ts) — three classes, each with a different retry policy:

Class Examples Behavior
permanent branch protection / GH006, PAT missing workflow scope, permission denied give up immediately
transient Claude usage limit, expired or invalid auth, network errors retry, without spending an attempt
retryable generation errored or timed out retry, up to MAX_FIX_ATTEMPTS (3)

Transient outages deliberately do not consume the retry budget — a multi-hour usage limit must not permanently condemn a perfectly fixable bug.

Backoff — retries wait 5 → 15 → 30 minutes based on attempts so far, replacing the every-cycle hammering. Checked locally in bugbotMonitor before any API call, so a backed-off bug costs nothing.

Give-up reporting — when a bug is abandoned it is recorded as FAILED_PERMANENT (terminal) and a comment is posted on the PR with the reason and the error detail, deduplicated by a per-bug-set marker. The error text goes through sanitizeGitError so tokens cannot leak into a public comment.

Bounded since bypass — the widened window is now 30 days instead of unlimited.

Schema — adds processed_bugs.attempts with an in-place ALTER TABLE migration for existing databases, and switches writes from INSERT OR REPLACE to UPSERT so the counter is not silently reset on every status update.

Verification

npm run typecheck and npm run build pass. The state machine and classifier were exercised against a throwaway SQLite database — 25 checks, all passing:

  • transient failures ×5 keep attempts at 0 and never go permanent
  • counted failures increment 1 → 2 → 3 and stop exactly at the cap
  • FAILED_PERMANENT is terminal and clears the repo's retryable flag
  • attempts survives a later success (UPSERT, not REPLACE)
  • a legacy database without the column migrates in place with rows preserved
  • the classifier returns the expected class for the exact error strings found in the production logs (workflow-scope rejection, GH006, You've hit your limit, 401 OAuth access token has expired, EADDRNOTAVAIL, timeout, generic exit 1)

Running on the Mac mini daemon from this branch before merge.

🤖 Generated with Claude Code


Note

Medium Risk
Changes core daemon failure handling and SQLite state semantics (migration + new terminal statuses); misclassification could give up too early or retry too aggressively, but scope is localized to autofix retry logic.

Overview
Stops failed bugs from triggering a full claude -p run every poll cycle by classifying errors, capping counted retries, and backing off between attempts.

Adds failureClassifier.ts with permanent (give up + PR comment), transient (retry without spending an attempt; abort the rest of the cycle), and retryable (up to 3 attempts). SQLite state gains attempts, recordFailedBugs, backoff checks in bugbotMonitor, expireStaleFailures, and FAILED_PERMANENT; writes use upsert so attempt counts are not reset. Repo comment scan widens to 30 days when retryable failures exist instead of dropping since entirely. fixGenerator embeds CLI output in errors for classification; main posts deduplicated give-up comments with sanitized error detail. README documents the retry policy.

Reviewed by Cursor Bugbot for commit 17b3cdd. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d6b93fa. Configure here.

Comment thread src/main.ts
Comment thread src/bugbotMonitor.ts
Senna46 added 2 commits July 25, 2026 06:03
A bug whose fix could not be pushed was re-marked FAILED every cycle and
retried indefinitely, re-running a full claude -p fix generation every
~2 minutes. The d6e-valuation workflow-scope incident burned six full
generations before the token was fixed manually; nothing would have
stopped it.

- Classify failures: permanent (branch protection, missing token scope,
  denied permission) gives up at once; transient (usage limit, expired
  auth, network) retries without spending an attempt; anything else
  retries up to 3 times.
- Back retries off 5/15/30 minutes instead of every cycle.
- Post a PR comment when a bug is given up on, so the cause is visible.
- Bound the since-filter bypass to a 30-day window; dropping the filter
  entirely paginated every review comment in the repo on every cycle
  (6 pages/cycle on the busiest repo) for as long as any bug was failing.

Adds processed_bugs.attempts with an in-place migration, and switches
the writes to UPSERT so INSERT OR REPLACE cannot reset the counter.
…rows

Addresses both Bugbot findings on this PR, plus the push-token failure it
surfaced while running against its own PR.

- claude -p rejected with only a generic exit-code message, so the text
  that identifies a usage limit or expired auth never reached the
  classifier and every such failure was misread as retryable. The stdout
  or stderr tail is now part of the error message.
- A FAILED row that discovery stopped surfacing (comment deleted, or aged
  out of the window) held its repo on the widened 30-day scan forever.
  Rows untouched for 24h are retired to FAILED_PERMANENT each cycle.
- Invalid push credentials break every repo at once, so they are treated
  as transient: abandoning bugs and posting give-up comments across every
  PR while a token is rotated would be worse than waiting.
- On a transient failure the rest of the cycle is skipped. Otherwise one
  environment outage costs a full fix generation per PR to rediscover the
  same problem.
@Senna46
Senna46 force-pushed the fix/retry-policy branch from d6b93fa to 17b3cdd Compare July 24, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant