fix(scheduler): retry rate-limited comment fetches with backoff - #1459
Conversation
Root-caused why github-hourly-review-repair.yml's most recent run
inspected 50 PRs and dispatched zero autofixes (autofix_dispatches: 0),
which is why 40 of .github's 81 open PRs were stuck reporting "This
branch has conflicts that must be resolved" with no automatic repair ever
reaching them: every candidate PR's decision read "error: API rate limit
exceeded for installation ID ...".
Two compounding causes in pr_review_fix_scheduler.py:
1. issue_comments() fetched a PR's entire issue-comment history with the
default 30-per-page pagination, even though recent_fix_marker_exists()
only ever needs the most recent marker.
2. process_queue()'s concurrent comment-prefetch (up to 10 simultaneous
`gh api --paginate` calls against the same shared, org-wide-contended
OpenCode app installation) silently swallowed a failed fetch and then
had inspect_pr() immediately retry the same doomed call sequentially
with zero backoff, doubling the wasted request volume for every
already-failing PR.
issue_comments() now requests per_page=100 (cutting page count for long
comment threads by up to 3x) and retries a detected rate-limit error with
a short linear backoff (up to 2 attempts) before propagating.
process_queue() now caps prefetch concurrency at 4 workers instead of 10,
and a PR whose comment fetch still fails after retries is deferred to the
next scheduled pass ("wait") instead of being silently swallowed and then
redundantly re-fetched and reported as a scary "error".
This is a single shared script, so the fix applies identically to every
one of the ~19 product-specific hourly review-repair callers, not just
.github's own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthrough시간별 PR 리뷰 복구 스케줄러가 댓글 조회에 ChangesPR 리뷰 복구 스케줄러
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Scheduler
participant issue_comments
participant GitHubAPI as GitHub API
participant inspect_pr
Scheduler->>issue_comments: PR 댓글 조회(per_page=100)
issue_comments->>GitHubAPI: 댓글 API 요청
GitHubAPI-->>issue_comments: 댓글 또는 rate-limit 오류
issue_comments->>GitHubAPI: 필요 시 backoff 후 재시도
issue_comments-->>Scheduler: 댓글 결과 또는 조회 오류
alt 조회 성공
Scheduler->>inspect_pr: PR 검사
else 조회 실패
Scheduler->>Scheduler: "wait" 상태로 다음 실행에 연기
end
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Standing down on the required Evidence: a broad sample of recent Ironically apt: this PR fixes the mechanism that should eventually help clear that same conflict backlog, including #1441. Will keep this PR watched rather than guess at a fix for the deeper Strix-dispatch gap without further diagnosis. Generated by Claude Code |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| "-f", | ||
| "per_page=100", |
There was a problem hiding this comment.
🔴 Comment retrieval becomes a write request
-f per_page=100 changes the request to POST without a comment body. Every candidate is deferred, so hourly repairs stop dispatching.
| "-f", | |
| "per_page=100", | |
| "-X", | |
| "GET", | |
| "-f", | |
| "per_page=100", |
Was this helpful? React with 👍 or 👎 to provide feedback.
…eduler-rate-limit-backoff # Conflicts: # CHANGELOG.md
Devin Review found both immediately after each PR merged (bypass-merged past the org-wide opencode-review outage); both are real defects the local test suites' mocks couldn't catch since they never exercised gh's actual HTTP-method-defaulting or GitHub's actual statuses-history shape. 1. issue_comments() (#1459) added -f per_page=100 to its gh api call with no explicit -X GET. gh api defaults to POST once any -f/-F field is present unless -X overrides it, so every comment fetch became a malformed POST against the comment-creation endpoint -- failing every call outright, the opposite of this fix's purpose. Now pins -X GET. 2. rest_pr_node() (#1456) fetched classic statuses from the plural commits/{sha}/statuses endpoint, which returns full history with no dedup -- a stale success could outlive a later real failure for strix_evidence_state(). Switched to the singular, combined commits/{sha}/status endpoint, which already reports only the most recent status per context. Co-authored-by: Claude <noreply@anthropic.com>
2 participants
Summary
Investigating the user's report that many
.githubPRs show "This branch has conflicts that must be resolved," I surveyed all open PRs across.github,noema, andcontextual-orchestrator: 40 of.github's 81 open PRs aremergeable_state: dirty. This repo already has a purpose-built, safe, automated conflict-resolution mechanism for exactly this (scripts/ci/pr_review_fix_scheduler.py→.github/workflows/pr-review-autofix.yml, gated so the resulting head is fully re-reviewed and re-checked before it can merge), dispatched hourly bygithub-hourly-review-repair.yml. So rather than hand-resolving 40 conflicts (duplicative, risky, and not scalable — the same mechanism also needs to keep working for future conflicts), I root-caused why that mechanism has stopped working.Root cause: the most recent hourly run inspected 50 candidate PRs and dispatched zero autofixes (
"autofix_dispatches": 0), with essentially every candidate PR's decision reading"error": "API rate limit exceeded for installation ID 141441800"(the shared OpenCode GitHub App installation, contended by many concurrent org-wide scheduled workflows). Two compounding causes inpr_review_fix_scheduler.py:issue_comments()fetched a PR's entire issue-comment history with the default 30-per-page pagination, even thoughrecent_fix_marker_exists()only ever needs the most recent marker comment.process_queue()'s concurrent comment-prefetch (up to 10 simultaneousgh api --paginatecalls) silently swallowed a failed fetch (except Exception: pass) and then hadinspect_pr()immediately retry the same doomed call sequentially with zero backoff — doubling the wasted request volume for every already-failing PR.Developer experience
issue_comments()now requestsper_page=100(cutting page count for long comment threads by up to 3x) and retries a detected rate-limit error with a short linear backoff (up to 2 attempts) before propagating.process_queue()now caps prefetch concurrency at 4 workers instead of 10, and a PR whose comment fetch still fails after retries is deferred to the next scheduled pass ("wait") instead of being silently swallowed and then redundantly re-fetched and reported as a scary"error".contextual-orchestrator-hourly-review-repair.yml, etc.), not just.github's own.User experience
The hourly conflict/review-feedback autofix loop should start actually dispatching repairs again instead of erroring out on nearly every candidate PR, gradually clearing the current 40-PR
dirtybacklog (bounded by the existingmax_dispatches: 1-per-hour setting, left unchanged in this PR — a separate, more speculative lever I didn't want to mix into this root-cause fix).Test plan
is_rate_limit_errorsignature matching,issue_commentsretry-then-succeed / exhaust-retries-then-raise / no-retry-for-non-rate-limit-errors,process_queuedeferring a PR whose comment fetch failed (both the single-PR sequential path and the multi-PR concurrent path), proving only the failing PR is deferred while others proceed normally.coverage run -m pytest tests -q(full suite) — 1936 passed, 1 skipped, 21 subtests passedcoverage report --show-missing— 100%interrogate— 100%Generated by Claude Code
Summary by CodeRabbit