Skip to content

fix(ci): log pr-feedback-dispatch's rerun-ci-status and post-verify-notes API readings - #9882

Merged
kyle-sexton merged 4 commits into
mainfrom
fix/feedback-dispatch-call-budget
Oct 25, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
fix/feedback-dispatch-call-budget

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

No related issue: follow-up to the 2026-10-24 18:57-19:57Z GITHUB_TOKEN window analysis, which found about 2,000 of 4,015 core calls unattributed

Summary

pr-feedback-dispatch ran 91 times in that window and posted no api-budget notice for any of them. That made it the top suspect for the unattributed calls. Counting from the code (the per-path counts are pinned by scripts/ci-rerun-ci-status.test.sh) shows it spends about 200 to 250 calls an hour, with a ceiling near 550. So it is not the main source. This PR changes no call. It makes the two jobs that posted no notice post one, so the next window analysis can attribute their calls instead of guessing.

Metric: pr-feedback-dispatch github-token core calls per hour, and how many of its executing runs post an api-budget notice.

Baseline (window 18:57-19:57Z):

Trigger (upstream run) Runs in window Job that runs Calls per run (code)
pr-review-hosted / pr-review-security-hosted, pull_request, same repository 52 (26 + 26, 0 cancelled, 16 skipped start nothing) rerun-ci-status 1 if the other lane is still running; 3 if there is no run yet, or ci-status has not started or already read both lanes; 4 reads + 1 re-run POST = 5 to re-run; +1 per 60 s poll, at most 10, so 15 at most
the same, on a work-resolve-issue/issue-* head a subset of the 52 dispatch 0 when the head is recorded as handled; self-reported cost= (an upper bound, since it includes concurrent jobs' calls) of 1 to 3 on most of the 15 notices from the neighbouring windows, 18 and 19 on the full path
pr-verify, workflow_run on main 35 (34 skipped start nothing) post-verify-notes 1 (artifact listing) with no fail verdict; with one, the download, the verify-notes reads and one POST per note, then dispatch
pr-unblock-dispatch, schedule 1 to 2 dispatch (sweep) 1 PR listing + 1 check-run read per open lanes App PR, + the dispatch path for a released head
pr-unblock-dispatch, push 11 none (run skipped) 0

Estimate: rerun-ci-status 26 × 1 + 26 × 5 = 156, post-verify-notes 35 × 1 = 35, plus the dispatch and sweep runs, for about 200 to 250 calls an hour. Ceiling with every last-lane run hitting the 10-poll cap: 26 × 1 + 26 × 15 = 416 for rerun-ci-status, about 550 for the workflow. Notices posted: 0 of the 87 rerun-ci-status and post-verify-notes runs (the dispatch job already logs).

Expected change: calls per hour unchanged, since no call is added or removed and the log step reads no /rate_limit. Notices: from 0 to one per executing rerun-ci-status and post-verify-notes run (about 87 an hour). rerun-ci-status keeps a reading for every read, so its notice carries a cost=N whenever two readings fall in the same reset window.

Fix

  • rerun-ci-status: the sparse checkout adds /.github/actions/log-api-budget/. The find step sources api-budget.bash, and read_api passes each gh api -i response, refusals included, through api_budget_read. The job ends with the log-api-budget step. A failed checkout leaves no library, and the job runs as before with no reading.
  • post-verify-notes: the download step wraps gh with api_budget_wrap_gh github-token, and the job ends with the log-api-budget step. The notice covers only the artifact listing's reading, so it understates the job's spend: on the fail-verdict path, gh run download goes through the wrapper but is not gh api and leaves no reading, and verify-notes.mjs calls through address-feedback's createGitHubApi, a plain fetch outside the guard. Routing createGitHubApi through the guard is a separate follow-up, not done here.
  • .github/actions/log-api-budget on main has both functions; the rate guard has since merged, so the wrapped calls also stop while a limit holds.

Considered and not done:

  • cancel-in-progress: true: the group covers the dispatch job. Cancelling it between its dispatch POST and the dispatched-run record would leave a pr-unblock run that the gate and the broker refuse.
  • Skipping a cancelled review run in rerun-ci-status: the window had 0 cancelled review runs, so the saving is 0, and workflow-gates.test.mjs pins that such a run still re-runs.
  • Replacing the 60 s poll: it reads one run per minute for at most 10 minutes, and only while a ci-status that started before both lanes finished is still going. How often it polls is not measured here; its cost is from the code (at most 10 run reads per run, pinned by the test), and the new notices will measure it.

Verification

On the head merged with main:

  • bash scripts/ci-rerun-ci-status.test.sh: 152 passed. check_calls also requires one reading per API call on every counted path (1, 3, 4, 6, 13 and the rate-limited 5). New cases cover the refusal's reading and a missing library. It fails 14 checks on the base workflow.
  • node --test over log-api-budget.test.mjs, rate-guard.test.mjs, check-trusted-trigger.test.mjs, and both feedback-dispatch.test.mjs suites (pr-unblock and resolve-config): 537 passed. workflow-gates.test.mjs and callers.test.mjs: 205 passed. Two new log-api-budget.test.mjs cases fail on the base workflow. One requires each API-calling job to end with the log step. The other runs post-verify-notes' download body against a stub gh and checks it keeps the artifact listing's reading and still reports no fail verdict.
  • actionlint is clean on pr-feedback-dispatch.yml.

Related

  • Calls still not recorded: the re-run POST, gh run download and verify-notes.mjs on the fail-verdict path, and in the dispatch job, resolve-config's fact reads (resolveFeedbackSlot builds its own client outside budgetedFetch). These are candidates for the api-budget logging follow-up.
  • POLL_SECONDS and POLL_LIMIT_SECONDS are inline in the workflow, not in docs/conventions/pr-pipeline.yaml. They are unchanged here.
  • Leads for the unattributed calls: no notice bounds the two largest gaps in used, +657 from 19:19 to 19:24Z and +574 from 19:26 to 19:29Z. The notices around them are pr-unblock-dispatch's dispatch job (self-reported 54, 25 and 6) and pr-merge's enable-auto-merge (cost=442 at 19:26:25Z). pr-merge's merge job reported cost=1332 at 19:35:05Z.

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits October 24, 2026 16:12
…otes API readings

Both jobs spent github-token calls without posting an api-budget notice,
so GitHub Support's 18:57-19:57Z window counted their runs (87 of the
workflow's 91 executing runs) among the unattributed spend.

- rerun-ci-status: the sparse checkout adds the log-api-budget library;
  read_api keeps each response's reading (refusals included) through
  api_budget_read, and the job ends with the log-api-budget step. A failed
  checkout leaves no library and the job runs as before.
- post-verify-notes: the artifact listing's gh api call runs through
  api_budget_wrap_gh, and the job ends with the log-api-budget step.

Call counts per path are unchanged; scripts/ci-rerun-ci-status.test.sh now
pins one reading per call on every counted path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 25, 2026 01:27
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T01:29:00.495754Z b36589f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 6f8c834b369ed262242ac7ef868283f3cab2f617: 0 finding(s), 0 as inline comments.

No security issues found in this pull request.

critical-findings: 0

The change adds api-budget logging to two jobs in pr-feedback-dispatch.yml, post-verify-notes and rerun-ci-status, plus tests. None of the three changed files matches a trusted path, and no plugin manifest changed, so there are no manifest <field> lines.

  • Code that runs: the new code comes from .github/actions/log-api-budget/ (api-budget.bash and the composite action.yml). Both jobs check it out from refs/heads/main into .base, with persist-credentials: false. Nothing from the PR head is sourced or run in these workflow_run jobs, so there is no untrusted-code path.
  • rerun-ci-status checkout: the sparse checkout now includes /.github/actions/log-api-budget/. The local action's action.yml and api-budget.bash are both in that directory, and the action only sources $GITHUB_ACTION_PATH/api-budget.bash.
  • Permissions: no permissions: block changed. The log step uses no extra token, and tokens: github-token is a fixed kebab-case name that the action checks against a regex.
  • Failure handling: post-verify-notes sources the library inside { ...; } 2>/dev/null || true. rerun-ci-status does the same with || true, and its budget() helper does nothing when the library is missing. A failed checkout therefore cannot change the re-run decision logic or block the re-run, and the new test pins this.
  • Data handling: the budget "$out" helper passes gh api -i output to api_budget_read on stdin via a here-string, with no eval or word-splitting on the content. $1 is quoted, and /dev/stdin is a fixed path.
  • Embedded instructions: the PR description contains no instructions aimed at the reviewer.
  • Spend: nothing here can start a real charge. The change adds no API calls, and it logs about 87 extra notices an hour.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 6f8c834b369ed262242ac7ef868283f3cab2f617: 0 finding(s), 0 as inline comments.

Code review: no findings

I read the full diff (.claude-lane/pr.diff), the PR context, api-budget.bash, log-api-budget/action.yml, and the changed jobs in pr-feedback-dispatch.yml. I ran no tests myself. The test counts in the PR body are the author's claims and I did not check them.

What I checked

  • post-verify-notes. The download step sources the library and calls api_budget_wrap_gh github-token. The call is guarded with 2>/dev/null || true, so a missing library leaves plain gh in place. The wrapped gh api passes -i and strips the header block, so the --jq listing keeps its stdout and exit status. gh run download takes the non-api pass-through. The job-end log step uses if: always(), continue-on-error and timeout-minutes: 1, matching the dispatch job.
  • rerun-ci-status.
    • Library: the sparse checkout now includes /.github/actions/log-api-budget/, which holds action.yml and api-budget.bash.
    • budget() is a no-op when the library is missing, and api_budget_read always returns 0. It therefore cannot trip bash -e in the success or refusal paths of read_api.
    • Output: the reading goes to /dev/null, so the parsed response is unchanged.
    • Readings: each read_api response, refusals included, is counted. The re-run POST is not counted, and the PR says so.
  • Tests.
    • jobText ends at the next two-space job key, so comment lines between jobs do not break it. For the last job it runs to the end of the file.
    • The new ci-rerun-ci-status.test.sh cases pin one reading per API call and the missing-library path.
    • The post-verify-notes body test runs the real step body against a stub gh.
  • Trusted paths. No changed file matches the trusted-path globs.
  • Plugin manifests. The diff touches no manifest, so there are no manifest lines.
  • Closing issues. The PR closes none.

Observations, not findings

  • The PR says it changes no call. In post-verify-notes, the wrapped gh also applies the rate guard. If the core pool were already stopped, a later gh api call in that step would be refused where it was not before. The step makes one gh api call, so this is not reachable in practice.
  • The PR is upfront that the post-verify-notes notice covers only the artifact listing. gh run download and verify-notes.mjs calls stay unrecorded.

Spend: the new log steps add at most one notice per run and no extra API calls. They use no billed resources, so there is no hold.

@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 25, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

The pr-unblock lane stopped on this pull request and needs a person: the merge stages a change under .github/workflows/, which the lane does not push.

  • Reason: workflow-staged
  • Resume key: b36589f882e0d782b437cd0dc47792c3b5a08881 (the head the lane stopped on)

To resume, deal with the reason above, remove the needs-human label (and do-not-merge, if set), then dispatch the pr-unblock workflow with pr-number=9882 and activity=resolve-conflicts:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=9882 -f activity=resolve-conflicts

kyle-sexton and others added 2 commits October 25, 2026 02:59
…-call-budget

Compose log-api-budget.test.mjs's fs import: main's mkdirSync and this
branch's copyFileSync and mkdirSync. Narrow the post-verify-notes test
comment: its notice covers only the artifact listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the artifact listing leaves one. gh run download passes through the
gh wrapper but is not gh api, and verify-notes.mjs uses its own fetch
client, so the job-end notice understates the job's spend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 25, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 25, 2026
Merged via the queue into main with commit 4057afd Oct 25, 2026
68 of 70 checks passed
@kyle-sexton
kyle-sexton deleted the fix/feedback-dispatch-call-budget branch October 25, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant