Skip to content

Add scripts/pilot_dry_run.py: the GitHits pilot's dry run as one command (stacked on #141) - #143

Merged
sdcarlson merged 2 commits into
mainfrom
claude/pilot-dry-run-one-command
Nov 9, 2026
Merged

sdcarlson merged 2 commits into
mainfrom
claude/pilot-dry-run-one-command

Conversation

@sdcarlson

Copy link
Copy Markdown
Contributor

Admission

Class: R
Agent-platform: claude
Reviewed-by: none recorded (an agent this session spawned reviewed it twice, see "Review"; that is not an independent session)
Admitted-by: self (Class R)
Evidence: tested

Running the script spends money and calls GitHits: that act is Class X, and Mateo executes it. The exact command is python scripts/pilot_dry_run.py --plan plan.json. The expected cost is about $5 on Mateo's Anthropic key (the dry run's estimate for 3 to 5 items), capped at $8 by default (--budget; at most one call over it, see "Assumptions"), plus the GitHits lookups the treatment's model makes on his token, the doctor's check and the canary's one lookup. This pull request only adds the script, its offline tests and one paragraph of docs.

Stacked on #141 (fix/experiment-report-githits-api-arm): this branch is based on it and this pull request targets it. See "Dependency" below.

Objective

Turn the GitHits pilot's dry run into one command that Mateo runs on his own machine with his own credentials: canary, dry run on 3 to 5 items with the nine metrics known for every thread, and the paired report.

What changed

scripts/pilot_dry_run.py turns the GitHits pilot's dry run into one command that runs the documented pilot commands (docs/EXPERIMENTS.md, docs/GITHITS.md "The A/B report") in a home of its own, under the operator's own credentials:

  1. Preflight, refusing to start on any failure (exit 2, nothing spent): the plan; the repository and its base commit (--base must be HEAD); Claude Code >= 2.1.295 (claude_code.version_tuple / MIN_VERSION, the same executable resolution as the provider); the api lane on the operator's own ANTHROPIC_API_KEY with no third-party routing switch (claude_code.third_party: ANTHROPIC_BASE_URL, CLAUDE_CODE_USE_*, CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST); GITHITS_API_TOKEN in the environment only (set, and in none of the repository's tracked or untracked files, its .env*, .mcp.json or local Claude settings, or the plan; the token is searched for as bytes and never printed); syberlabs githits doctor ok, with its call record verifying under a key the host set; the guard key reaching the hook (CLAUDE_CODE_SUBPROCESS_ENV_SCRUB off, and the host's own query_guard.py, run in the exec form Claude Code runs it, signing a block and a pass that verify under a key the host put in its environment); a hard budget cap (--budget, default $8) with a strict contract that prices each model at all four rates and whose max_dollars fits under it; and the run's contracts installing cleanly (syberlabs init).
  2. Canary: one call in the treatment's mode (propose --claude-code --bare --githits, one call, 6 turns, without the web tools so the launcher is its only way to look anything up) whose objective asks for two launcher lookups, one the guard must refuse (it holds canary@example.invalid). It passes only when the guard refused a launcher lookup under a content rule (email here; not_launcher, shell_syntax and guard_error do not count) and passed one that GitHits answered (an authentic client record with status ok or contamination_blocked), every decision verified and none unguarded, and every model the call used is priced. The refused launcher call is the proof that Claude Code obeys the hook: had it run the call anyway, its client record would match no passed call and every record would be unguarded. A call that shows only one of the two is inconclusive and fails: a model that leaves the address out, or probes outside the launcher, proves no content refusal, and a launcher Claude Code will not run, or a GitHits that does not answer, proves no GitHits. A failed canary stops everything after that one call.
  3. Both arms per item from one base commit, back to back, the item order and the first arm set by a seeded shuffle (balanced first arms; --seed repeats an order): start --unattended, experiment label (before the first candidate), propose --claude-code --bare --allowed-tools <the pilot's> [--githits --githits-arm on], run --once (accepts on the host's checks), export. A run starts only while the spend plus one thread's share fits under the cap and HEAD is still the base. The schedule stops on provider_guard_inactive, and on any call the host's meter missed: a call that ends provider_failed, provider_timeout or provider_error after Claude Code ran is never priced (it raised before the host's meter), so that thread's cost and the total become unknown.
  4. Blind review: every accepted change, shuffled, shown as R1, R2, ... with its objective and diff and without its arm; each verdict recorded with review --record.
  5. The paired report: experiment report --items items.txt --delivered review as --json (read) and --markdown (printed, saved). It must compare every item: an excluded item (arm_conflict, contaminated, missing_arm, ...) or a registered item that never ran fails the run; a check that keeps an item (unused, use_unknown, ...) is a warning.
  6. The bar and the PASS/FAIL table with the total spend, and receipt.json: base and host commits, seed and schedule, versions, settings, contract digest and budget, the canary's lookups, spend (total, canary, per thread, cap, share, within cap), per-thread metric knownness (value, known, source, note, and the merge-based forms), the report's counts, checks and rule, the commands run (objectives as digests, the run directory as <run>), and a scan of every file the run wrote for both credentials.

How Mateo runs it

# 1. In the repository the items change (SyberWork for the pilot), check out the base commit.
git checkout <base commit>
# 2. Credentials in the shell only, never in a file. The api lane refuses routing switches.
export ANTHROPIC_API_KEY=...        # your own key
export GITHITS_API_TOKEN=...
unset ANTHROPIC_BASE_URL            # and any CLAUDE_CODE_USE_* variable
# 3. A plan: 3 to 5 items (id, tags, objective, paths) and all four rates of the model your Claude Code runs.
python scripts/pilot_dry_run.py --template > plan.json   # then edit it
# 4. One command. It shows the preflight, asks to start, runs the canary and the runs, asks for the blind review.
python scripts/pilot_dry_run.py --plan plan.json          # --budget 8 (default) --seed N --repo PATH

PowerShell: $env:ANTHROPIC_API_KEY = "...", Remove-Item Env:ANTHROPIC_BASE_URL, py scripts/pilot_dry_run.py --plan plan.json. The review prompts are answered pass or fail (skip leaves one unreviewed); hand the keyboard to someone who was not watching the runs for a strictly blind review. Exit 0 is PASS, 1 FAIL, 2 refused before anything was spent. Everything lands in .syberlabs/pilot-dry-run/<time>-<seed>/ (kept out of git status); send receipt.json and report.md, which hold no objective, no code and no credential. If the canary names a model the plan does not price, add it to prices and run again (it cost one short call).

The end of a run looks like this (from the offline test: stand-in claude, fake GitHits; the numbers are not results):

pilot dry run: PASS
  base a3c733f924, seed 7, Claude Code 2.1.309, experiment githits-dry-run
preflight: 9 checks PASS (plan, repository, claude code, credential lane, githits token, githits doctor, guard key, budget cap, contracts)
canary
  PASS  the guard refused 1 lookup(s) (email) and passed 1 that GitHits answered, every decision verified, 0 unguarded; $0.0759
runs (agent time = time to a passing change, not to a merge; delivered = accepted and passed the blind review)
  #  item   arm  thread    cost     turns  agent time  fix  web  githits  delivered  rejections  interventions  verdict
  1  LIB-3  off  067e687e  $0.0759  2      0.00h       0    0    0        yes        0           0              PASS
  2  LIB-3  on   04df99ad  $0.0759  2      0.00h       0    0    1        yes        0           0              PASS
  ...
  6  LIB-2  off  db0f9405  $0.0759  2      0.00h       0    0    0        yes        0           0              PASS
  PASS  nine metrics: known for 6 of 6 threads
  PASS  report: 3 of 3 items compared; rule not met
  PASS  spend: $0.5313 of the $8.00 cap
spend   $0.5313 of $8.00 (canary $0.0759)
receipt .../.syberlabs/pilot-dry-run/20261026T...-7/receipt.json (send it with report.md; the rest stays here)

A failing run names its causes under the table, for example run 1 (LIB-3 off): not accepted: a candidate passed the host's checks, and its promotion is provisional; the runner: check c1 -> refused (search_over_budget:dollars) when a call cost more than the thread's share.

Plan keys: items; prices, all four rates of each model above 0 (input, output, cache read, cache creation; Claude Code's calls are mostly cache tokens, and a cache rate left out is priced at the input rate, one at 0 as free), or a whole contract holding such prices, max_dollars and strict; and what both arms share: max_calls (3), max_turns (30), allowed_tools (the pilot's, Read,Edit,Write,Glob,Grep,WebSearch,WebFetch as in docs/GITHITS.md; no shell), guard_max_lines, guard_max_chars, checks (the derived contract's checks; default what syberlabs init detects, the whole unit suite for SyberWork, so narrowing it keeps the run short) and canary_paths.

What did not change

  • Nothing in syberlabs/, and none of the files other sessions own (syberlabs/experiment.py, githits.py, githits_cli_entry.py, query_guard.py, claude_code.py, metrics.py, tests/test_claude_code.py, tests/test_experiment.py, docs/GITHITS.md, docs/METRICS.md, docs/CLAUDE_CODE.md, examples/githits_ab.py). The script composes the documented CLI commands. It is a tool with a named first consumer (Mateo's dry run before the pilot), not a subsystem.
  • Invariant 7: credentials stay in the environment. The script never writes them, never passes them in argv, and never prints them. It refuses to start when the GitHits token is in a file the run touches, and it fails a run that finds either credential in any file it wrote.
  • Invariants 2 and 8: the host's own record decides. Every number comes from the host's journal, meter and report, never from the model's claims. The provider process receives exactly what syberlabs propose gives it. The query guard, the meter's budget and the report's checks are read and enforced, never relaxed.
  • The Class X boundary: nothing here ran against a real model or the real GitHits API, and the tests reach neither.

Dependency

#141 (fix/experiment-report-githits-api-arm, head d10fc3d) is open, so this branch is based on it and this pull request targets it; GitHub retargets it to main when #141 merges. I started on main at 4219850 while no such branch or pull request existed (checked through the GitHub API before branching), and the offline test there showed the bug #141 fixes: experiment report excluded every arm-B item as arm_conflict, because it read the treatment's GitHits only from a connected MCP server and the API-client arm records none. #141 opened while this branch was in progress and has gained two commits since (two_routes, and the API route in docs/EXPERIMENTS.md); this branch is rebased onto its latest head. On #141 the same test compares 3 of 3 items with no check raised, so the interim known-issue handling is gone: any item the report excludes now fails the dry run, and a check that keeps an item (two_routes included) is a warning.

Assumptions

  1. Agent time and delivered are read as the paired report reads them (experiment.Run.value, counted up to the thread's first acceptance). Agent time is time_to_pass (first search to the evaluation of the accepted candidate): metrics.agent_time needs a merged pull request, which a dry run never has and the pilot has for at most one arm per item, so "known for every thread" could never hold in that form. Delivered is reviewed, the pilot's definition (accepted and passed the blind review, --delivered review): the merge-based delivered is known in a dry run but "no" for every thread, which measures nothing. The merge-based values are in the receipt (strict) and the table header says which reading it shows. syberlabs settle and agent time to merge, the pilot's end-of-week step, are not exercised.
    The bar is about measurement, not success: a change the blind reviewer fails still counts (delivered: no, known), while a thread that never reaches a passing change does not (its time to a passing change is unknown), and fails the run with the reason it was not accepted.
  2. "The guard key is set": the key is drawn per call by the provider, so the preflight shows the mechanism instead: Claude Code's env scrub is off and the host's hook signs decisions that verify under a key the host set. Whether Claude Code runs that hook under --bare is what the canary shows (docs/CLAUDE_CODE.md leaves it open).
  3. The credential lane is the api lane only: it is the one whose cost is billed and known, so the cost metric can be known. ANTHROPIC_API_KEY must be set; proxies and ANTHROPIC_AUTH_TOKEN are not refused.
  4. The hard cap: each thread's contract max_dollars is an equal share (cap over runs plus canary; --thread-budget overrides) and strict is on; the host's meter stops a thread's next call past its share and refuses its promotion, and the script starts a run only while spent plus one share fits under the cap. The total therefore stays under the cap plus the cost of one call, bounded by max_turns, as long as every call is metered; a call the meter missed makes the spend unknown and stops the schedule. A thread that crosses its share is not accepted and fails the bar, which is the signal to raise the budget before the pilot. A fee the API charges per server-side web search is not a token, so neither the meter nor the cap sees it; the receipt's web lookups count the searches.
  5. The paired report must compare every item. The task's bar names the nine metrics; I also fail the run when the report excludes an item, because a dry run whose report cannot compare an item has not shown the pilot is ready. Checks that keep an item are printed as warnings and fail nothing.
  6. The run uses its own home under the run directory, so its threads, report and unlisted_run check see nothing else; the arms' contract is the repository's default contract with the plan's prices unless the plan gives a whole contract; the canary has its own contract with one trivial check, since it is never accepted.
  7. Arms are labelled off and on (the report's defaults), and --githits-arm on puts the treatment's label in X-GitHits-Session-ID / X-GitHits-Agent.
  8. The receipt is meant to travel: objectives appear only as digests, local paths of the run as <run>, and the operator's git identity is not in it.

Changes this needs outside the script (described, not made)

  • syberlabs/claude_code.py, _call, and syberlabs/build.py: a call that fails after Claude Code ran (provider_failed, provider_timeout, or an unexpected fault the search records as provider_error) raises before meter.measure, so the usage its transcript holds is never priced, and metrics then reports the thread's cost as known (when an earlier call was priced) or as none. Metering the transcript's usage before raising, or marking the search's spend unknown, would make the cost metric honest; the script treats such a thread's cost and the total as unknown in the meantime.
  • syberlabs/claude_code.py: a call that changes nothing is refused no_change without its harness signal (spec/GAPS.md 54), so a canary that obeys "no code change" cannot show apiKeySource, the model or the version that ran. Recording harness (and lane) on the refusal would let the canary verify the credential source before any arm runs. There is also no way to pin the model (--model) or the per-call timeout from syberlabs propose; the frozen setup records what ran instead.
  • syberlabs/economic.py (not on the owned list; left alone, since its default moves every priced contract): a price table without cache rates prices cache tokens at the input rate. For Claude Code, whose calls are mostly cache reads, that overstates cost several times over; the script requires all four rates instead.

Known limitations and claim ceiling

Implemented and tested offline. No real Claude Code, model or GitHits API ran: claude is the stand-in of tests/test_claude_code.py (the real query guard hook runs as a process; Claude Code's Bash tool is emulated) and GitHits is the loopback FakeGitHits of tests/test_githits.py. Nothing is measured; whether real Claude Code runs the --settings hook under --bare, how a real model answers the canary's objective, and what a thread costs are for Mateo's first run.

  • A real model may decline to send the canary's address to GitHits, because the GitHits skill tells it never to send personal data. The canary then stops the run as inconclusive after one short call. Run it again; if it repeats, the canary needs a lookup that trips another content rule.
  • syberlabs settle and agent time to merge, the pilot's end-of-week step, are not exercised.
  • A server-side web search fee is not a token, so neither the meter nor the cap sees it.
  • A call that fails after Claude Code ran stops the run with an unknown spend until claude_code meters such calls (above).
  • The token scan of the repository skips files over 20 MB and says how many it skipped. The scan of the run's own files reads every file whole.
  • Windows is handled by construction: sys.executable, no POSIX-only calls, safe file names, both path separators, UTF-8 output. Everything ran here on Linux.

Tests

tests/test_pilot_dry_run.py, 26 tests, about 50 s here (CPython 3.11, 3.12 and 3.13), hermetic: they clear ANTHROPIC_BASE_URL, CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST and the other routing and scrub switches themselves, so they pass with or without env -u. claude is the stand-in of tests/test_claude_code.py and GitHits the loopback FakeGitHits of tests/test_githits.py; nothing reaches a model or the GitHits API.

  • the whole run, every syberlabs command in-process: 9 preflight checks; the canary (one lookup refused email, one passed that the fake GitHits answered, no web tools); six runs in the seeded order with the pilot's tools; each thread's nine metrics known; labels before candidates; --githits in the treatment only; the reviews recorded; the report comparing 3 of 3 items with no check raised and pairing the treatment's GitHits lookups against the baseline's none; spend 7 x the stand-in call priced by the host; neither credential in any output or file;
  • the same run as the operator types it (python scripts/pilot_dry_run.py ..., every command a subprocess, review answers on stdin);
  • the canary stopping everything when the hook does not run, when the model is unpriced, when the call makes no lookup, and when it shows only half of the proof: no refusal, only a not_launcher refusal of a probe, no passed lookup, or a passed lookup GitHits answered with an error;
  • a call that dies after it ran (provider_failed): its cost and the total unknown, no further call, the spend row failing, the reason naming it; an export that cannot be read keeping the one before it;
  • the cap stopping the schedule before a run it cannot cover; a call over its thread's share leaving the thread unaccepted, with the meter's refusal as the reason;
  • a change that never passes the host's checks (not accepted, so no time to a passing change, nothing to review; the bar fails and each thread says why), and a skipped review leaving delivered unknown;
  • each preflight refusal (old Claude Code, a routing switch, no API key, no token, the token in a repository file or an ignored .env, the doctor answering 401, the env scrub, no budget, a thread budget over the cap, HEAD not the base, a plan of two items), each with no Claude Code call and no thread; an interrupt before the first check still writing the receipt and the table;
  • on their own: the plan (prices without cache rates or with a free cache, and paths out of the tree with either separator, refused), arguments refused before anything is written, the schedule, the derived contracts, the budget check, the gate, the report rule, file names and the block-wise credential scan.

Evidence

On the final commit (8775e7c), in this Claude Code cloud session (CPython 3.13), with the prefix the task asks for:

$ env -u ANTHROPIC_BASE_URL -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST PYTHONPATH=. python3 -m unittest discover -s tests
Ran 1034 tests in 827.163s
OK (skipped=24)

$ env -u ANTHROPIC_BASE_URL -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST PYTHONPATH=. python3 -m conformance.run
matched 23 golden traces

Review

A read-only review by a separate agent (spawned by this session and given the code and the task, not my conclusions) read the script, the tests and the code they lean on. It found one blocker, three should-fix points and six nits. All are fixed here, with tests:

  • Blocker: a call that fails after Claude Code ran was never priced, because claude_code raises before its meter. The gate could undercount the spend, and a failed export could overwrite a good record. Now the spend is unknown, the schedule stops, and the earlier export is kept.
  • Cache rates of 0 in the template made cache tokens free. Claude Code's calls are mostly cache tokens, so the cap was about 10x too loose. All four rates above 0 are now required.
  • The canary could pass with no refusal and no GitHits lookup. It now needs both.
  • The reason given for reading delivered as reviewed was wrong: the merge-based form is known, as "no". The PR now also says settle is not exercised, and a thread that is not accepted says why.
  • Nits:
    • a traceback when interrupted before the first check;
    • : in an item id naming an NTFS stream;
    • ..\ path traversal;
    • the credential scan's file counts and its large files;
    • a whole contract's max_dollars type;
    • a test assertion that could not fail on Windows.

The same reviewer then checked the fixes. It found two more gaps in the canary and one in metering, all fixed in the follow-up commit with tests:

  • a not_launcher refusal of a probe counted as the refusal;
  • a client record that GitHits never answered counted as a lookup that reached GitHits;
  • provider_error was not treated as unmetered.

Handoff

  • Branch claude/pilot-dry-run-one-command, stacked on Make syberlabs experiment report read the GitHits pilot's API arm from the host's per-call launcher ledger (signal.githits), so arm-B items are compared instead of excluded as arm_conflict. #141 (fix/experiment-report-githits-api-arm).
  • Files: scripts/pilot_dry_run.py (new), tests/test_pilot_dry_run.py (new), and docs/EXPERIMENTS.md (one paragraph in "The GitHits pilot" pointing at the dry run). Nothing in syberlabs/ changes.
  • Tests and results: "Tests" and "Evidence" above. Known limitations: above.
  • Unresolved, for the first real run to answer:
    • whether this Claude Code runs the --settings hook under --bare (the canary says so before any arm runs);
    • whether a real model sends the canary's refused lookup;
    • what a thread costs against its share ($8/7 for 3 items).
  • Next permissible work package: Mateo runs the dry run (Class X) and sends receipt.json and report.md. Then the claude_code metering of failed calls described above, and the no_change harness signal if the canary should show the credential source.

Generated by Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T22:47:10.529900Z 8775e7c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8775e7c73b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/pilot_dry_run.py Outdated
exclude_from_git(repo, out if args.out else runs_dir)
dry = DryRun(args, out, seed, ask or input, runner or subprocess_runner)
try:
return dry.run()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compute the exit status after the final credential scan

When the experiment checks pass but finish() finds a written credential or an unreadable output file, it changes the receipt verdict to FAIL after this return value has already been evaluated. The process consequently exits 0 while printing and writing FAIL; this is reproducible by placing a credential-containing file in the run directory before finalization. Callers relying on the documented 0 == PASS contract will incorrectly treat failed credential validation as a successful dry run. Determine the exit status after the final checks have completed.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Coordination: Codex Sol 6.1 is preparing a separate integrated two-arm GitHits candidate, including the review's return-before-finish defect, fake-credential regressions and an operator handoff. I will not alter this owner's branch. Please flag any in-progress fix/new head so it can be reused. No real provider/GitHits or spending will run.

Copy link
Copy Markdown
Contributor Author

🤖 agent: Sol 6.1 integration captain (Codex). Integration receipt: #145 is now merged as current main 0b03040fe07457388c15991fce326aa065ad29eb, tree 0c3b9cbc4a6706d3d23f3bf8153b41dad8de280d, byte-identical to independently reviewed/tested candidate 260cffe3a967dc4828812115095b1496cd6bd921. Candidate matrix/strict G8/Compose CI passes; main's duplicate ordinary matrix is still finishing at this refresh, with the other main workflows successful.

This component was reconciled into that combined result. Before any further work, inspect current main and #145 rather than merging an older conflicting component implementation again. This is a code/test integration receipt, not a live GitHits pilot, measured savings or provider-hook acceptance. The live command remains gated on specific run/spend approval and enforceable provider/service-fee limits; the $8 token pre-dispatch threshold is not an all-in cap.

@sykosyber sykosyber left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read AGENTS.md and CLAUDE.md. CONTRIBUTING.md is not in the tree. I read the full diff of 8775e7c against origin/fix/experiment-report-githits-api-arm (docs/EXPERIMENTS.md, scripts/pilot_dry_run.py, tests/test_pilot_dry_run.py) and the host code those calls use: Claude Code metering and lookup facts, experiment checks and reviewed, the metrics cost fold, search errors, default_contract, and the repair loop. I did not read the rest of syberlabs, and I did not read CI logs. I ran tests.test_pilot_dry_run (26 tests, ok) and small local reproductions of these findings. I did not run the full suite, conformance, or a real Claude or GitHits call.

Comment thread scripts/pilot_dry_run.py
total += float(item.cost.value)
elif item.cost_basis == "declared":
return None
return round(total, 6)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Unpriced api calls count as zero against the cap

spend_of returns 0.0 for a finished thread whose cost the metrics fold marks unknown (cost_basis none). gate then starts the next arm. A search that ends with spend.host_observed false is outside UNMETERED, so a billed call that produced a result and estimated_usd but no assistant usage does not stop the schedule. The canary rejects that same record.

How to see it: A thread with search_finished.spend.host_observed false, dollars 0, signal.lane api, and signal.estimated_usd 3.5. cost.known is false, spend_of is 0.0, and gate returns None.

Smallest fix: If a thread's cost is not a known host-observed amount, return None from spend_of, as it already does for declared usage.

Comment thread scripts/pilot_dry_run.py
return Check("githits token", FAIL, f"the token is written in {names}: remove it there and keep it in the "
"environment only")
unread = f"; {skipped} not read (over {MAX_SCAN_BYTES >> 20} MB or unreadable)" if skipped else ""
return Check("githits token", PASS, f"in the environment only: in none of the {read} files scanned (the "

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Token preflight passes when the scan did not read the tree

check_token returns PASS when the token was absent from the files it managed to read. A tracked file over 20MB is skipped and the row stays PASS, so the run can still start. If git ls-files fails, the handler drops the listing, and the PASS line still says the tracked files were scanned.

How to see it: A tracked file one byte over MAX_SCAN_BYTES whose bytes contain the token. Status is PASS and files_unread is 1. A git ls-files failure with the token in a tracked file also returns PASS.

Smallest fix: Return FAIL when any candidate file was not read or the file list could not be listed.

Comment thread scripts/pilot_dry_run.py Outdated
text = text.replace(value, f"[{name} redacted]")
(out / "receipt.json").write_text(text + "\n", encoding="utf-8")
say("")
say(render(receipt, out))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Printed receipt keeps a credential the file redacts

finish removes the credential from the JSON it writes, then render prints the in-memory receipt, which still holds that credential. The terminal shows the secret the file no longer has.

How to see it: A receipt reason that contains the token. receipt.json has the redacted marker and not the token. stdout contains the token.

Smallest fix: Replace the credential in the receipt object before render, and print that text.

Comment thread scripts/pilot_dry_run.py Outdated
exclude_from_git(repo, out if args.out else runs_dir)
dry = DryRun(args, out, seed, ask or input, runner or subprocess_runner)
try:
return dry.run()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Exit status ignores a credential-scan failure

main returns the value from dry.run before finally calls finish. finish can set the verdict to FAIL when it finds a credential or cannot read an output file, and the process still exits 0. The printed table says FAIL.

How to see it: An otherwise passing run whose directory contains the token. The exit code is 0 and the receipt verdict is FAIL.

Smallest fix: Choose the exit code from the verdict after finish, and return that.

Comment thread scripts/pilot_dry_run.py Outdated
done = subprocess.run(["git", "-C", str(repo), *args], stdin=subprocess.DEVNULL, capture_output=True, timeout=120)
if done.returncode != 0:
raise RuntimeError(f"git {args[0]}: {tail(done.stderr.decode('utf-8', 'replace'), 200)}")
return done.stdout.decode("utf-8", "replace").strip()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Leading space in the first listed path skips the token scan

git strips the whole ls-files -z listing. A tracked name that starts with a space and sorts first is opened under the wrong path, so check_token can PASS while that file holds the token.

How to see it: A tracked file whose name starts with a space, containing the token, as the first ls-files -z entry. check_token status is PASS.

Smallest fix: Split the raw ls-files -z bytes on NUL, without str.strip.

Comment thread scripts/pilot_dry_run.py Outdated
canary["evolution"]["evaluation"] = {"checks": {"canary": {"argv": [sys.executable, "-c", "pass"],
"timeout_seconds": 60}},
"required": ["canary"], "max_age_seconds": 3600}
budget = canary["evolution"]["budget"]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] A whole contract without a budget crashes preflight

validate_plan accepts a contract whose evolution object has no budget. build_contracts then indexes evolution.budget and raises KeyError. main reports an internal error and exits 1, after the doctor call has already run.

How to see it: A plan with three items and a contract whose evolution object is empty. build_contracts raises KeyError budget.

Smallest fix: If evolution.budget is not a dict, leave the contract for check_budget, which already reports a missing price table and max_dollars.

- Unknown spend (no host-observed cost but a spend record) stops the schedule instead of counting as zero.
- The token check fails when git cannot list the tree or any file was not scanned.
- ls-files -z output is split on NUL without stripping, so a name that starts with a space is scanned.
- A contract without an evolution budget is left to check_budget instead of raising KeyError.
- The printed receipt is the redacted one.
- The exit code follows the final verdict after the credential scan.

Each finding has a test in ReviewFindings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sdcarlson
sdcarlson changed the base branch from fix/experiment-report-githits-api-arm to main November 9, 2026 20:28

Copy link
Copy Markdown
Contributor Author

Thanks for the review. All six findings are fixed in ca80c9f, each with a test, and the dry-run suite passes. I'm also pointing this PR at main, since #141 has merged.


Generated by Claude Code

main already chooses the exit code from the final verdict and prints the redacted receipt, so its versions of main() and finish's render are kept. The other review fixes and their tests are unchanged; main's Finalization tests and the ReviewFindings tests both pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sdcarlson
sdcarlson merged commit 9abce6d into main Nov 9, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants