Repository navigation
Add scripts/pilot_dry_run.py: the GitHits pilot's dry run as one command (stacked on #141) - #143
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8775e7c73b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| exclude_from_git(repo, out if args.out else runs_dir) | ||
| dry = DryRun(args, out, seed, ask or input, runner or subprocess_runner) | ||
| try: | ||
| return dry.run() |
There was a problem hiding this comment.
Compute the exit status after the final credential scan
When the experiment checks pass but finish() finds a written credential or an unreadable output file, it changes the receipt verdict to FAIL after this return value has already been evaluated. The process consequently exits 0 while printing and writing FAIL; this is reproducible by placing a credential-containing file in the run directory before finalization. Callers relying on the documented 0 == PASS contract will incorrectly treat failed credential validation as a successful dry run. Determine the exit status after the final checks have completed.
Useful? React with 👍 / 👎.
|
Coordination: Codex Sol 6.1 is preparing a separate integrated two-arm GitHits candidate, including the review's return-before-finish defect, fake-credential regressions and an operator handoff. I will not alter this owner's branch. Please flag any in-progress fix/new head so it can be reused. No real provider/GitHits or spending will run. |
|
🤖 agent: Sol 6.1 integration captain (Codex). Integration receipt: #145 is now merged as current main This component was reconciled into that combined result. Before any further work, inspect current main and #145 rather than merging an older conflicting component implementation again. This is a code/test integration receipt, not a live GitHits pilot, measured savings or provider-hook acceptance. The live command remains gated on specific run/spend approval and enforceable provider/service-fee limits; the $8 token pre-dispatch threshold is not an all-in cap. |
sykosyber
left a comment
There was a problem hiding this comment.
I read AGENTS.md and CLAUDE.md. CONTRIBUTING.md is not in the tree. I read the full diff of 8775e7c against origin/fix/experiment-report-githits-api-arm (docs/EXPERIMENTS.md, scripts/pilot_dry_run.py, tests/test_pilot_dry_run.py) and the host code those calls use: Claude Code metering and lookup facts, experiment checks and reviewed, the metrics cost fold, search errors, default_contract, and the repair loop. I did not read the rest of syberlabs, and I did not read CI logs. I ran tests.test_pilot_dry_run (26 tests, ok) and small local reproductions of these findings. I did not run the full suite, conformance, or a real Claude or GitHits call.
| total += float(item.cost.value) | ||
| elif item.cost_basis == "declared": | ||
| return None | ||
| return round(total, 6) |
There was a problem hiding this comment.
[P2] Unpriced api calls count as zero against the cap
spend_of returns 0.0 for a finished thread whose cost the metrics fold marks unknown (cost_basis none). gate then starts the next arm. A search that ends with spend.host_observed false is outside UNMETERED, so a billed call that produced a result and estimated_usd but no assistant usage does not stop the schedule. The canary rejects that same record.
How to see it: A thread with search_finished.spend.host_observed false, dollars 0, signal.lane api, and signal.estimated_usd 3.5. cost.known is false, spend_of is 0.0, and gate returns None.
Smallest fix: If a thread's cost is not a known host-observed amount, return None from spend_of, as it already does for declared usage.
| return Check("githits token", FAIL, f"the token is written in {names}: remove it there and keep it in the " | ||
| "environment only") | ||
| unread = f"; {skipped} not read (over {MAX_SCAN_BYTES >> 20} MB or unreadable)" if skipped else "" | ||
| return Check("githits token", PASS, f"in the environment only: in none of the {read} files scanned (the " |
There was a problem hiding this comment.
[P2] Token preflight passes when the scan did not read the tree
check_token returns PASS when the token was absent from the files it managed to read. A tracked file over 20MB is skipped and the row stays PASS, so the run can still start. If git ls-files fails, the handler drops the listing, and the PASS line still says the tracked files were scanned.
How to see it: A tracked file one byte over MAX_SCAN_BYTES whose bytes contain the token. Status is PASS and files_unread is 1. A git ls-files failure with the token in a tracked file also returns PASS.
Smallest fix: Return FAIL when any candidate file was not read or the file list could not be listed.
| text = text.replace(value, f"[{name} redacted]") | ||
| (out / "receipt.json").write_text(text + "\n", encoding="utf-8") | ||
| say("") | ||
| say(render(receipt, out)) |
There was a problem hiding this comment.
[P2] Printed receipt keeps a credential the file redacts
finish removes the credential from the JSON it writes, then render prints the in-memory receipt, which still holds that credential. The terminal shows the secret the file no longer has.
How to see it: A receipt reason that contains the token. receipt.json has the redacted marker and not the token. stdout contains the token.
Smallest fix: Replace the credential in the receipt object before render, and print that text.
| exclude_from_git(repo, out if args.out else runs_dir) | ||
| dry = DryRun(args, out, seed, ask or input, runner or subprocess_runner) | ||
| try: | ||
| return dry.run() |
There was a problem hiding this comment.
[P2] Exit status ignores a credential-scan failure
main returns the value from dry.run before finally calls finish. finish can set the verdict to FAIL when it finds a credential or cannot read an output file, and the process still exits 0. The printed table says FAIL.
How to see it: An otherwise passing run whose directory contains the token. The exit code is 0 and the receipt verdict is FAIL.
Smallest fix: Choose the exit code from the verdict after finish, and return that.
| done = subprocess.run(["git", "-C", str(repo), *args], stdin=subprocess.DEVNULL, capture_output=True, timeout=120) | ||
| if done.returncode != 0: | ||
| raise RuntimeError(f"git {args[0]}: {tail(done.stderr.decode('utf-8', 'replace'), 200)}") | ||
| return done.stdout.decode("utf-8", "replace").strip() |
There was a problem hiding this comment.
[P3] Leading space in the first listed path skips the token scan
git strips the whole ls-files -z listing. A tracked name that starts with a space and sorts first is opened under the wrong path, so check_token can PASS while that file holds the token.
How to see it: A tracked file whose name starts with a space, containing the token, as the first ls-files -z entry. check_token status is PASS.
Smallest fix: Split the raw ls-files -z bytes on NUL, without str.strip.
| canary["evolution"]["evaluation"] = {"checks": {"canary": {"argv": [sys.executable, "-c", "pass"], | ||
| "timeout_seconds": 60}}, | ||
| "required": ["canary"], "max_age_seconds": 3600} | ||
| budget = canary["evolution"]["budget"] |
There was a problem hiding this comment.
[P3] A whole contract without a budget crashes preflight
validate_plan accepts a contract whose evolution object has no budget. build_contracts then indexes evolution.budget and raises KeyError. main reports an internal error and exits 1, after the doctor call has already run.
How to see it: A plan with three items and a contract whose evolution object is empty. build_contracts raises KeyError budget.
Smallest fix: If evolution.budget is not a dict, leave the contract for check_budget, which already reports a missing price table and max_dollars.
- Unknown spend (no host-observed cost but a spend record) stops the schedule instead of counting as zero. - The token check fails when git cannot list the tree or any file was not scanned. - ls-files -z output is split on NUL without stripping, so a name that starts with a space is scanned. - A contract without an evolution budget is left to check_budget instead of raising KeyError. - The printed receipt is the redacted one. - The exit code follows the final verdict after the credential scan. Each finding has a test in ReviewFindings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. All six findings are fixed in ca80c9f, each with a test, and the dry-run suite passes. I'm also pointing this PR at main, since #141 has merged. Generated by Claude Code |
main already chooses the exit code from the final verdict and prints the redacted receipt, so its versions of main() and finish's render are kept. The other review fixes and their tests are unchanged; main's Finalization tests and the ReviewFindings tests both pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Admission
Class: R
Agent-platform: claude
Reviewed-by: none recorded (an agent this session spawned reviewed it twice, see "Review"; that is not an independent session)
Admitted-by: self (Class R)
Evidence: tested
Running the script spends money and calls GitHits: that act is Class X, and Mateo executes it. The exact command is
python scripts/pilot_dry_run.py --plan plan.json. The expected cost is about $5 on Mateo's Anthropic key (the dry run's estimate for 3 to 5 items), capped at $8 by default (--budget; at most one call over it, see "Assumptions"), plus the GitHits lookups the treatment's model makes on his token, the doctor's check and the canary's one lookup. This pull request only adds the script, its offline tests and one paragraph of docs.Stacked on #141 (
fix/experiment-report-githits-api-arm): this branch is based on it and this pull request targets it. See "Dependency" below.Objective
Turn the GitHits pilot's dry run into one command that Mateo runs on his own machine with his own credentials: canary, dry run on 3 to 5 items with the nine metrics known for every thread, and the paired report.
What changed
scripts/pilot_dry_run.pyturns the GitHits pilot's dry run into one command that runs the documented pilot commands (docs/EXPERIMENTS.md,docs/GITHITS.md"The A/B report") in a home of its own, under the operator's own credentials:--basemust beHEAD); Claude Code>= 2.1.295(claude_code.version_tuple/MIN_VERSION, the same executable resolution as the provider); the api lane on the operator's ownANTHROPIC_API_KEYwith no third-party routing switch (claude_code.third_party:ANTHROPIC_BASE_URL,CLAUDE_CODE_USE_*,CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST);GITHITS_API_TOKENin the environment only (set, and in none of the repository's tracked or untracked files, its.env*,.mcp.jsonor local Claude settings, or the plan; the token is searched for as bytes and never printed);syberlabs githits doctorok, with its call record verifying under a key the host set; the guard key reaching the hook (CLAUDE_CODE_SUBPROCESS_ENV_SCRUBoff, and the host's ownquery_guard.py, run in the exec form Claude Code runs it, signing a block and a pass that verify under a key the host put in its environment); a hard budget cap (--budget, default $8) with astrictcontract that prices each model at all four rates and whosemax_dollarsfits under it; and the run's contracts installing cleanly (syberlabs init).propose --claude-code --bare --githits, one call, 6 turns, without the web tools so the launcher is its only way to look anything up) whose objective asks for two launcher lookups, one the guard must refuse (it holdscanary@example.invalid). It passes only when the guard refused a launcher lookup under a content rule (emailhere;not_launcher,shell_syntaxandguard_errordo not count) and passed one that GitHits answered (an authentic client record with statusokorcontamination_blocked), every decision verified and noneunguarded, and every model the call used is priced. The refused launcher call is the proof that Claude Code obeys the hook: had it run the call anyway, its client record would match no passed call and every record would beunguarded. A call that shows only one of the two is inconclusive and fails: a model that leaves the address out, or probes outside the launcher, proves no content refusal, and a launcher Claude Code will not run, or a GitHits that does not answer, proves no GitHits. A failed canary stops everything after that one call.--seedrepeats an order):start --unattended,experiment label(before the first candidate),propose --claude-code --bare --allowed-tools <the pilot's> [--githits --githits-arm on],run --once(accepts on the host's checks),export. A run starts only while the spend plus one thread's share fits under the cap andHEADis still the base. The schedule stops onprovider_guard_inactive, and on any call the host's meter missed: a call that endsprovider_failed,provider_timeoutorprovider_errorafter Claude Code ran is never priced (it raised before the host's meter), so that thread's cost and the total become unknown.review --record.experiment report --items items.txt --delivered reviewas--json(read) and--markdown(printed, saved). It must compare every item: an excluded item (arm_conflict,contaminated,missing_arm, ...) or a registered item that never ran fails the run; a check that keeps an item (unused,use_unknown, ...) is a warning.receipt.json: base and host commits, seed and schedule, versions, settings, contract digest and budget, the canary's lookups, spend (total, canary, per thread, cap, share, within cap), per-thread metric knownness (value, known, source, note, and the merge-based forms), the report's counts, checks and rule, the commands run (objectives as digests, the run directory as<run>), and a scan of every file the run wrote for both credentials.How Mateo runs it
PowerShell:
$env:ANTHROPIC_API_KEY = "...",Remove-Item Env:ANTHROPIC_BASE_URL,py scripts/pilot_dry_run.py --plan plan.json. The review prompts are answeredpassorfail(skipleaves one unreviewed); hand the keyboard to someone who was not watching the runs for a strictly blind review. Exit 0 is PASS, 1 FAIL, 2 refused before anything was spent. Everything lands in.syberlabs/pilot-dry-run/<time>-<seed>/(kept out ofgit status); sendreceipt.jsonandreport.md, which hold no objective, no code and no credential. If the canary names a model the plan does not price, add it topricesand run again (it cost one short call).The end of a run looks like this (from the offline test: stand-in
claude, fake GitHits; the numbers are not results):A failing run names its causes under the table, for example
run 1 (LIB-3 off): not accepted: a candidate passed the host's checks, and its promotion is provisional; the runner: check c1 -> refused (search_over_budget:dollars)when a call cost more than the thread's share.Plan keys:
items;prices, all four rates of each model above 0 (input, output, cache read, cache creation; Claude Code's calls are mostly cache tokens, and a cache rate left out is priced at the input rate, one at 0 as free), or a wholecontractholding such prices,max_dollarsandstrict; and what both arms share:max_calls(3),max_turns(30),allowed_tools(the pilot's,Read,Edit,Write,Glob,Grep,WebSearch,WebFetchas indocs/GITHITS.md; no shell),guard_max_lines,guard_max_chars,checks(the derived contract's checks; default whatsyberlabs initdetects, the whole unit suite for SyberWork, so narrowing it keeps the run short) andcanary_paths.What did not change
syberlabs/, and none of the files other sessions own (syberlabs/experiment.py,githits.py,githits_cli_entry.py,query_guard.py,claude_code.py,metrics.py,tests/test_claude_code.py,tests/test_experiment.py,docs/GITHITS.md,docs/METRICS.md,docs/CLAUDE_CODE.md,examples/githits_ab.py). The script composes the documented CLI commands. It is a tool with a named first consumer (Mateo's dry run before the pilot), not a subsystem.syberlabs proposegives it. The query guard, the meter's budget and the report's checks are read and enforced, never relaxed.Dependency
#141 (
fix/experiment-report-githits-api-arm, headd10fc3d) is open, so this branch is based on it and this pull request targets it; GitHub retargets it tomainwhen #141 merges. I started onmainat4219850while no such branch or pull request existed (checked through the GitHub API before branching), and the offline test there showed the bug #141 fixes:experiment reportexcluded every arm-B item asarm_conflict, because it read the treatment's GitHits only from a connected MCP server and the API-client arm records none. #141 opened while this branch was in progress and has gained two commits since (two_routes, and the API route indocs/EXPERIMENTS.md); this branch is rebased onto its latest head. On #141 the same test compares 3 of 3 items with no check raised, so the interim known-issue handling is gone: any item the report excludes now fails the dry run, and a check that keeps an item (two_routesincluded) is a warning.Assumptions
experiment.Run.value, counted up to the thread's first acceptance). Agent time istime_to_pass(first search to the evaluation of the accepted candidate):metrics.agent_timeneeds a merged pull request, which a dry run never has and the pilot has for at most one arm per item, so "known for every thread" could never hold in that form. Delivered isreviewed, the pilot's definition (accepted and passed the blind review,--delivered review): the merge-baseddeliveredis known in a dry run but "no" for every thread, which measures nothing. The merge-based values are in the receipt (strict) and the table header says which reading it shows.syberlabs settleand agent time to merge, the pilot's end-of-week step, are not exercised.The bar is about measurement, not success: a change the blind reviewer fails still counts (delivered: no, known), while a thread that never reaches a passing change does not (its time to a passing change is unknown), and fails the run with the reason it was not accepted.
--bareis what the canary shows (docs/CLAUDE_CODE.mdleaves it open).ANTHROPIC_API_KEYmust be set; proxies andANTHROPIC_AUTH_TOKENare not refused.max_dollarsis an equal share (cap over runs plus canary;--thread-budgetoverrides) andstrictis on; the host's meter stops a thread's next call past its share and refuses its promotion, and the script starts a run only while spent plus one share fits under the cap. The total therefore stays under the cap plus the cost of one call, bounded bymax_turns, as long as every call is metered; a call the meter missed makes the spend unknown and stops the schedule. A thread that crosses its share is not accepted and fails the bar, which is the signal to raise the budget before the pilot. A fee the API charges per server-side web search is not a token, so neither the meter nor the cap sees it; the receipt's web lookups count the searches.unlisted_runcheck see nothing else; the arms' contract is the repository's default contract with the plan's prices unless the plan gives a whole contract; the canary has its own contract with one trivial check, since it is never accepted.offandon(the report's defaults), and--githits-arm onputs the treatment's label inX-GitHits-Session-ID/X-GitHits-Agent.<run>, and the operator's git identity is not in it.Changes this needs outside the script (described, not made)
syberlabs/claude_code.py,_call, andsyberlabs/build.py: a call that fails after Claude Code ran (provider_failed,provider_timeout, or an unexpected fault the search records asprovider_error) raises beforemeter.measure, so the usage its transcript holds is never priced, andmetricsthen reports the thread's cost as known (when an earlier call was priced) or asnone. Metering the transcript's usage before raising, or marking the search's spend unknown, would make the cost metric honest; the script treats such a thread's cost and the total as unknown in the meantime.syberlabs/claude_code.py: a call that changes nothing is refusedno_changewithout its harness signal (spec/GAPS.md54), so a canary that obeys "no code change" cannot showapiKeySource, the model or the version that ran. Recordingharness(andlane) on the refusal would let the canary verify the credential source before any arm runs. There is also no way to pin the model (--model) or the per-call timeout fromsyberlabs propose; the frozen setup records what ran instead.syberlabs/economic.py(not on the owned list; left alone, since its default moves every priced contract): a price table without cache rates prices cache tokens at the input rate. For Claude Code, whose calls are mostly cache reads, that overstates cost several times over; the script requires all four rates instead.Known limitations and claim ceiling
Implemented and tested offline. No real Claude Code, model or GitHits API ran:
claudeis the stand-in oftests/test_claude_code.py(the real query guard hook runs as a process; Claude Code's Bash tool is emulated) and GitHits is the loopbackFakeGitHitsoftests/test_githits.py. Nothing is measured; whether real Claude Code runs the--settingshook under--bare, how a real model answers the canary's objective, and what a thread costs are for Mateo's first run.syberlabs settleand agent time to merge, the pilot's end-of-week step, are not exercised.claude_codemeters such calls (above).sys.executable, no POSIX-only calls, safe file names, both path separators, UTF-8 output. Everything ran here on Linux.Tests
tests/test_pilot_dry_run.py, 26 tests, about 50 s here (CPython 3.11, 3.12 and 3.13), hermetic: they clearANTHROPIC_BASE_URL,CLAUDE_CODE_PROVIDER_MANAGED_BY_HOSTand the other routing and scrub switches themselves, so they pass with or withoutenv -u.claudeis the stand-in oftests/test_claude_code.pyand GitHits the loopbackFakeGitHitsoftests/test_githits.py; nothing reaches a model or the GitHits API.syberlabscommand in-process: 9 preflight checks; the canary (one lookup refusedemail, one passed that the fake GitHits answered, no web tools); six runs in the seeded order with the pilot's tools; each thread's nine metrics known; labels before candidates;--githitsin the treatment only; the reviews recorded; the report comparing 3 of 3 items with no check raised and pairing the treatment's GitHits lookups against the baseline's none; spend7 xthe stand-in call priced by the host; neither credential in any output or file;python scripts/pilot_dry_run.py ..., every command a subprocess, review answers on stdin);not_launcherrefusal of a probe, no passed lookup, or a passed lookup GitHits answered with an error;provider_failed): its cost and the total unknown, no further call, the spend row failing, the reason naming it; an export that cannot be read keeping the one before it;.env, the doctor answering 401, the env scrub, no budget, a thread budget over the cap,HEADnot the base, a plan of two items), each with no Claude Code call and no thread; an interrupt before the first check still writing the receipt and the table;Evidence
On the final commit (
8775e7c), in this Claude Code cloud session (CPython 3.13), with the prefix the task asks for:main: optional tools and services that are not installed here (hypothesis, npm and Playwright, PostgreSQL).tests.test_pilot_dry_runand Make syberlabs experiment report read the GitHits pilot's API arm from the host's per-call launcher ledger (signal.githits), so arm-B items are compared instead of excluded as arm_conflict. #141'stests.test_experiment_githits_apialso pass on CPython 3.11 and 3.12 (38 tests each).claude2.1.310 in this session refuses, as it should:ANTHROPIC_BASE_URLandCLAUDE_CODE_PROVIDER_MANAGED_BY_HOSTare set, and there is no API key and no token. It exits 2 with nothing spent; GitHits was pointed at an unused loopback port and no request left the machine.Review
A read-only review by a separate agent (spawned by this session and given the code and the task, not my conclusions) read the script, the tests and the code they lean on. It found one blocker, three should-fix points and six nits. All are fixed here, with tests:
claude_coderaises before its meter. The gate could undercount the spend, and a failed export could overwrite a good record. Now the spend is unknown, the schedule stops, and the earlier export is kept.reviewedwas wrong: the merge-based form is known, as "no". The PR now also sayssettleis not exercised, and a thread that is not accepted says why.:in an item id naming an NTFS stream;..\path traversal;max_dollarstype;The same reviewer then checked the fixes. It found two more gaps in the canary and one in metering, all fixed in the follow-up commit with tests:
not_launcherrefusal of a probe counted as the refusal;provider_errorwas not treated as unmetered.Handoff
claude/pilot-dry-run-one-command, stacked on Make syberlabs experiment report read the GitHits pilot's API arm from the host's per-call launcher ledger (signal.githits), so arm-B items are compared instead of excluded as arm_conflict. #141 (fix/experiment-report-githits-api-arm).scripts/pilot_dry_run.py(new),tests/test_pilot_dry_run.py(new), anddocs/EXPERIMENTS.md(one paragraph in "The GitHits pilot" pointing at the dry run). Nothing insyberlabs/changes.--settingshook under--bare(the canary says so before any arm runs);receipt.jsonandreport.md. Then theclaude_codemetering of failed calls described above, and theno_changeharness signal if the canary should show the credential source.Generated by Claude Code