Skip to content

Integrate honest two-arm GitHits accounting and dry-run exit status - #145

Merged
sdcarlson merged 41 commits into
mainfrom
codex/githits-two-arm-integrated
Oct 27, 2026
Merged

sdcarlson merged 41 commits into
mainfrom
codex/githits-two-arm-integrated

Conversation

@sdcarlson

@sdcarlson sdcarlson commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Class: R
Agent-platform: codex
Admitted-by: self (Class R; protected landing authorized)
Reviewed-by: independent Codex correctness/security and simplification agents, final readback 260cffe
Evidence: tested offline; no live measurement or deployment

Eight concurrent Windows launchers could receive eight GitHits responses but silently lose authentic records. The unchanged #139 head reproduced the failure in 4/100 eight-process runs. This integration serializes open/append/flush/sync with the existing journal OS lock, preserves interrupted fragments as ignored evidence, and keeps missing, synthetic and unreconciled observations unknown, including no_change refusals.

The report distinguishes attempted counts from guard-passed use: blocked MCP attempts do not establish use; mixed client/MCP attempts are counted once and flagged. Final credential scans, printed verdicts and process status agree. Independent review found and repaired scan masking across credentials, raw credential output entering command logs before redaction, and wrapped transport write failures incorrectly labelled unsent. Regressions preserve unknown send state and fail closed without sensitive output.

Integration: #136, #139 and final #142 as one release boundary, #141/#143, #140, and reconciled #138 output behavior; current main b7297d8 is included. The skill follows the final guard quoting rules and its digest is pinned. No worker branch was overwritten.

Exact candidate 260cffe, tree 0c3b9cbc4a6706d3d23f3bf8153b41dad8de280d:

  • CI 40424019602: Python 3.11/3.12/3.13 each 1,154 tests OK (2 skips), Windows 1,154 OK (25 skips), strict G8 1,154 OK (2 allowed skips), G8 aggregate and Compose all pass.
  • All bridge/protected-text workflows and admit 40424019451 pass.
  • Local Windows: 1,154 tests OK (26 optional environment/tool skips), 23 golden traces, 11 scenarios, clean wheel and four public examples pass.
  • Windows concurrency stress: 100 eight-process rounds / 800 requests paired by server ID and digest on runtime-identical 24575f1; syberlabs source is byte-identical at the final candidate.
  • Real subprocess operator recipe with fake credentials and stand-ins: canary, six arm runs, three compared pairs, PASS/exit 0. Native CLI missing-credential preflight: REFUSED/exit 2, zero requests. Final scan, log redaction and failure exits have regressions.
  • Independent correctness/security and simplification reviewers cleared the final source and the Windows fixture/environment corrections; no assertions were weakened.

A complete, separate ENGINEERING SMOKE plan (CSV quoting, Unicode counts, strict chunked UTF-8) is in examples/githits-engineering-smoke/. scripts/githits_engineering_smoke.py prepares its frozen fixture/plan/manifest and invokes the existing pilot CLI. The default only prepares offline; --run-approved is for Mateo after specific paid-run approval. No formal pilot task, hidden check or solution is consumed. MasterMind pack 09864ac, its proposed 21 and old formal baseline remain unchanged; SyberEval redesign is not a dependency.

Proposed smoke setup: Anthropic direct Claude Code API, CLI 2.1.311, claude-sonnet-5-5, medium effort, forced five-minute cache, optional web tools off. Verified standard rates per million tokens: input $2, output $10, cache write $2.50, cache read $0.10 (https://platform.claude.com/docs/en/about-claude/pricing). Native binary integrity/version/exact-option help checks pass; real hook behavior awaits Mateo's canary.

The $8 setting is a token-cost dispatch threshold, permits one-call overshoot and excludes GitHits/server fees. It is not an all-in cap. Mateo owns credentials/spending and must explicitly approve and execute the live command; no paid run occurred. Exit 2 precedes model calls; GitHits doctor/preflight fees are outside that meter.

Operator command, prerequisites, outputs and stop conditions: docs/GITHITS_OPERATOR_HANDOFF.md. Claim ceiling: engineering implemented and tested offline. No live savings, successful pilot or deployment is claimed.

sdcarlson and others added 30 commits October 26, 2026 13:08
tests/test_claude_code.py inherited the suite's own environment. Wherever
ANTHROPIC_BASE_URL or CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST is set, as in
Claude Code cloud sessions, the provider rightly reads the run as
provider_3p and refuses the api lane. On main (7caa7c2) the file therefore
failed 5 and errored 15 of its 44 tests there. #83 noted 1 failure and
9 errors before #134 added the GitHits launcher tests.

ClaudeCodeCase.setUp now snapshots os.environ and removes
THIRD_PARTY_SWITCHES for each test, the approach #83 took in
WithClaudeCode.api_lane. Tests that want a switch still set it with
mock.patch.dict, and the environment is restored afterwards. No runtime
change.

- tests/test_claude_code.py, switches set (cloud session): 44/44 OK (was 5 F, 15 E)
- same file, switches unset (CI-like): 44/44 OK
- full suite, switches set: 987 OK, 24 skipped (as on main)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
… where the journal cannot say

tests/test_githits_accounting.py is the ground truth for "GitHits usage per
work item is measured exactly". CountingGitHits wraps the loopback FakeGitHits
with its own request log: method, path, a unique X-Request-ID, session, agent
and client headers, and its own digest of what it received. Real
ClaudeCodeProvider(githits=True) runs go through the stand-in claude, the real
launcher and the host's client. Per work item, the test compares the server's
requests, the journal's githits.<op> lookup facts and githits_lookups (up to
the first acceptance through experiment.runs), and pairs each fact with its
request by id, operation and digest.

The count matches the server for these cases:
- a lookup;
- results dropped for SyberLabs or the licence;
- guard-blocked queries;
- 401, 404, 500, 503 and 429 (no retries);
- a timeout and a dropped connection;
- six lookups in one call;
- eight launcher processes at once;
- fix rounds.

A refused target and a connection that never opened are counted by
definition, though GitHits received nothing.

metrics.py: githits_lookups is now unknown when the journal itself says the
count cannot be exact: a call's signal.githits.mismatch (a launcher call the
guard passed left no record), or a client-record fact with guard unguarded.
Before, a usage error (nothing sent) and a client that crashed after sending
both read 1, while the server saw 1 and 2. A forged record signed with the
call's key read 2 against the server's 1.

docs/METRICS.md: the new unknown condition and the ground-truth result. Stale
duplicate accuracy bullets removed.

Four findings in files not edited are pinned as expected failures:
- githits.py records nothing when _filter raises after the request was sent
  (a non-object search hit);
- githits.py records a timeout and a refused connection identically;
- claude_code.py leaves no trace of a launcher call in flight when the call
  times out;
- claude_code.py sends s1-b as the session for every thread when no
  thread_label is given.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…ailing check implicates a third-party package, at its installed version, with capped reads and hard stop rules

skills/githits/SKILL.md is appended to arm B's system prompt on every request, so its body went from 4,219 to
2,755 bytes (-35%) and from seven copies of the per-call launcher path to three, while gaining the rules that
decide cost per delivered item:

- when: only on a failing check that implicates a third-party package and does not already state the fix;
  never before a check has failed, for the repository's own logic, or for the standard library (E3: the host's
  check output is the signal that wins);
- how: registry:package@installed-version from the check output, a lockfile or installed metadata, and the
  exact symbol or error text; the characters the query guard blocks even inside quotes are named;
- lean reads: search --limit 3, grep --max-matches 5 with a 30 s wait (grep's server default wait is 0, so a
  cold target would read as no match), and reads of at most 60 explicit lines; example and list are dropped;
- stop: at most three launcher commands per task, none repeated, and a stop at a refusal, an error or a second
  guard block;
- apply and cite: results fit the failing call to the pinned API, no code is copied from them, and sources go
  in the reply, never in the code.

The Apache-2.0 attribution is unchanged; NOTICE and the front matter state the changes. tests/test_githits_skill.py
checks every command the skill teaches against the query guard (with this checkout's tree), the launcher's
parser and entry, the pinned OpenAPI contract and the stated caps, and the skill's character rule against the
guard both ways; tests/test_claude_code.py pins the new digest and runs the commands through a whole arm-B call.

Class: R
Agent-platform: claude

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TE74GRCKTiK6ZoexYxVqtU
…he experiment report

PRs #129 and #134 moved arm B of the GitHits pilot from an MCP server to the host's API
client (ClaudeCodeProvider(githits=True)), but syberlabs/experiment.py still told the arms
apart only by signal.harness.mcp_servers listing githits:connected. Every arm-B thread of an
API-route pilot therefore read "not connected" and every item was excluded as arm_conflict:
run end to end through the real provider (stand-in claude, loopback GitHits, real guard and
launcher), the report compared 0 items and the CLI exited 1 with "nothing was compared".

The report now also counts a call as having GitHits when its candidate carries
signal.githits, the ledger ClaudeCodeProvider writes for every call it gave the launcher, and
for no other, holding the host's own count of the client records it verified (records). That
is the strongest host-owned per-call evidence the provider records: written only after the
host wrote the call's launcher and record directory and read them back, and present whether
or not the model used it. Model-driven counts (signal.tool_usage, launcher_calls), the skill
text (signal.harness.skill) and the operator's arm name (signal.githits.arm) are not read for
the arm. The MCP route is read as before; existing tests are unchanged.

contaminated and unused also read the lookup facts the host observed (an mcp__githits__* call,
or a githits.<op> client record a guarded launcher call stands behind; never githits.doctor),
so a lookup from a call that failed before its candidate is seen. An unknown githits_lookups
(a call that declared no count, or, with #139's metrics, signal.githits.mismatch or an
unguarded record) is never read as zero: a baseline that shows no lookup is then
contamination_unknown, and a treatment use_unknown (new, kept), never unused. arm_conflict,
server_mixed and arm_unverified keep their meaning per call.

tests/test_experiment_githits_api.py adds synthetic checks for the API route, an end-to-end
run of both arms through ClaudeCodeProvider labelled and reported with the CLI (it prints the
paired result and the rule's verdict), a misconfigured pilot, and the rehearsal.
examples/githits_ab.py gains --route api (default: the real provider with its own stand-in
claude and loopback GitHits) beside --route mcp. docs/GITHITS.md documents the report on this
route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…truth branch

syberlabs/metrics.py conflicted. The resolution keeps both changes:
- #135: a githits.doctor fact is never counted, and a thread whose only client
  records are doctor records stays host-observed at 0, not the declared sum;
- this branch: the count is unknown when a call's signal.githits.mismatch says
  a launcher call the guard passed left no record, or when a counted record is
  unguarded.

An unguarded doctor record alone does not make the count unknown, because it
is never counted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
… refused launcher commands, arm A

#135 makes the launcher's entry run only the five lookups (example only with
license_mode strict) and strips every GitHits variable from arm A.

Rechecked against it:

- One case changed: doctor through the launcher is now refused before a client
  exists. The several-lookups case is five lookups (5/5/5), and the counter
  confirms the example went out with license_mode strict.
- New case: doctor, --help and --license-mode through the launcher send nothing
  and leave no record. The guard passed them, so the call reports
  githits_unrecorded, and the count is unknown: the journal cannot tell such a
  refusal from a lost request.
- New case: arm A has no launcher and no GitHits token; the counter sees nothing
  and the count is a known 0.
- The other cases are unchanged, and #135 fixed none of the four expected
  failures.

docs/METRICS.md: the githits_lookups row now says a githits.doctor fact is
never counted (#135's rule, in #138's wording), and the ground-truth bullet
records the above.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…-line test fall back when git cannot list files

tests/test_githits_skill.py: "it searches source, changelogs and docs" and the 30 s grep wait now rest on the
pinned OpenAPI fixture: GrepTarget.corpus defaults to all and the skill's grep sends no corpus; GrepRequest's
server-side wait defaults to 0, which is why the skill passes --wait-ms. The tree-line test lists the checkout's
files with git ls-files and, when git is missing or fails, walks the checkout instead (a superset, so the guard
check only gets stricter) rather than erroring.

Mutation check: 18 broken variants of the skill, 18 caught (two new: a grep scoped to source only, and the
corpus claim dropped).

Class: R
Agent-platform: claude

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TE74GRCKTiK6ZoexYxVqtU
…tHits MCP server

A treatment thread whose calls had both the host's API launcher (signal.githits) and a
connected GitHits MCP server is still the treatment, so it is not arm_conflict, but it is not
the single route the pilot runs. Where it has client records, metrics also counts only those
and leaves the MCP lookups out (spec/GAPS.md 59). The report now says so with a kept check,
two_routes. It is not excluded, because the pilot pre-registers its exclusions. A stray server
in arm A stays arm_conflict, and its lookups contaminated.

tests/test_experiment_githits_api.py runs it end to end through ClaudeCodeProvider with
githits=True and --mcp-config, plus the same stray server in arm A. docs/GITHITS.md lists the
check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…error: lines and the drop reasons

tests/test_githits_skill.py: "Stop at a `refused:` or `error:` line" and "Results the host drops (license, excluded
owner)" now match what the launcher prints: a SyberLabs target exits 2 with "refused: ", a call with no token exits
1 with "error: ", and the compact header names "dropped (excluded owner)" and "dropped (license)". Renaming any of
them fails the test, so the skill's stop rule cannot drift from the output it tells the model to watch for.

Mutation check: 19 broken variants of the skill, 19 caught (new: the stop rule's prefixes renamed).

Class: R
Agent-platform: claude

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TE74GRCKTiK6ZoexYxVqtU
tests/test_githits_skill.py: every registry the skill names as a target prefix (pypi, npm, crates) is in
githits.REGISTRIES and parses to a pinned target, and a target with no @Version asks for the latest, as the skill
says. A registry the client does not know would fail a search before any request, with no record.

Mutation check: 20 broken variants of the skill, 20 caught (new: a wrong registry name).

Class: R
Agent-platform: claude

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TE74GRCKTiK6ZoexYxVqtU
…they describe

tests/test_githits_skill.py: the module docstring now names every rule the module checks against its source of
truth (the guard, the pinned contract, the launcher's output and the client's registries). Text only.

Class: R
Agent-platform: claude

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TE74GRCKTiK6ZoexYxVqtU
…its Claude Code launcher before the A/B pilot

An independent adversarial verification of #129 and #134 (with #135), not built through
`syberlabs propose`. Each finding has a regression test in tests/test_githits_adversarial.py
that fails on main (4219850) and passes here.

Contamination control (githits.py):
- every exclusion layer reads a text as a URL parser (WHATWG URL with IDNA) or a lenient
  server may: percent-decoded to a fixed point, NFKC-folded with format characters removed,
  ideographic and fullwidth stops as ".", a backslash as "/", a host's port or root dot
  ignored, and for a locator "." and ".." segments resolved and whitespace removed
  (https://github.com:443/SyberLabs/x, https://github.com/x/../SyberLabs/x, fullwidth
  hosts and "github: SyberLabs/x" left the machine before);
- an owner's own domains (syberlabs.io, www.syberlabs.io, rise.syberlabs.io) are excluded
  like <owner>.github.io; before, only the bare apex was refused, and only as a target;
- a search hit, grep hit or list entry whose text cites an excluded owner is dropped and
  counted, as a read body citing one is blocked.

Licenses (githits.py): a code-host source in any Markdown or HTML link form (reference,
titled, angle-bracketed, autolink, anchor, bare URL with its license) and any
SPDX-License-Identifier tag in an example's code are judged; before, one permissive inline
link admitted an example whose other source was copyleft.

The launcher and its guard (query_guard.py, githits_cli_entry.py): an unquoted word a shell
reads differently from the guard is refused (=cmd under zsh, NAME=value under bash set -k,
which would put GITHITS_API_URL into the launcher's environment, @name under PowerShell,
and ^ under zsh's EXTENDED_GLOB); a Bash timeout under the two-minute default is refused, so
the agent cannot end the client after its request left and before it records; a SIGTERM
becomes SystemExit in the launcher's entry, so a launcher ended that way records its call.

Records and accounting (githits.py, claude_code.py; #139's four findings):
- a call records whatever happens after its request may have left (an answer this client
  cannot read is malformed_response, an interruption is recorded before it propagates);
- each record says whether its request was written to the server (sent), so a timeout
  after GitHits received a request and a refused connection no longer leave the same record;
- a failed or timed-out call whose launcher calls outnumber its records journals those
  records guard: unguarded, or a githits.unrecorded fact for each launcher call when it has
  none, instead of losing the lookup that was in flight;
- X-GitHits-Session-ID is <thread>-<arm> for a provider built in Python too, not s1-b.

Token: doctor's Retry-After is scrubbed like X-Request-ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…sh rendering after an ok record, and pin #138's delivery gap

#135 lets the launcher run only the five lookups, but their path arguments were never checked:
read site:docs.python.org --path //syberlabs.io/... (a network-path reference), --path
@syberlabs.io/... (userinfo, when a server concatenates site and path), an absolute URL, and
the same in list --path, grep --glob and search path_prefix pass the guard as plain words. A
server that joins the site and the path as a URL would read SyberLabs' site. request_paths
collects them and call refuses one that mentions an excluded owner, before any request.

The CLI writes the record before it renders. A hit outside the contract (a locator that is
not an object, a list entry or grep hit that is not one) made render raise after an ok record:
the agent saw a traceback while the record said ok. render now reads such an item as empty.

#138 (open) writes the record after its deliver callback and records error only for an OSError
or ValueError from it; any other exception leaves no record for a request that reached GitHits.
Pinned as an expected failure until #138 records every failed delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…d them differently

skills/githits/SKILL.md (body 2,755 -> 2,844 bytes, still 33% under the old skill's 4,219):

- When: "the check's output does not already say what the package expects" became "does not already show how to
  fix it, for example by naming the replacement". On an error raised inside PyYAML 6.0.1 ("could not determine a
  constructor for the tag"), a Sonnet stand-in read the old clause as "the output does not say how to register a
  constructor" and looked up; a Haiku stand-in read it as "the output says what is missing" and did not. The new
  clause names the test (does the output show the fix) and the common case that passes it (a named replacement).
- Locate: "Locate with one command ...; the other only if the first finds nothing". On a removed keyword argument,
  where both a missing name (grep) and an existing callable used wrongly (search) apply, Haiku stand-ins ran both,
  spending the whole three-command budget before the read; Sonnet ran one.

tests/test_claude_code.py: SKILL_SHA256 is the new blob's digest. tests/test_githits_skill.py passes unchanged, and
the mutation check still catches 20 of 20 broken variants.

Class: R
Agent-platform: claude

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TE74GRCKTiK6ZoexYxVqtU
#142 closes the four accounting findings this branch pinned as expected
failures:
- the client records every failure after a request is sent;
- records carry sent;
- a failed call journals its unrecorded launcher calls;
- a provider built in Python names its thread in the session.

It is based on main 4219850, which this branch already merged. The merge was
clean. #142 should merge first; this branch includes its commits until then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…t name one of its modules, classes or functions

skills/githits/SKILL.md (body 2,844 -> 2,874 bytes): "an import, attribute or argument error names its symbol"
became "... names one of its modules, classes or functions". On AttributeError: 'dict' object has no attribute
'data' (marshmallow 3 returning a dict where marshmallow 2 returned a result), Sonnet stand-ins read the old clause
as intended (no package symbol is named) and made no lookup; a Haiku stand-in read the bare attribute name as the
package's symbol and grepped "data", a word that matches nearly every file. Naming the kinds of symbol keeps the
trigger on the package's API surface for smaller models too.

tests/test_claude_code.py: SKILL_SHA256 is the new blob's digest. The mutation check still catches 20 of 20.

Class: R
Agent-platform: claude

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TE74GRCKTiK6ZoexYxVqtU
…ed against the server

#142 closes the four findings this module pinned as expected failures:

- The markers are gone. The three tests of githits.py and claude_code.py
  behaviour now pin what #142 does, in a Fixed class: an unreadable answer is
  recorded (malformed_response, sent), a timeout and a refused connection
  differ in sent, and a provider built in Python names its thread in the
  session.
- The in-flight-at-timeout case moves to Unknowable. Its failed call now
  journals its records unguarded, or a githits.unrecorded fact when it has
  none, so the count is unknown instead of a known number one short.
- The crash-after-send twin is now recorded, so it moves to Exact (2/2/2). The
  usage-error case stays unknown: a SIGKILL or a failed record write after a
  request left still leaves the same journal.

Every case now also compares the facts with sent: true to the server's count.
They match in every case except a lookup killed in flight, which the journal
marks unguarded or githits.unrecorded.

docs/METRICS.md: the unknown condition names githits.unrecorded facts, and the
ground-truth bullet records the above.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…ent report's --server help

docs/EXPERIMENTS.md described only the MCP route. It now covers both routes. On the API
route (--githits, since #134), arm B is identified from signal.githits, the host's per-call
launcher ledger; on the MCP route (--mcp-config), from signal.harness.mcp_servers. The doc
also says how a GitHits lookup is shown and that an unknown count is never read as zero.

The pilot protocol's commands move to the API route, with a note on the MCP era. The
checks table rewrites the arm_conflict, contaminated, arm_unverified, server_mixed, unused
and contamination_unknown rows, and adds two_routes and use_unknown with what each means and
what an operator should do.

The --server help in syberlabs/cli.py no longer implies an MCP server is the only route: for
githits it also covers the host's API launcher. The edit is to a line that the open #137
and #138 do not touch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
… differently

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Preflight (Claude Code >= 2.1.295, the api lane with no third-party routing
switch, GITHITS_API_TOKEN in the environment and in no file, githits doctor
with a verified record, the guard key reaching the host's hook, a hard budget
cap, the run's contracts), then a canary in the treatment's mode that must show
a verified guard decision, then both arms of 3 to 5 items from one base commit
in a seeded order, the blind review, experiment report --items --delivered
review, and the bar: the nine metrics known for every thread. Prints a
PASS/FAIL table with the total spend and writes a JSON receipt.

Tests run it offline with the stand-in claude and FakeGitHits; one runs it as
the operator does, every syberlabs command a subprocess. Nothing in syberlabs/
changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122mN5uhCiz5UAZwEGzVK8z
…al output; more tests

The summary no longer repeats a preflight FAIL as a reason line, and only says to send
receipt.json with report.md when report.md exists. Tests cover argument refusals that must
create nothing, and a thread whose check never passes (agent time stays unknown, FAIL).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122mN5uhCiz5UAZwEGzVK8z
The report now reads the API arm from the host's launcher ledger (signal.githits), so the
script's known-issue path for arm_conflict is gone: any item the report excludes, or a
registered item that never ran, fails the run, and each check that keeps an item (unused,
use_unknown, ...) is a warning. The offline run compares 3 of 3 items. docs/EXPERIMENTS.md
points at the dry run from the GitHits pilot's protocol.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122mN5uhCiz5UAZwEGzVK8z
- A call that ends provider_failed or provider_timeout after Claude Code ran is never priced
  (claude_code raises before its meter), and neither is a search that never finished: the
  thread's cost and the total are unknown, no further run starts, and the spend row fails.
  A failed export keeps the one before it; a thread never exported has an unknown spend.
- Prices need all four rates above 0: a cache rate left out is priced at the input rate,
  one at 0 as free, and Claude Code's calls are mostly cache tokens.
- The canary passes only when the guard refused a lookup and passed one that reached
  GitHits; it runs without the web tools, so the launcher is its only way to look up.
- A thread that is not accepted says why (its candidates, the runner's word, its search).
- The plan's tools default to the pilot's (docs/GITHITS.md): the web tools included.
- Small: no traceback when interrupted before the first check, item ids made safe as file
  names, Windows path traversal refused, the credential scan reads every file whole and
  counts what it read, a whole contract's max_dollars must be a positive number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122mN5uhCiz5UAZwEGzVK8z
…p answered

From the reviewer's check of the fixes: a not_launcher refusal (a probe outside the launcher)
proves neither that a content rule fires nor that Claude Code obeys a refused launcher call,
so only the guard's content rules count; and a client record whose request GitHits did not
answer (error, excluded target) is no lookup that reached GitHits. provider_error, a fault
the search records after the provider raised, joins the calls the meter may have missed.
The runs' header says agent time is not to a merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122mN5uhCiz5UAZwEGzVK8z
…t head

Rebased onto fix/experiment-report-githits-api-arm at d10fc3d (two_routes, the API route in
docs/EXPERIMENTS.md); the report's new kept check is a warning like the others. Each thread's
harness in the receipt now carries the license policy digest the host recorded for the
launcher (signal.githits.license_policy), part of the pilot's frozen setup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122mN5uhCiz5UAZwEGzVK8z
The stop rule said that a name neither search nor grep finds is absent from
that version. The compact text the agent reads prints neither a grep's
traversal nor its unavailable targets, so a target GitHits has not prepared
yet reads as "grep: 0 matches of 0", exactly like a name the target lacks;
--wait-ms 30000 only bounds the preparation wait. The rule now says that
finding nothing proves nothing and sends the agent back to the check's
output, which already shows what the installed version rejects (Codex review
on #140).

tests/test_githits_skill.py pins the reason: the skill's own grep, served an
unprepared target that is valid under the pinned contract, prints the same
text as an empty grep. SKILL_SHA256 updated; the body is 2,914 bytes (+40).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TE74GRCKTiK6ZoexYxVqtU
@sdcarlson
sdcarlson marked this pull request as ready for review October 27, 2026 00:20
@sdcarlson
sdcarlson merged commit 0b03040 into main Oct 27, 2026
25 checks passed
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T00:22:04.309719Z 260cffe Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants