Repository navigation
Make syberlabs experiment report read the GitHits pilot's API arm from the host's per-call launcher ledger (signal.githits), so arm-B items are compared instead of excluded as arm_conflict. - #141
Conversation
…he experiment report PRs #129 and #134 moved arm B of the GitHits pilot from an MCP server to the host's API client (ClaudeCodeProvider(githits=True)), but syberlabs/experiment.py still told the arms apart only by signal.harness.mcp_servers listing githits:connected. Every arm-B thread of an API-route pilot therefore read "not connected" and every item was excluded as arm_conflict: run end to end through the real provider (stand-in claude, loopback GitHits, real guard and launcher), the report compared 0 items and the CLI exited 1 with "nothing was compared". The report now also counts a call as having GitHits when its candidate carries signal.githits, the ledger ClaudeCodeProvider writes for every call it gave the launcher, and for no other, holding the host's own count of the client records it verified (records). That is the strongest host-owned per-call evidence the provider records: written only after the host wrote the call's launcher and record directory and read them back, and present whether or not the model used it. Model-driven counts (signal.tool_usage, launcher_calls), the skill text (signal.harness.skill) and the operator's arm name (signal.githits.arm) are not read for the arm. The MCP route is read as before; existing tests are unchanged. contaminated and unused also read the lookup facts the host observed (an mcp__githits__* call, or a githits.<op> client record a guarded launcher call stands behind; never githits.doctor), so a lookup from a call that failed before its candidate is seen. An unknown githits_lookups (a call that declared no count, or, with #139's metrics, signal.githits.mismatch or an unguarded record) is never read as zero: a baseline that shows no lookup is then contamination_unknown, and a treatment use_unknown (new, kept), never unused. arm_conflict, server_mixed and arm_unverified keep their meaning per call. tests/test_experiment_githits_api.py adds synthetic checks for the API route, an end-to-end run of both arms through ClaudeCodeProvider labelled and reported with the CLI (it prints the paired result and the rule's verdict), a misconfigured pilot, and the rehearsal. examples/githits_ab.py gains --route api (default: the real provider with its own stand-in claude and loopback GitHits) beside --route mcp. docs/GITHITS.md documents the report on this route. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be677064a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if tool.startswith(f"mcp__{server}__"): | ||
| found += 1 |
There was a problem hiding this comment.
Exclude guard-blocked MCP attempts from observed lookups
When an MCP lookup is blocked by the query guard and its provider call fails before registering a candidate, this branch still counts the attempt as evidence that GitHits was used. Transcript.lookups() in syberlabs/claude_code.py journals blocked attempts too, with the blocking rule in guard and ok: false; a verified observation therefore does not establish that the request reached the server. This newly excludes an otherwise clean baseline as contaminated, or suppresses unused for an idle treatment, even though the guard prevented the lookup. Check the guard outcome before counting MCP facts as observed use, as the client-record branch already does.
Useful? React with 👍 / 👎.
…tHits MCP server A treatment thread whose calls had both the host's API launcher (signal.githits) and a connected GitHits MCP server is still the treatment, so it is not arm_conflict, but it is not the single route the pilot runs. Where it has client records, metrics also counts only those and leaves the MCP lookups out (spec/GAPS.md 59). The report now says so with a kept check, two_routes. It is not excluded, because the pilot pre-registers its exclusions. A stray server in arm A stays arm_conflict, and its lookups contaminated. tests/test_experiment_githits_api.py runs it end to end through ClaudeCodeProvider with githits=True and --mcp-config, plus the same stray server in arm A. docs/GITHITS.md lists the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…ent report's --server help docs/EXPERIMENTS.md described only the MCP route. It now covers both routes. On the API route (--githits, since #134), arm B is identified from signal.githits, the host's per-call launcher ledger; on the MCP route (--mcp-config), from signal.harness.mcp_servers. The doc also says how a GitHits lookup is shown and that an unknown count is never read as zero. The pilot protocol's commands move to the API route, with a note on the MCP era. The checks table rewrites the arm_conflict, contaminated, arm_unverified, server_mixed, unused and contamination_unknown rows, and adds two_routes and use_unknown with what each means and what an operator should do. The --server help in syberlabs/cli.py no longer implies an MCP server is the only route: for githits it also covers the host's API launcher. The edit is to a line that the open #137 and #138 do not touch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Add scripts/pilot_dry_run.py: the GitHits pilot's dry run as one command (stacked on #141)
Summary
Since #134,
syberlabs experiment reportexcluded every arm-B item of an API-route GitHits pilot asarm_conflict, so it compared nothing and could give no go/no-go. It now reads arm B from the host's per-call launcher ledger (signal.githits), and the pilot's go/no-go comes out decided. Three commits:be67706: the fix insyberlabs/experiment.py.signal.githits.use_unknown(new, kept) andcontamination_unknown.--route api, anddocs/GITHITS.md.86e509b:two_routes(new, kept), for a stray GitHits MCP server beside the launcher in arm B.d10fc3d:docs/EXPERIMENTS.mdand the--serverhelp.use_unknownandtwo_routes.--serverhelp no longer implies MCP is the only route.What was wrong
PRs #129 and #134 moved arm B of the GitHits pilot from an MCP server to the host's API client:
ClaudeCodeProvider(githits=True)writes a per-call launcher, delivers the adapted skill, and journals onelookupfact per authentic client record. #134 taughtsyberlabs/metrics.pyto count those lookups, butsyberlabs/experiment.pystill told the arms apart only by an MCP server:SERVER = "githits", read fromsignal.harness.mcp_servers(githits:connected). #135 did not touchexperiment.py.I proved it end to end before changing anything, and again on the current
main(4219850, after #135):ClaudeCodeProvider: arm A withgithits=False, arm B withgithits=Trueand one launcher search.claudefromtests/test_claude_code.py, the loopbackFakeGitHits, and the real query guard and launcher.syberlabs experiment labelbefore its first candidate, then accepted.Results on
main:signal.githits = {"arm": "on", "launcher_calls": 1, "launcher_passed": 1, "records": 1, ...}, nomcp_serversinsignal.harness, and onegithits.searchlookup fact.experiment.runsreadserverasFalsefor both arms.experiment.reportreturneditems {'labelled': 1, 'compared': 0, 'excluded': 1}, with flags['arm_conflict']and excluded['arm_conflict']. The rule wasmet: None("no item delivered in both arms").syberlabs experiment report --name githits-pilotexited 1. It printedcheck arm_conflict (excluded): LIB-1: the record contradicts the label: the server was connected in the baseline, or not in the treatment, thenno item ran in both arms without a check that excludes it, so nothing was compared.So every arm-B item of an API-route pilot is excluded, and the go/no-go cannot be computed.
--server nonewould hide the problem only by dropping the arm, contamination and use checks altogether. On this branch the same run givescompared: 1, no check,serverTruefor arm B, and exit 0.The fix
How the report identifies the arm. Per call, the report now counts GitHits as present when the candidate carries
signal.githits(for--server githits, the default), as well as in the MCP form. Why this field, after readingclaude_code.py:ClaudeCodeProvider._callwritessignal.githitson the candidate of every call it ran withgithits=True, and on no other. It writes it only after it has written that call's launcher, created the record directory and key, and read the records back under that key. It is the host's own ledger of the capability.records: 0), so it reads asunused, not as a conflict. This keeps provisioning (the arm) separate from use.mcp_servers, so a thread that is inconsistent across calls is still detected.signal.tool_usageandsignal.githits.launcher_calls;signal.harness.skill. Instructions without the launcher give no access, and its digest (LF-normalized since Harden the GitHits launcher route before the A/B pilot: the launcher entry accepts only the five lookup operations with license_mode strict, arm A loses every GitHits variable, the skill digest is LF- #135) plays no part, so no expected digest has to follow that normalization;signal.githits.arm(from--githits-arm);githits.<op>lookup facts. They are the strongest evidence of use, but they are absent when nothing was looked up, so they cannot tell an idle treatment from a misconfigured one. They feedcontaminatedandunusedinstead.recordscount. A baregithits: 2ortruein an adapter'ssignaldoes not count.What each check now means (existing names unchanged; two kept checks added,
use_unknownandtwo_routes):arm_conflict(excluded): a baseline call had GitHits (the ledger, or the server connected), or a treatment thread's calls had none.contaminated(excluded): the baseline shows a GitHits lookup. That is either itsgithits_lookupsis above zero, or its thread holds a lookup fact the host observed and can vouch for: anmcp__githits__*call, or agithits.<op>client record that a guarded launcher call stands behind (guardpass). The second reading also catches a lookup from a call that failed before its candidate: such a lookup is journaled but never counted by a registration. As inmetricssince Harden the GitHits launcher route before the A/B pilot: the launcher entry accepts only the five lookup operations with license_mode strict, arm A loses every GitHits variable, the skill digest is LF- #135, agithits.doctorfact is a readiness check, not a lookup.contamination_unknown(kept, as before): the baseline shows no lookup and its count is unknown.unused(kept): the treatment shows no lookup and its count is known to be zero. A launcher call the guard refused never reached GitHits.use_unknown(new, kept): the treatment shows no lookup and its count is unknown, so it can be shown neither to have used GitHits nor not to.server_mixed(kept): a treatment thread had GitHits in some calls only. A baseline thread like that isarm_conflict.two_routes(new, kept; follow-up commit): a treatment thread had GitHits by both routes: the ledger, and a connected GitHits MCP server left in arm B by mistake.arm_conflict.metricscounts only those and leaves the MCP lookups out (spec/GAPS.md59, item 14).arm_conflict, and its lookupscontaminated.arm_unverified(kept): no call recorded its MCP servers or the ledger.MCP-era records are read as before: a call counts by either route, and
--server <other>never reads the launcher.tests/test_experiment.pyis unchanged and passes.How this interacts with #139. With #139's metrics,
githits_lookupsbecomes unknown when the journal cannot say how many lookups reached GitHits: a call'ssignal.githits.mismatch, or anunguardedclient record. The report never reads that unknown as zero:contamination_unknown, unless a vouched-for lookup fact shows a lookup, which makes itcontaminated. An unguarded record is deliberately not vouched for, which matches Check githits_lookups against an independent loopback GitHits counter in seventeen cases and make it unknown where the journal cannot say how many requests reached GitHits (merge #142 first). #139's "the record cannot say".use_unknown, unless such a fact shows use. Neverunused.arm_conflict.With
main's metrics the same journals readunusedandcontaminated(it counts what it has). The new tests assert whichever outcome matches the countmetricsreturns, so they assume neither reading. I checked them against #139's commit cherry-picked onto this branch in a scratch worktree, with itsmetrics.pyhunk merged with #135'sdoctorexclusion: my tests andtests/test_experiment.pypass (47 tests). There, a mismatch treatment readsuse_unknownand an unguarded baseline record readscontamination_unknown. (#139's owntest_githits_accountingand onetest_metricsassertion need adjusting for #135 when #139 rebases. That is unrelated to this change.)Rehearsal.
examples/githits_ab.pygains--route api, now the default, alongside--route mcp(the previous behaviour):ClaudeCodeProvideron the api lane, arm B withgithits=True.claudeis self-contained: it prints stream-json, runs the--settingshook before each tool call, and its Bash tool runs only the launcher. GitHits is a loopback stand-in.LIB-4's treatment runs without--githitsand is excluded.Docs. I updated the experiment module docstring, and added a section "The A/B report" to
docs/GITHITS.md, placed after "The pinned contract".docs/EXPERIMENTS.md(third commit) now covers both routes:two_routesanduse_unknown, with what an operator should do.The
--serverhelp insyberlabs/cli.pynow says that forgithitsthe host's API launcher counts too. Indocs/GITHITS.md: It covers the commands for both arms, what the report reads, the checks on this route, unknown counts, and the limit: a proposer that writes its ownsignalcould declare a ledger, just as it could declare an MCP server. The doc change is additive only, and #135's text is unchanged.Evidence
New tests.
tests/test_experiment_githits_api.pyadds 12 tests:ClaudeCodeProvider, labelled and reported with the CLI;githits=Trueplus--mcp-config), and the same server in arm A;Against
main'sexperiment.pythey fail (15 failures and 1 error). There the end-to-end readscompared: 0, excluded: 2and the API-route rehearsal readscompared: 0, excluded: 7.Printed result. The end-to-end test prints the paired result and the rule's verdict:
Rehearsal, API route.
python examples/githits_ab.py: 6 items compared,check arm_conflict (excluded): LIB-4andcheck unused (kept): LIB-4. The rule is not met overall and met onneeds_lookup, asdocs/EXPERIMENTS.mdsays. The loopback GitHits received 9 requests.--route mcpgives the same items, checks and verdicts as before.Full suite.
env -u ANTHROPIC_BASE_URL -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST PYTHONPATH=. python3 -m unittest discover -s testson this branch, rebased on4219850:Ran 1007 tests in 828.085s,OK (skipped=24)atbe67706;Ran 1008 tests in 744.475s,OK (skipped=24)at86e509b;Ran 1008 tests in 828.271s,OK (skipped=24)atd10fc3d. Conformance was re-run atd10fc3dand matched 23 golden traces.Conformance.
PYTHONPATH=. python3 -m conformance.run: matched 23 golden traces.CI at
d10fc3d: every check is green in both thepull_requestrun and thepushrun: Python 3.11–3.13, Windows, compose, G2–G8, admit, policy.pushrun's first Windows attempt failed in the sharded unit-test step. I couldn't fetch its log here: the session proxy blocks the blob-storage log URL.pull_requestrun, and again when the failed job was re-run. The commit changes only a doc and a help string, which no test reads, so it was a flake.Critic. The reference critic's integrity rules (
examples/critic_rules.py) find nothing in the three changed Python files.The numbers come from stand-ins: they exercise the pipeline and say nothing about GitHits.
Rehearsal: a valid go/no-go on the API route
python examples/githits_ab.py(default--route api), run at86e509b, sends seven items through both arms of the realClaudeCodeProvider, with a stand-inclaudeand a loopback GitHits. Simulated: the numbers are invented to exercise the pipeline and say nothing about GitHits. The go/no-go comes out decided:--delivered review): not met, because the median paired cost over the 5 items delivered in both arms is +$0.0027;needs_lookup: met, at $0.0321 -> $0.0160 per item;LIB-4, whose treatment ran without--githits, is excluded asarm_conflict, with 6 of 7 items compared.On
mainthe same rehearsal compares nothing (compared: 0, excluded: 7).Full output of
python examples/githits_ab.pyRelated open pull requests
eval-score). It does not label threads intosyberlabs experimentor call this report, so the two do not overlap. They read the same host record, though: SyberEval's E6, the GitHits pilot, will run twoclaude-codearms that differ only ingithits. Its arm-B candidates will therefore carry the samesignal.githitsledger this report now reads as the arm's evidence.eval-scorecan take that reading (and "an unknown count is never zero") from here rather than derive its own. Add the SyberEval core (syberlabs/eval: pre-registered specs with frozen digests, tasks with host-held hidden acceptance checks, arms mapped to the real providers, a runner that executes every task x #137 already conflicts withmaininspec/GAPS.md; this branch adds no conflict with it.mainin six files,docs/GITHITS.mdamong them (one region: its added line at the end of "The launcher's entry"). This branch adds no conflict region there: its doc change is additive and placed after "The pinned contract".syberlabs/cli.py, which Add the SyberEval core (syberlabs/eval: pre-registered specs with frozen digests, tasks with host-held hidden acceptance checks, arms mapped to the real providers, a runner that executes every task x #137 and Make the GitHits launcher entry survive a console that cannot encode a result: it reuses the CLI's unicode reconfiguration before any output, and the client delivers its output before it writes the re #138 also edit. This branch changes one line: the--serverhelp, now three lines. Add the SyberEval core (syberlabs/eval: pre-registered specs with frozen digests, tasks with host-held hidden acceptance checks, arms mapped to the real providers, a runner that executes every task x #137's hunks are the module docstring, the imports, and the parser and dispatch aroundgithits.add_parser. Make the GitHits launcher entry survive a console that cannot encode a result: it reuses the CLI's unicode reconfiguration before any output, and the client delivers its output before it writes the re #138's aresurvive_unicode, the--githitsoptions and the provider call. Neither touches the--serverline, and a test merge of each PR head (bff5251,49c8aca) with this branch adds no conflict beyond the ones it already has withmain.Left to others
test_githits_accounting, and onetest_metricsassertion, need adjusting for Harden the GitHits launcher route before the A/B pilot: the launcher entry accepts only the five lookup operations with license_mode strict, arm A loses every GitHits variable, the skill digest is LF- #135'sdoctorchange when Check githits_lookups against an independent loopback GitHits counter in seventeen cases and make it unknown where the journal cannot say how many requests reached GitHits (merge #142 first). #139 rebases. I've left that to the agent updating Check githits_lookups against an independent loopback GitHits counter in seventeen cases and make it unknown where the journal cannot say how many requests reached GitHits (merge #142 first). #139.metricscounts only client records once a thread has any (spec/GAPS.md59, item 14). So a thread with both routes undercounts its GitHits lookups. The report says so (two_routes); fixing the count ismetrics' to do.An agent session built this directly in a worktree, not through
syberlabs propose.Class: R
Agent-platform: claude
Admitted-by: self (Class R)
Claim ceiling: implemented and tested (stand-in
claude, loopback GitHits). Not measured; no pilot has run on it.https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Generated by Claude Code