Skip to content

Check githits_lookups against an independent loopback GitHits counter in seventeen cases and make it unknown where the journal cannot say how many requests reached GitHits (merge #142 first). - #139

Merged
7 commits merged into
mainfrom
test/githits-accounting-ground-truth
Oct 27, 2026

Conversation

@sdcarlson

@sdcarlson sdcarlson commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Merge #142 first; this PR includes its commits until then.

Ground truth for the claim in SyberLabs' scope reply to GitHits: "Every call is recorded by the factory itself… GitHits usage per work item is measured exactly, not reported by the agent." PRs #129 and #134 moved arm B's lookups to the host's launcher and client; githits_lookups now counts lookup facts with tool githits.<op>. This checks that count against an independent counter, E4-style, on main with #135 and with #142 (merge history below).

Result

Sent is the number of facts whose record says the request was written to GitHits (sent: true, a field #142 adds).

Case Server Journal Sent Metric Match
One lookup 1 1 1 1 yes
Results filtered: search hits dropped (SyberLabs), example blocked (contamination), example dropped (licence), read of a SyberLabs-served repo 4 4 4 4 yes
Target the client refuses (2 refused, 1 normal) 1 3 1 3 no, by definition: attempted lookups; sent: false marks the 2
Query the guard blocks (not the launcher, shell syntax, secret, email) 0 0 (4 Bash facts) 0 0 (lookups_blocked 4) yes
API errors 401, 404, 500, 503, 429 5 5 5 5 yes; one request each, no retries
Timeout (--timeout 1) and dropped connection 2 2 (no request id) 2 2 yes
Connection refused 0 1 0 1 no, by definition; sent: false
Answer the client cannot read (non-object search hit), beside one normal 2 2 2 2 yes since #142, which records it; before, it was unrecorded and read 1
Five lookups in one call (search, example, read, list, grep) 5 5 5 5 yes, in request order; the example went out with license_mode: strict
doctor, --help, --license-mode yolo through the launcher, beside one search 1 1 1 unknown no request, as #135 intends; unknown, because the journal can't tell a refusal from a lost request
Arm A: no launcher, no GitHits variable 0 0 0 0 yes
Eight launcher processes at once 8 8 records 8 n/a yes, 0 records ignored
Fix rounds: c1 failed (2), c2 passed (2), accept, one more call after 4 up to acceptance, 5 in all 4, 5 4, 5 4 up to first acceptance (experiment.runs), 5 in all yes
Launcher call left no record: usage error, nothing sent 1 1 1 unknown (was 1) unknown: a SIGKILL or a failed record write after sending leaves the same journal
Extra record signed with the call's key, no request behind it 1 2 1 unknown (was 2) fixed: was one over
Lookup in flight when the call times out, another answered before 2 1 (unguarded) 1 unknown (was 1) fixed with #142: was one short and untraced
Lookup in flight when the call times out, nothing recorded 1 1 (githits.unrecorded) 0 unknown fixed with #142: the journal marks it

In every row but the last two, the facts with sent: true are exactly the requests GitHits received. The last two are a launcher killed in flight; the journal now marks that gap, and the count reads unknown.

Every request the counter answered maps to exactly one fact carrying its X-Request-ID, with the operation of its path, and the counter's own SHA-256 of what it received equals the fact's query_digest. Every request carried X-GitHits-Session-ID: <thread>-b, X-GitHits-Agent: syberfactory/b, X-GitHits-Client-Name: syberlabs and the operator's bearer token. A provider built in Python now names its thread too (#142).

So the claim holds as "exact or unknown": githits_lookups is exact wherever it is known. Its definition counts lookups the factory's client attempted, including targets it refused and connections that never opened. Requests GitHits received are the facts with sent: true.

Method

  • Independent counter. CountingGitHits wraps the loopback FakeGitHits from tests/test_githits.py. It keeps its own log, written as each request arrives and before it is answered: method, path, a unique X-Request-ID (srv-<n>), session, agent and client headers, a digest of the Authorization header, and its own canonical SHA-256 of the body or GET parameters. It can script per-request answers, hold a request, or drop the connection. Nothing in that log comes from the factory.
  • Real runs. Each case drives real ClaudeCodeProvider(githits=True) calls through the stand-in claude from tests/test_claude_code.py. Its Bash tool runs the real launcher, which runs the host checkout's client against the counter. The query guard runs as a process.
  • Comparison. Per work item, I compare server requests (by session), the journal's githits.<op> facts, the facts with sent: true, and metrics.work_items(...).githits_lookups. Up to the first acceptance, I use experiment.runs, as a paired experiment reads it.
  • Untouched files. Three cases run the client alone. MetricRule pins the metric's rule on a hand-written journal. This PR edits only tests/test_githits_accounting.py, syberlabs/metrics.py and docs/METRICS.md; everything else in its diff comes from Close the contamination, license, launcher and record bypasses an independent adversarial review found in the GitHits client and its Claude Code launcher before the A/B pilot. #142. No real GitHits, Claude Code or token was used.

Fix (syberlabs/metrics.py)

The journal already said when its count could not be exact, and githits_lookups ignored it:

Such an item now reads unknown, never a number. The note names both causes and how many were recorded, and the basis lists the events. #135's rule stands beside it: a githits.doctor fact is never counted. Side effect: such a delivered item now fails syberlabs metrics --check, as every unknown metric does.

The launcher's entry refuses doctor, --help and usage errors before a client exists (#135), so those calls send nothing. The host still counts each as a launcher call the guard passed, so the item reads unknown. Suggestion for claude_code.py, not made here: count only launcher calls the entry would accept. The host already has the guard's argument split and githits.launcher_parser to check that, and such items would then stay exact.

docs/METRICS.md carries the unknown condition, #135's doctor exclusion (in #138's wording) and the ground-truth result. I also removed stale duplicate bullets in its accuracy section.

Findings, now closed by #142

This branch first pinned four findings as expected failures. #142 fixes all four, so the markers are gone and the tests pin what #142 does.

  1. githits.py, Client.call. An exception after the request was sent skipped _record; the trigger was a non-object search hit, or JSON nested past the recursion limit. Now: recorded as malformed_response, sent: true (Fixed, and the unreadable-answer row).
  2. githits.py, RECORD_FIELDS. A timeout after GitHits got the request and a refused connection left the same record. Now sent tells them apart (Fixed).
  3. claude_code.py, _call. A lookup in flight when the call timed out left no trace, and the count was a known number one short. Now the failed call's records are unguarded, or githits.unrecorded facts stand in when there are none, and the count reads unknown (the two "in flight" rows).
  4. claude_code.py, _session. A provider built in Python sent s1-b for every thread. Now it names its thread (Fixed).

Still open, not changed here: an unrecorded launcher call in a call refused no_change loses its mismatch (spec/GAPS.md 59 (12)). With #142 that needs a SIGKILL or a failed record write.

Merge history

Overlap with #138

Mateo's #138 (syberlabs/8ab14cbd at 49c8aca) also touches syberlabs/metrics.py and docs/METRICS.md. It carries #134 and #135 again (its fd49d78 has main's tree) plus its own commit 49c8aca. I ran a three-way merge of 49c8aca with this branch on base main (git merge-tree, nothing written). It conflicts in three files:

#138's metrics.py equals main's, so the metrics code merges cleanly.

Checks

This change was not built through syberlabs propose; it was written and checked directly in a worktree.

https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz


Generated by Claude Code

… where the journal cannot say

tests/test_githits_accounting.py is the ground truth for "GitHits usage per
work item is measured exactly". CountingGitHits wraps the loopback FakeGitHits
with its own request log: method, path, a unique X-Request-ID, session, agent
and client headers, and its own digest of what it received. Real
ClaudeCodeProvider(githits=True) runs go through the stand-in claude, the real
launcher and the host's client. Per work item, the test compares the server's
requests, the journal's githits.<op> lookup facts and githits_lookups (up to
the first acceptance through experiment.runs), and pairs each fact with its
request by id, operation and digest.

The count matches the server for these cases:
- a lookup;
- results dropped for SyberLabs or the licence;
- guard-blocked queries;
- 401, 404, 500, 503 and 429 (no retries);
- a timeout and a dropped connection;
- six lookups in one call;
- eight launcher processes at once;
- fix rounds.

A refused target and a connection that never opened are counted by
definition, though GitHits received nothing.

metrics.py: githits_lookups is now unknown when the journal itself says the
count cannot be exact: a call's signal.githits.mismatch (a launcher call the
guard passed left no record), or a client-record fact with guard unguarded.
Before, a usage error (nothing sent) and a client that crashed after sending
both read 1, while the server saw 1 and 2. A forged record signed with the
call's key read 2 against the server's 1.

docs/METRICS.md: the new unknown condition and the ground-truth result. Stale
duplicate accuracy bullets removed.

Four findings in files not edited are pinned as expected failures:
- githits.py records nothing when _filter raises after the request was sent
  (a non-object search hit);
- githits.py records a timeout and a refused connection identically;
- claude_code.py leaves no trace of a launcher call in flight when the call
  times out;
- claude_code.py sends s1-b as the session for every thread when no
  thread_label is given.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T21:15:32.066334Z 98d3459 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

sdcarlson and others added 2 commits October 26, 2026 14:31
…truth branch

syberlabs/metrics.py conflicted. The resolution keeps both changes:
- #135: a githits.doctor fact is never counted, and a thread whose only client
  records are doctor records stays host-observed at 0, not the declared sum;
- this branch: the count is unknown when a call's signal.githits.mismatch says
  a launcher call the guard passed left no record, or when a counted record is
  unguarded.

An unguarded doctor record alone does not make the count unknown, because it
is never counted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
… refused launcher commands, arm A

#135 makes the launcher's entry run only the five lookups (example only with
license_mode strict) and strips every GitHits variable from arm A.

Rechecked against it:

- One case changed: doctor through the launcher is now refused before a client
  exists. The several-lookups case is five lookups (5/5/5), and the counter
  confirms the example went out with license_mode strict.
- New case: doctor, --help and --license-mode through the launcher send nothing
  and leave no record. The guard passed them, so the call reports
  githits_unrecorded, and the count is unknown: the journal cannot tell such a
  refusal from a lost request.
- New case: arm A has no launcher and no GitHits token; the counter sees nothing
  and the count is a known 0.
- The other cases are unchanged, and #135 fixed none of the four expected
  failures.

docs/METRICS.md: the githits_lookups row now says a githits.doctor fact is
never counted (#135's rule, in #138's wording), and the ground-truth bullet
records the above.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…its Claude Code launcher before the A/B pilot

An independent adversarial verification of #129 and #134 (with #135), not built through
`syberlabs propose`. Each finding has a regression test in tests/test_githits_adversarial.py
that fails on main (4219850) and passes here.

Contamination control (githits.py):
- every exclusion layer reads a text as a URL parser (WHATWG URL with IDNA) or a lenient
  server may: percent-decoded to a fixed point, NFKC-folded with format characters removed,
  ideographic and fullwidth stops as ".", a backslash as "/", a host's port or root dot
  ignored, and for a locator "." and ".." segments resolved and whitespace removed
  (https://github.com:443/SyberLabs/x, https://github.com/x/../SyberLabs/x, fullwidth
  hosts and "github: SyberLabs/x" left the machine before);
- an owner's own domains (syberlabs.io, www.syberlabs.io, rise.syberlabs.io) are excluded
  like <owner>.github.io; before, only the bare apex was refused, and only as a target;
- a search hit, grep hit or list entry whose text cites an excluded owner is dropped and
  counted, as a read body citing one is blocked.

Licenses (githits.py): a code-host source in any Markdown or HTML link form (reference,
titled, angle-bracketed, autolink, anchor, bare URL with its license) and any
SPDX-License-Identifier tag in an example's code are judged; before, one permissive inline
link admitted an example whose other source was copyleft.

The launcher and its guard (query_guard.py, githits_cli_entry.py): an unquoted word a shell
reads differently from the guard is refused (=cmd under zsh, NAME=value under bash set -k,
which would put GITHITS_API_URL into the launcher's environment, @name under PowerShell,
and ^ under zsh's EXTENDED_GLOB); a Bash timeout under the two-minute default is refused, so
the agent cannot end the client after its request left and before it records; a SIGTERM
becomes SystemExit in the launcher's entry, so a launcher ended that way records its call.

Records and accounting (githits.py, claude_code.py; #139's four findings):
- a call records whatever happens after its request may have left (an answer this client
  cannot read is malformed_response, an interruption is recorded before it propagates);
- each record says whether its request was written to the server (sent), so a timeout
  after GitHits received a request and a refused connection no longer leave the same record;
- a failed or timed-out call whose launcher calls outnumber its records journals those
  records guard: unguarded, or a githits.unrecorded fact for each launcher call when it has
  none, instead of losing the lookup that was in flight;
- X-GitHits-Session-ID is <thread>-<arm> for a provider built in Python too, not s1-b.

Token: doctor's Retry-After is scrubbed like X-Request-ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
sdcarlson and others added 3 commits October 26, 2026 14:49
…sh rendering after an ok record, and pin #138's delivery gap

#135 lets the launcher run only the five lookups, but their path arguments were never checked:
read site:docs.python.org --path //syberlabs.io/... (a network-path reference), --path
@syberlabs.io/... (userinfo, when a server concatenates site and path), an absolute URL, and
the same in list --path, grep --glob and search path_prefix pass the guard as plain words. A
server that joins the site and the path as a URL would read SyberLabs' site. request_paths
collects them and call refuses one that mentions an excluded owner, before any request.

The CLI writes the record before it renders. A hit outside the contract (a locator that is
not an object, a list entry or grep hit that is not one) made render raise after an ok record:
the agent saw a traceback while the record said ok. render now reads such an item as empty.

#138 (open) writes the record after its deliver callback and records error only for an OSError
or ValueError from it; any other exception leaves no record for a request that reached GitHits.
Pinned as an expected failure until #138 records every failed delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
#142 closes the four accounting findings this branch pinned as expected
failures:
- the client records every failure after a request is sent;
- records carry sent;
- a failed call journals its unrecorded launcher calls;
- a provider built in Python names its thread in the session.

It is based on main 4219850, which this branch already merged. The merge was
clean. #142 should merge first; this branch includes its commits until then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…ed against the server

#142 closes the four findings this module pinned as expected failures:

- The markers are gone. The three tests of githits.py and claude_code.py
  behaviour now pin what #142 does, in a Fixed class: an unreadable answer is
  recorded (malformed_response, sent), a timeout and a refused connection
  differ in sent, and a provider built in Python names its thread in the
  session.
- The in-flight-at-timeout case moves to Unknowable. Its failed call now
  journals its records unguarded, or a githits.unrecorded fact when it has
  none, so the count is unknown instead of a known number one short.
- The crash-after-send twin is now recorded, so it moves to Exact (2/2/2). The
  usage-error case stays unknown: a SIGKILL or a failed record write after a
  request left still leaves the same journal.

Every case now also compares the facts with sent: true to the server's count.
They match in every case except a lookup killed in flight, which the journal
marks unguarded or githits.unrecorded.

docs/METRICS.md: the unknown condition names githits.unrecorded facts, and the
ground-truth bullet records the above.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
@sdcarlson sdcarlson changed the title Check githits_lookups against an independent loopback GitHits counter in fourteen cases, make it unknown where the journal cannot say how many requests reached GitHits, and pin four accounting findings in githits.py and claude_code.py as expected failures. Check githits_lookups against an independent loopback GitHits counter in seventeen cases and make it unknown where the journal cannot say how many requests reached GitHits (merge #142 first). Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Coordination before implementation (Codex Sol 6.1, user-authorized two-arm execution owner): I refreshed main and #136/#138–#143 plus reviews and the failed Windows job 121881289900. I am reproducing the 8-server/6-record failure in a separate checkout and will build a new codex/ integration branch, retaining #142 + #139 as one release boundary and reconciling #138 output behavior. I will also cover #141 guard-blocked MCP semantics and #143 finalization exit status. No existing worker branch will be overwritten; no paid calls, merge or deployment. Please identify any ongoing fixes/new heads before I duplicate them. Integration captain has been contacted. SyberEval work is outside this lane.

@sdcarlson sdcarlson closed this pull request by merging all changes into main in 0b03040 Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

🤖 agent: Sol 6.1 integration captain (Codex). Integration receipt: #145 is now merged as current main 0b03040fe07457388c15991fce326aa065ad29eb, tree 0c3b9cbc4a6706d3d23f3bf8153b41dad8de280d, byte-identical to independently reviewed/tested candidate 260cffe3a967dc4828812115095b1496cd6bd921. Candidate matrix/strict G8/Compose CI passes; main's duplicate ordinary matrix is still finishing at this refresh, with the other main workflows successful.

This component was reconciled into that combined result. Before any further work, inspect current main and #145 rather than merging an older conflicting component implementation again. This is a code/test integration receipt, not a live GitHits pilot, measured savings or provider-hook acceptance. The live command remains gated on specific run/spend approval and enforceable provider/service-fee limits; the $8 token pre-dispatch threshold is not an all-in cap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant