Skip to content

Fail GitHits lookups closed before the network unless a valid guard decision exists, and add a hooked API lane (api-hooked) so --githits only runs where the PreToolUse guard runs - #155

Merged
sykosyber merged 1 commit into
mainfrom
syberlabs/6c983f71
Oct 31, 2026

Conversation

@sykosyber

Copy link
Copy Markdown
Member

Objective: Fail GitHits lookups closed before the network unless a valid guard decision exists, and add a hooked API lane (api-hooked) so --githits only runs where the PreToolUse guard runs

Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested

Accepted in SyberLabs thread 6c983f71-04ce-469b-b9cf-5cdad1478205 as candidate c2 (commit c1211d08c8f270c2fd161bbc48cfd9bb6334024f).
Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.

… decision; add the api-hooked lane, refuse --githits with --bare, and a default GitHits deny-list

SyberLabs-Thread: 6c983f71-04ce-469b-b9cf-5cdad1478205
SyberLabs-Candidate: c2
SyberLabs-Provider: claude@opus-5.5
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T01:44:55.686893Z c1211d0 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sykosyber

Copy link
Copy Markdown
Member Author

Handoff: Build Thread 6c983f71 (repo-change.v5, class R, unattended, actor sykosyber) on base d0bf5380ce. Result c1211d08c8, 21 files.

What changed

  • The GitHits launcher fails closed before the network. When the query guard's hook passes a launcher command, its decision now also carries arguments_digest (the SHA-256 of the words it read), covered by the MAC. Before any request is paced or sent, the launcher's entry claims a decision that names its exact argv (query_guard.claim_launch), and each decision can be claimed once. If there is no such decision (no hook ran, no guard key, or a forged, mismatched or used decision), the call is recorded status: unguarded with sent: false. The launcher prints refused: unguarded: ... and exits 2. No HTTP request is made. The operator's syberlabs githits (lookups and doctor) has no gate.

  • A new lane, --lane api-hooked. It runs on the API key without --bare, so the hook runs. Each call gets a fresh, empty CLAUDE_CONFIG_DIR and HOME (plus USERPROFILE on Windows) inside its guard directory, and --disable-slash-commands. As on every lane, it also gets --restricted, --setting-sources user and --strict-mcp-config. ANTHROPIC_API_KEY is kept and required. Other credential, login and proxy variables are removed. ANTHROPIC_BASE_URL and the third-party switches are refused. The lane needs --trusted-tree and is recorded in signal.harness.lane.

  • --githits --bare is refused: Claude Code 2.1.312 runs no --settings hook under --bare. --lane api without --bare is refused and points to api-hooked. Both pilot arms run api-hooked, and their argv differ only in the GitHits additions. The pilot dry run, examples/githits_ab.py and docs/EXPERIMENTS.md now use this lane. The eval spec's lane enum accepts api-hooked, so the eval GitHits pilot test (tests/test_eval_adversarial.py) can run its treatment arm, which --bare now refuses.

  • Docs:

    • docs/CLAUDE_CODE.md answers the --bare hook question with the canary evidence and documents the new lane and the pre-send refusal.
    • docs/GITHITS.md now says that records carry sent and that search and grep answers carry X-RateLimit-Remaining (example and doctor answers do not). It also says paced_ms can be small on a call that was not held.
    • spec/GAPS.md 64 is appended.
  • The GitHits client has a default deny-list, from the pilot's pre-registration (githits.DEFAULT_EXCLUDE). On top of SyberLabs it covers:

    • the owner sykosyber;
    • pypi:syberwork (the distribution name in pyproject.toml; there is no npm package);
    • deepwiki.com and deepwiki.org, with their subdomains;
    • forks of SyberLabs repositories, detected by name under any owner on a code host (githits.FORK_NAMES).

    A matching target is refused before anything is sent. A result whose identity or text points at the list is dropped, and an example is dropped whole. SYBERLABS_GITHITS_EXCLUDE adds to the list and now also takes registry:name and host entries. The list is documented in docs/GITHITS.md.

  • From Make the GitHits A/B pilot ready to run: keep permissive-licence examples from GitHits' documented responses, pace example calls host-side to one per 40 s across processes and arms with the wait exclu #153's verifier:

    • Arm B now has SYBERLABS_GITHITS_PACE_DIR pinned to the host's resolved directory.
    • A fence or indented block inside a blockquote now counts as code.
    • The sources-heading parser now accepts singular forms, headings with extra words, and CRLF.

Evidence

  • Full suite on the scratch branch syberlabs/6c983f71-ci (2f140d5e33, same tree as the result) passed: test 3.11, 3.12 and 3.13, windows, G8 and compose. https://github.com/SyberLabs/SyberWork/actions/runs/41154422654
  • Bite (the new and changed tests against the unchanged code, 103218d54b) failed as intended: https://github.com/SyberLabs/SyberWork/actions/runs/41154426192
    • On 3.12: 1085 tests ran, with 32 failures and 61 errors, all in new or changed tests.
    • Behavioural failures:
      • test_githits: LauncherGuard, LauncherEntry, LauncherOutput, Exclusion (the deny-list) and DocumentedExamples (headings and quoted fences);
      • test_githits_skill, test_githits_pace, test_query_guard (arguments_digest);
      • test_eval_adversarial (the api-hooked arm).
    • Import failures: seven modules that import claude_code.HOOKED or the new test helper failed as whole modules.
  • Targeted tests, run locally on Windows: 346 ran and passed (3 skipped, 4 expected failures). The modules were test_query_guard, test_githits, test_githits_pace, test_githits_skill, test_claude_code, test_githits_accounting, test_githits_adversarial, test_experiment_githits_api, test_pilot_dry_run, test_eval_core, test_eval_adversarial, test_experiment and test_githits_engineering_smoke.
  • Conformance, run locally: matched 23 golden traces.
  • Scenarios, run locally: 11 replayed, exit 0, digest b41f7cbb2e70447c4e57bf7c253b976b81cd7ab1f95daf7b9c459564db1f44bb.
  • c1 (without the deny-list) was refused by the thread's tests check on this machine. The failures were two test_evolve tests and a shard timeout, all in code this change does not touch. test_evolve passed when re-run alone. c2 is the candidate.
  • Thread checks on c2 passed: conformance, critic.rules, scenarios and tests.
  • run --once: pass 1 found c2 acceptable. Pass 2 accepted it.
  • Certify: c2 was signed with the factory ed25519 key 4b6ae2deea. The chain head is 2f350846d1 at 18 events.
  • Verify: signature ok, repository ok (trees, 21 changed paths and diff digest match), journal ok (18 of 18 events certified). Verdict: pass.

Claim ceiling. Implemented and tested against the stand-in claude and a loopback fake GitHits only.

  • No real Claude Code ran on api-hooked. Its first live call must show apiKeySource: ANTHROPIC_API_KEY, and every lookup must read pass or a rule.
  • Live, the hook's words and the launcher's argv have to agree under Git Bash on Windows. If they don't, the call is refused (fail closed).
  • The flags come from 2.1.309's --help. 2.1.312's was not available on this machine.
  • WebSearch, WebFetch and --mcp-config lookups on a --bare lane are still caught only after the fact (GAPS 64 (3)).
  • The deny-list detects forks by name only (GAPS 64 (6)).

Publishing note: the first open_pull_request attempt was recorded effect_unknown (destination_error). Python's default certificate store on this machine failed TLS verification for api.github.com. recover settled it not_applied once the settle window passed, and the PR was then opened with SSL_CERT_FILE pointing at certifi's CA bundle for that one command. Verification stayed on.

No paid model or real GitHits API was called. The owner merges.

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c1211d08c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/EXPERIMENTS.md
2. The primary metric and the group the decision rests on (for example: the delivery rule on `needs_lookup` items, with `--delivered review`).
3. The exclusion rules: the nine excluded checks above, and nothing else.
4. The frozen setup: the SyberWork commit, Claude Code version, model, effort, `--allowed-tools`, `--max-turns`, the query guard's thresholds, the credential lane, and arm B's GitHits route (`--githits`) with its license policy (`SYBERLABS_GITHITS_LICENSES`, recorded as `signal.githits.license_policy`). Use the `api` lane (`--bare`) in both arms: it is the lane whose cost is billed and known, and in bare mode the tree's `CLAUDE.md` is not read in either arm. The contract prices every model the runs report (`evolution.budget.prices`); a run that used a model the contract does not price has an unknown cost. Anything that learns from runs stays off for the week. AutoSkill is a design (`docs/AUTOSKILL.md`) and is in no proposer path; the `jev` and `empirical` branch policies belong to `--evolve`, which the pilot does not use.
4. The frozen setup: the SyberWork commit, Claude Code version, model, effort, `--allowed-tools`, `--max-turns`, the query guard's thresholds, the credential lane, and arm B's GitHits route (`--githits`) with its license policy (`SYBERLABS_GITHITS_LICENSES`, recorded as `signal.githits.license_policy`). Use the `api-hooked` lane (`--lane api-hooked --trusted-tree`) in both arms: the API key, so the cost is billed and known, without `--bare`, because Claude Code 2.1.312 runs no `--settings` hook under `--bare` and the provider refuses `--githits` there (`docs/CLAUDE_CODE.md`, the hooked API lane); each call gets an empty config directory and home. Not in bare mode, the tree's `CLAUDE.md` is read, equally in both arms: record it with the conditions. The contract prices every model the runs report (`evolution.budget.prices`); a run that used a model the contract does not price has an unknown cost. Anything that learns from runs stays off for the week. AutoSkill is a design (`docs/AUTOSKILL.md`) and is in no proposer path; the `jev` and `empirical` branch policies belong to `--evolve`, which the pilot does not use.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the pilot commands to use the hooked API lane

The following “Each item, each arm” commands still use --bare, and arm B combines it with --githits. This commit makes --githits with --bare fail with invalid_provider, so readers following the pilot instructions cannot run it; update both commands to use --lane api-hooked --trusted-tree.

Useful? React with 👍 / 👎.

@sykosyber

Copy link
Copy Markdown
Member Author

Independent verification of #155 (guard-lane, thread 6c983f71): pass with findings

Head c1211d08c8, base d0bf5380ce (merge-base equals main). This was a read-only review in a fresh clone on Windows 11 with Python 3.12. No paid model and no real GitHits were used.

What I ran

  • Unit tests. test_githits, test_githits_pace, test_githits_adversarial, test_githits_accounting, test_githits_skill, test_query_guard, test_claude_code, test_experiment_githits_api, test_pilot_dry_run, test_eval_adversarial, test_gaps and test_metrics: 310 tests OK (3 skipped, 4 expected failures). I also re-ran test_githits.LauncherGuard, test_githits.Exclusion and test_gaps on their own: 16 tests OK.
  • python -m conformance.run: 23 golden traces matched, exit 0.
  • python benchmarks/run_scenarios.py: 11 replayed, exit 0, digest b41f7cbb….
  • gh pr checks 155: 25 of 25 pass, including admit, windows and G8.
  • syberlabs verify --public guard-lane-6c983f71.cert.pub --repo .: signature ok (ed25519 4b6ae2deea, candidate c2, result c1211d0, repo-change.v5, class R). Repository ok: trees match, and the 21 paths and the diff digest match. The journal was not checked because it is not in the clone.

Claims confirmed

  1. Fail closed before the network.
    • githits.run calls gate before pace and before Client.call (syberlabs/githits.py ~L1454).
    • claim_launch (syberlabs/query_guard.py:555-591) requires all four conditions: rule: pass, a shell tool, arguments_digest equal to the SHA-256 of the argv, and a MAC that verifies under SYBERLABS_GUARD_KEY.
    • Each decision is claimed once, through an O_CREAT|O_EXCL file in launches/.
    • A refused call is recorded with Client.refuse: unguarded, sent:false, exit 2, and the fake server sees no request.
    • Exception path: I found no route to _send without a claim. An OSError becomes claim_failed. Any other exception escapes githits.run (it catches only GitHitsError), so the entry exits 1 with nothing sent and no record, and the host flags githits_unrecorded. That still fails closed.
    • Environment: the only inputs are the key and the launcher path, both set by the host. The model cannot set them because NAME= words are refused.
    • Digest equivalence: the hook digests the words from bash_arguments. The launcher digests sys.argv[1:] as passed through exec … "$@". These agree under a POSIX shell, and test_the_hooks_decision_for_a_passed_launcher_command_is_what_the_launcher_claims covers it end to end, though without a real shell. The Windows exception is finding L1.
    • Replay: cross-call replay is impossible because the key and guard directory are per call. Two concurrent identical calls each claim their own decision.
    • Operator CLI: syberlabs githits and doctor have no gate. The model cannot reach them: the hook passes only the launcher's absolute path, and --allowedTools names only Bash(<launcher>:*).
  2. The api-hooked lane (syberlabs/claude_code.py: refusals at 662-689, environment at 795-814, isolation at 816-829, argv at 780-786).
    • No --bare, plus --disable-slash-commands, --restricted, --setting-sources user and --strict-mcp-config.
    • A fresh, empty config directory and HOME (and USERPROFILE on Windows) per call, inside the guard directory, removed after the call (tested).
    • ANTHROPIC_API_KEY is required and kept. The other login-lane variables and CLAUDE_CODE_OAUTH_TOKEN are dropped.
    • ANTHROPIC_BASE_URL and the third-party switches are refused at build and again per call.
    • --trusted-tree is required.
    • All three refusals hold: --githits with --bare, --bare with api-hooked, and non-bare --lane api.
    • The dry run enforces a single lane for both arms (scripts/pilot_dry_run.py, LANE).
  3. Deny-list.
    • DEFAULT_EXCLUDE and FORK_NAMES match the description.
    • The original owner regex is unchanged. REPLACE=1 now also turns off the fork rule (tests only, and documented).
    • The fork rule needs a code-host locator <owner>/<name> with an exact name, so the risk of over-dropping looks low. I did not search GitHub for public repos with these names, because the API budget is limited. Note that . is allowed after the name, so x/syberlabs.github.io would also be dropped.
  4. Items from Make the GitHits A/B pilot ready to run: keep permissive-licence examples from GitHits' documented responses, pace example calls host-side to one per 40 s across processes and arms with the wait exclu #153. Each has a test:
    • SYBERLABS_GITHITS_PACE_DIR pinned: tests/test_claude_code.py:1080-1082 and :1296.
    • A fence inside a blockquote counts as code: test_a_fenced_block_inside_a_blockquote_is_code.
    • Singular headings, headings with extra words, and CRLF: test_a_singular_heading_a_heading_with_more_words_and_crlf_still_open_a_sources_section.
  5. Scope. syberlabs/eval/spec.py:56 only adds api-hooked to the LANES enum. That is needed because test_eval_adversarial's claude-code arm now runs api-hooked (--githits refuses --bare). It is harmless: the enum only widens.
  6. Docs and GAPS.
    • CLAUDE_CODE.md:129 answers the --bare question in line with the canary.
    • GITHITS.md matches the code.
    • GAPS 64 is appended with no renumbering, and test_gaps passes.

Findings

Medium

  • M1. Other lookups on a --bare lane still have no pre-send gate, and the docs still show those invocations as plain usage.
    • claude_code.py:671 refuses only --githits with --bare.
    • --bare --mcp-config githits.json (mcp__githits__*) and --bare with WebSearch/WebFetch still send queries with no hook running. The host catches them only after the fact (provider_guard_inactive), which is the same class of failure the canary showed.
    • It is disclosed in GAPS 64 (3) and in CLAUDE_CODE.md:129.
    • However, docs/CLAUDE_CODE.md:7 and the "GitHits A/B through MCP" block (:187-189) still present --bare --mcp-config githits.json and --bare … WebSearch,WebFetch,mcp__githits as usage, with no caveat at that point.
    • Suggestion: refuse --bare when --mcp-config names a server or the tools include WebSearch/WebFetch (same evidence), or at least add the caveat at those lines. This does not block the pilot, which uses --githits on api-hooked.

Low

  • L1. Windows Git Bash rewrites arguments that start with /, which refuses honest launcher calls.
    • Reproduced locally: python.exe -c "…print(sys.argv)" grep /api/v1 prints ['grep', 'C:/Program Files/Git/api/v1'].
    • The hook digests /api/v1, so the digests differ and the call is refused. It fails closed and is disclosed in GAPS 64 (2).
    • Possible fix: export MSYS_NO_PATHCONV=1 and MSYS2_ARG_CONV_EXCL='*' in launcher_text (claude_code.py:320-322) before exec.
    • The pilot VM runs Linux, so this does not block.
  • L2. syberlabs/eval/arms.py:74 does not repeat the new provider refusals (bare with api-hooked, non-bare api, githits with bare). Such a spec passes validation and fails only when the provider is built.
  • L3. The githits_environment docstring at claude_code.py:836 still says SYBERLABS_GITHITS_EXCLUDE "only adds to SyberLabs". It now adds to the whole default deny-list.

Info (what loads on api-hooked; read from docs and code, not verified live)

  • Kept out:
    • User settings, memory, skills, plugins, login and apiKeyHelper (empty config directory and HOME).
    • The worktree's .claude/settings.json hooks (--setting-sources user plus --restricted).
    • .mcp.json (--strict-mcp-config).
    • Project skills and commands (--disable-slash-commands).
    • Project subagents, because no Task tool is given.
  • Still loads:
    • Managed (policy) settings, and the system-path managed or enterprise CLAUDE.md, if the machine has them.
    • The worktree's CLAUDE.md, CLAUDE.local.md and .claude/rules, and any CLAUDE.md in the temporary worktree's parent directories.
    • The Bash tool's login shell still reads /etc/profile.
  • Unclear: whether the CLI's --setting-sources user also suppresses the project CLAUDE.md. In the Agent SDK it loads only with the project source. The PR assumes it is read, which is the conservative choice. The first live call should record which happened.
  • Pilot design: all of this is the same in both arms. But if a pilot item targets this repository, its CLAUDE.md describes a GitHits tool that only arm B can use. Record that among the conditions.

Interplay with #152

Verdict: pass with findings. None of them blocks the claims this PR makes. M1 is the one to schedule next.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant