Skip to content

Close the follow-ups from PR #155's verifier: refuse unscreened --bare lookups (MCP servers, WebSearch, WebFetch) unless an explicit recorded override is given, stop Git Bash rewriting launcher argume - #162

Merged
sykosyber merged 1 commit into
mainfrom
syberlabs/b4b760bd
Nov 2, 2026

Conversation

@sykosyber

Copy link
Copy Markdown
Member

Objective: Close the follow-ups from PR #155's verifier: refuse unscreened --bare lookups (MCP servers, WebSearch, WebFetch) unless an explicit recorded override is given, stop Git Bash rewriting launcher arguments, mirror the provider's lane refusals in eval spec validation, and fix the stale docstring and skill digest in the docs. Also make the cross-process GitHits pace test robust on slow runners by asserting on recorded stamps instead of wall-clock arrival.

Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested

Accepted in SyberLabs thread b4b760bd-a5dc-4ca5-abfc-db9772103579 as candidate c1 (commit 14c13da728efb91ce8c0eb302abc7a9cae1e4c56).
Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.

…ash rewriting launcher arguments, mirror lane refusals in eval spec validation, fix docs digest and a flaky pace test

SyberLabs-Thread: b4b760bd-a5dc-4ca5-abfc-db9772103579
SyberLabs-Candidate: c1
SyberLabs-Provider: claude@opus-5.5
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T08:35:12.791718Z 14c13da PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sykosyber

Copy link
Copy Markdown
Member Author

Handoff: packet guard-lane-2, Build Thread b4b760bd, candidate c1, head 14c13da728efb91ce8c0eb302abc7a9cae1e4c56 on base dc69499491 (#156). Contract repo-change.v5, Class R, unattended runner, actor sykosyber. 10 files: syberlabs/claude_code.py, syberlabs/cli.py, syberlabs/eval/arms.py, docs/CLAUDE_CODE.md, docs/EXPERIMENTS.md, spec/GAPS.md (entry 65), tests/test_claude_code.py, tests/test_eval_core.py, tests/test_experiment.py, tests/test_githits_pace.py.

What changed. ClaudeCodeProvider refuses --bare when WebSearch/WebFetch are allowed or --mcp-config names an MCP server, unless an explicit override is given; the override is recorded. The launcher exports MSYS_NO_PATHCONV=1 and MSYS2_ARG_CONV_EXCL='*' so Git Bash stops rewriting its arguments. eval/arms.py mirrors the provider's lane refusals at spec validation. The stale docstring and the skill digest in docs/CLAUDE_CODE.md are fixed (digest 59e25c01…). The cross-process GitHits pace test asserts on recorded slot stamps instead of wall-clock arrival.

Scratch CI on the exact trees (branches pushed only for this, not PR heads).

  • Full suite on base + c1 (branch syberlabs/b4b760bd-ci, d5415a95, tree identical to this PR's head): https://github.com/SyberLabs/SyberWork/actions/runs/41410541606 — success on all 7 jobs (test 3.11/3.12/3.13, windows, compose, G8 Existing suite, G8 strict 3.12).
  • Bite: base + the 4 changed test files only (branch syberlabs/b4b760bd-bite, 85b90fcf): https://github.com/SyberLabs/SyberWork/actions/runs/41410546222 — failure, identical on every job (Ran 1085 tests, failures=6, errors=7): 6 failures are the new test_eval_core.Arms.test_claude_code_lane_refusals_are_the_providers_at_spec_validation subtests; the 7 errors are one root, tests/test_claude_code.py:37 importing web_tools from syberlabs.claude_code (added by c1), which takes down test_claude_code, test_experiment and the five modules that import from test_claude_code (test_pilot_dry_run, test_githits_engineering_smoke, test_githits_adversarial, test_githits_accounting, test_experiment_githits_api). Nothing outside the new or changed tests fails.

Host evaluation (unattended syberlabs run --once, twice). Pass 1: check c1 -> acceptable, checks conformance passed, critic.rules passed, scenarios passed, tests passed. Pass 2: accept c1 -> succeeded; target refs/heads/syberlabs/b4b760bd moved to 14c13da728. Journal: 13 events at acceptance, chain verified. Re-run on the accepted tree in the clone: python -m conformance.run → matched 23 golden traces, exit 0; benchmarks/run_scenarios.py → scenarios: 11 replayed, exit 0, digest b41f7cbb2e70447c4e57bf7c253b976b81cd7ab1f95daf7b9c459564db1f44bb.

Certificate. syberlabs certify c1 signed with the rotated factory key (ed25519 public prefix 3312866a18, seed from $SYBERLABS_CERTIFICATE_SEED, chain head be7fc2bef7 at 13 events). syberlabs verify --public factory-key-current.pub --repo . → signature ok, repository ok (base and result present, trees match, 10 changed paths and diff digest match), journal ok (13 of 13 events certified), verdict pass. Receipts: guard-lane-2-b4b760bd.cert.json, .cert.pub, and the syberlabs export record guard-lane-2-b4b760bd.json, held in the officer's scratch receipts folder.

Publishing. publish push_branch and publish open_pull_request ran under the thread; no recover was needed. Spend $0.00; no paid model and no GitHits call in this publishing session. Not done, by the rules of this packet: the PR body is composed by the host (publish.py) and is not edited, so it carries the organization's attribution fields rather than a tool trailer; the accepted commit is the host's registration commit and is not amended.

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 14c13da728

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


def example(request):
arrivals.append(time.time())
slots.append(githits_pace._last(stamp_file)) # the slot the arriving request took (the latest stamp)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Capture each request's slot separately

If a process is paused after wait() stamps its slot and the next process takes another slot before the first request arrives, _last() reads the newer shared stamp for both requests. The assertion that the recorded slots are at least one interval apart then fails on a slow runner, so this test can remain flaky; record each caller's own slot instead of rereading the shared latest stamp.

Useful? React with 👍 / 👎.

@sykosyber

Copy link
Copy Markdown
Member Author

Independent verification of #162 (guard-lane-2, thread b4b760bd): pass with findings

Head 14c13da728, base dc69499491 (merge-base equals the main it was built on). Read-only review in a fresh clone on Windows 11 with Python 3.12 (Git Bash 5.2.37 as sh). No paid model and no real GitHits were used; no new exploits were written.

What I ran

  • The nine requested modules (test_claude_code, test_experiment, test_eval_core, test_githits_pace, test_pilot_dry_run, test_githits_adversarial, test_githits_accounting, test_experiment_githits_api, test_gaps): 225 tests OK (2 skipped), re-run once more in a detached worktree at 14c13da: OK.
  • test_githits_pace.CrossProcess.test_two_concurrent_launcher_examples_are_spaced_recorded_and_other_ops_unpaced on its own, 5 of 5 runs OK (2.67–2.72 s each).
  • test_the_launcher_hands_words_starting_with_a_slash_over_unchanged and UnscreenedBare under the real Git Bash sh: OK.
  • python -m conformance.run: 23 golden traces matched, exit 0. python benchmarks/run_scenarios.py: 11 replayed, exit 0, digest b41f7cbb… (unchanged from Fail GitHits lookups closed before the network unless a valid guard decision exists, and add a hooked API lane (api-hooked) so --githits only runs where the PreToolUse guard runs #155).
  • gh pr checks 162: 25 of 25 pass (admit, windows, G8 included).
  • syberlabs verify --public factory-key-current.pub --repo . on guard-lane-2-b4b760bd.cert.json: signature ok (ed25519 3312866a18, thread b4b760bd, candidate c1, result 14c13da728, repo-change.v5, class R); repository ok (trees match, 10 changed paths and diff digest match). Journal skipped (not in the clone).

Claims confirmed

  1. M1, unscreened --bare lookups refused. claude_code.py:752-761: with bare and no override, web_tools(tools) (:282-286: WEB_TOOLS names, case-insensitive, specifier dropped via built_in_tools; websearch, WebFetch(domain:…) and the server names web_search/web_fetch all count) or --mcp-config with names_mcp_server (an unreadable or non-JSON file counts) raises invalid_provider pointing to --lane api-hooked --trusted-tree. Tested in tests/test_claude_code.py:972-1047 (eight refused shapes, three accepted, the hooked counterpart) and from the terminal (:1673-1693).
    • The override is not reachable by the model. allow_unscreened_bare is a constructor keyword only: set by the CLI flag (cli.py:179-183, the operator) and by nothing else. eval/arms.build (arms.py:189-200) does not pass it, ARM_SCHEMA has additionalProperties: false and no such key, and CLAUDE_ONLY (arms.py:57) does not list it, so an eval spec cannot set it; no environment variable reads it. It never reaches Claude Code's argv: command() (claude_code.py:806-823) emits no --allow-* flag, and the test asserts --allow-unscreened-bare is absent from the stand-in's argv while signal.guard.allow_unscreened_bare is True (:1021-1022).
    • The docs scan test does catch a documented bare invocation. test_no_documented_bare_invocation_runs_unscreened_lookups_without_the_override (:1026-1047) joins \ continuations, strips trailing # comments, and requires the override on every syberlabs propose --claude-code … --bare line with --mcp-config or WebSearch/WebFetch, with a seen > 3 floor. Mutation check: restoring the old docs/CLAUDE_CODE.md:7 (--bare --mcp-config githits.json) makes it fail with the expected assertion. See L2 below for its scope.
  2. L1, Git Bash argument conversion. launcher_text (claude_code.py:343) emits export MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*' before exec. Reproduced on this host: without the export, sh -c 'python … grep /api/v1 /usr/lib' yields ['grep', 'C:/Program Files/Git/api/v1', 'C:/Program Files/Git/usr/lib']; with the launcher's export line, ['grep', '/api/v1', '/usr/lib']. With an outer environment of MSYS_NO_PATHCONV=0 MSYS2_ARG_CONV_EXCL= the launcher's own export still wins (/api/v1 unchanged), so the model's environment cannot re-enable the conversion. Under a POSIX sh the two are ordinary exported variables that nothing reads; the Linux CI matrix passes with them.
  3. L2, eval spec refusals. arms.claude_code_lane (arms.py:85-103) refuses the same set as the provider (claude_code.py:682-694 plus :752-761): bare with a login lane, bare with api-hooked, api without bare, githits with bare, bare with a web tool, bare with mcp_config. test_claude_code_lane_refusals_are_the_providers_at_spec_validation builds each refused arm with the provider too and checks the invalid_provider code; three allowed shapes return [].
  4. L3. The githits_environment docstring (claude_code.py:866-867) now says SYBERLABS_GITHITS_EXCLUDE adds to githits.DEFAULT_EXCLUDE and never replaces it.
  5. Pace test (tests/test_githits_pace.py:132-168): each fake-server request records githits_pace._last(stamp_file) on arrival; the test asserts both slots exist, second - first >= INTERVAL, each arrival is at or after its slot, and the last stamp is the second slot. If the first request arrived after the second slot was stamped, both would read the same stamp and second - first >= INTERVAL would fail, so the test still proves the first request left within one interval of its slot and the second only after a slot a whole interval later. The paced_ms assertions (held[1] >= 1000, held[0] < 1000) and the four unpaced operations (no paced_ms, stamp file unchanged under a 600 s interval) are unchanged.
  6. Docs. docs/CLAUDE_CODE.md:7 and the MCP A/B block (:193-196) use --lane api-hooked --trusted-tree; :16-19 and :198 state the refusal and override; :134 and :151 are updated; the skill digest at :184 reads 59e25c01…a409, and I recompute sha256(SKILL.md, CRLF→LF) = 59e25c0197550bb83e5ecf98d5b08ca6631842cb2c521f305f02c1a373b3a409 (the file has no CRLF). docs/EXPERIMENTS.md:148,153 use api-hooked. spec/GAPS.md:129 appends entry 65 with no renumbering; test_gaps passes.
  7. Pilot freeze. The PR touches none of syberlabs/githits.py, query_guard.py, the metrics table, run_shards, the settlement poll, the prompt builder or the metric service. The only change adjacent to the GitHits client is the launcher script text (two exported variables, which reach the client's process environment and are read by nothing in it); the request path itself is untouched.

Findings

Low

  • L1. arms.claude_code_lane is stricter than the provider for an mcp_config naming no server (arms.py:100-102 refuses any mcp_config on bare; claude_code.py:753-754 builds one whose mcpServers is empty). Disclosed in GAPS 65 (2). Acceptable: the spec has no override, an arm has no use for a server-less mcp_config, and the direction is fail-closed.
  • L2. The docs scan test's scope. It reads docs/*.md and the claude_code module docstring, and only lines that start exactly syberlabs propose --claude-code. A $ -prefixed line, a reordered syberlabs propose --bare --claude-code, or a line in README.md, AGENTS.md, cli.py's docstring or scripts/ would not be checked. Today none exists (the only --bare propose lines outside docs/ are cli.py:7, plain --bare, and GAPS prose), so this is a maintenance note, not a gap.
  • L3. The pace test no longer bounds the arrivals' spacing. A first request whose client setup and transit approached a whole interval would still pass. That is the stated intent (the guarantee is the slots), and the paced_ms assertions still show the second call was held, so I accept it; the lower bound on arrival spacing is now second_slot - first_arrival > 0 rather than INTERVAL - 0.25.

Info

  • web_tools refuses the server-tool names web_search/web_fetch only when they are listed in --allowed-tools; a server tool Claude Code invokes for itself without being listed is not refused under --bare. Disclosed in GAPS 65 (4).
  • My first local suite run overlapped a trial-merge checkout in the same clone, so the result I cite is the clean re-run in a detached worktree at 14c13da.

Interplay with #161 (now merged as 78724d8fb2)

Verdict: pass with findings. Nothing found blocks the claims this PR makes; the findings are scope notes for later maintenance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant