diff --git a/docs/CLAUDE_CODE.md b/docs/CLAUDE_CODE.md index 089af68..d1313c5 100644 --- a/docs/CLAUDE_CODE.md +++ b/docs/CLAUDE_CODE.md @@ -6,9 +6,13 @@ syberlabs propose --claude-code --bare # ANTHROPIC_AP syberlabs propose --claude-code --bare --max-calls 1 # one shot instead of the repair loop syberlabs propose --claude-code --bare --mcp-config githits.json --allowed-tools Read,Edit,Write,Glob,Grep,mcp__githits syberlabs propose --claude-code --bare --guard-max-lines 3 --guard-max-chars 400 # the query guard's thresholds (defaults) -syberlabs propose --claude-code --bare --githits # GitHits through the API client (arm B of the pilot) +syberlabs propose --claude-code --lane api-hooked --trusted-tree # ANTHROPIC_API_KEY without --bare: the guard's hook runs +syberlabs propose --claude-code --lane api-hooked --trusted-tree --githits # GitHits through the API client (arm B) ``` +`--githits` refuses `--bare`: Claude Code 2.1.312 does not run `--settings` hooks under `--bare` (below, the query +guard), so the treatment runs only on a lane where its guard runs, `api-hooked` for an API key. + `syberlabs.claude_code.ClaudeCodeProvider` is the repair loop (`providers.RepairProvider`, the factory's default proposer path) with one call replaced. Each call: 1. checks out the parent, the thread's base on the first call and the failed candidate on a repair, in a detached temporary worktree (`gitspace.Repo.worktree`; the developer's working tree is never touched); @@ -44,7 +48,8 @@ claude -p [--bare] --output-format stream-json --verbose --permission-mode dontA | `--append-system-prompt` | One fixed harness note: edit files in the current directory, the host reads the change with git, do not commit. The task prompt itself is the adapter's. | | `--settings FILE` | The query guard (below): a temporary settings document the host writes per call, in its own directory outside the worktree, holding one `PreToolUse` hook and nothing else. `--settings` sits above user, project and local settings and below managed settings (`code.claude.com/docs/en/settings`, read 2026-10-23); `--setting-sources` names only `user`, `project` and `local`, so it does not filter it. | | `--max-turns N` | Optional cap on agentic turns (listed in the CLI reference; not in 2.1.309's `--help` text). A run that reaches it still yields whatever it changed. | -| `--bare` | The API-key lane: skips hooks, plugins, MCP auto-discovery, auto memory and `CLAUDE.md`, and never reads a subscription login or the keychain. | +| `--bare` | The `api` lane: skips hooks, plugins, MCP auto-discovery, auto memory and `CLAUDE.md`, and never reads a subscription login or the keychain. On 2.1.312 that includes the `--settings` hook (live, 2026-10-30), so a `--bare` call runs no query guard; `--githits` refuses it. | +| `--disable-slash-commands` | `api-hooked` only: "Disable all skills and commands" (2.1.309's `--help`). With the empty config directory and home below, no user skill, command or plugin loads. | | `--mcp-config FILE` | One more MCP server, for example GitHits. | With `--githits` (below) three items differ and nothing else: `--tools` gains `Bash`, `--allowedTools` gains @@ -61,12 +66,15 @@ Every candidate this provider registers carries `signal.lane` and `signal.cost_b | Lane | When | `cost_basis` | Cost in the record | |---|---|---|---| | `api` | `--bare` (default there) | `billed` | Tokens × the contract's `budget.prices`, through the host's meter, so the dollar budget applies | +| `api-hooked` | `--lane api-hooked`, never with `--bare` (needs `--trusted-tree`) | `billed` | As `api`: the same key, priced the same way | | `provider_3p` | `--bare --lane provider_3p` (Bedrock, Vertex or Foundry credentials) | `estimated` | Priced the same way; the bill is the cloud provider's | | `subscription_local` | No `--bare` (default there): an inherited login | `none` | Not priced: there is no per-call bill; the dollar budget does not apply | | `seat_team_enterprise` | `--lane seat_team_enterprise` | `none` | As above | Claude Code's own `total_cost_usd` is recorded as `signal.estimated_usd`. It is a client-side estimate and is never used as billed spend. On a lane without a per-call bill, `syberlabs metrics` keeps cost `unknown` and says why; turns, lookups and the other metrics are still counted. `--bare` with `subscription_local` or `seat_team_enterprise` is refused (bare mode never reads a login). On those two lanes the provider removes from Claude Code's environment every variable in `claude_code.LOGIN_LANE_DROPPED_ENV`: each one Claude Code's environment reference (`code.claude.com/docs/en/env-vars`, read 2026-10-23) documents as an Anthropic credential or endpoint other than the login (`ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL`, `ANTHROPIC_CUSTOM_HEADERS`, workload-identity and mTLS settings) or as a third-party switch, endpoint, project, region or credential (`CLAUDE_CODE_USE_BEDROCK`, `CLAUDE_CODE_USE_VERTEX`, `CLAUDE_CODE_USE_FOUNDRY`, `CLAUDE_CODE_USE_MANTLE`, `CLAUDE_CODE_USE_ANTHROPIC_AWS`, `CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST`, `CLAUDE_CODE_SKIP_*_AUTH`, `ANTHROPIC_BEDROCK_*`, `ANTHROPIC_VERTEX_*`, `ANTHROPIC_FOUNDRY_*`, `ANTHROPIC_AWS_*`, `CLOUD_ML_REGION`, `VERTEX_REGION_*`), and the cloud SDKs' `AWS_*`, `GOOGLE_*`, `GCLOUD_*` and `AZURE_*`; `CLAUDE_CONFIG_DIR` (the page: "Override the configuration directory ... All settings, session history, and plugins are stored under this path", so it can point at another login or `apiKeyHelper`); and the proxy variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` (on the page) and `ALL_PROXY` (not on the page; common HTTP clients read it), each also in lower case, which do not bill but change where requests and lookups go. `CLAUDE_CODE_OAUTH_TOKEN` stays: it is the login. So a run labelled subscription cannot be billed by a key or routed to a billed provider. On `--bare`, a set `ANTHROPIC_BASE_URL` or third-party switch (`claude_code.THIRD_PARTY_SWITCHES`; `0`, `false`, `no` and `off` count as unset) means the run does not go to Anthropic's API under the operator's key: the lane must be `provider_3p`, and `api` is refused when the provider is built (`invalid_provider`) and again at each call (`provider_lane_mismatch`) in case the environment changed. The provider never passes `SYBERLABS_CERTIFICATE_SEED` on any lane. Otherwise Claude Code inherits the caller's environment, as every provider command does. +**The hooked API lane (`api-hooked`).** The operator's `ANTHROPIC_API_KEY` with Claude Code *not* in bare mode, so the query guard's `--settings` hook runs (the live canary: without `--bare` the hook ran, with `apiKeySource: ANTHROPIC_API_KEY`). Bare mode also kept the machine's own configuration out; this lane keeps it out another way, per call: a fresh, empty `CLAUDE_CONFIG_DIR` (the env-vars page: "All settings, session history, and plugins are stored under this path") and `HOME` (and `USERPROFILE` on Windows, where Node reads the home from it), both made in the call's guard directory outside the worktree and removed after the call, so no user settings, memory, skills, plugins, login or `apiKeyHelper` exist; `--disable-slash-commands`; and, as on every lane, `--restricted` (user, project and local settings files ignored, "managed settings and --settings still apply"), `--setting-sources user` and `--strict-mcp-config`. `--setting-sources` offers only `user`, `project` and `local` (2.1.309's `--help`: "Comma-separated list of setting sources to load (user, project, local)"); it has no value that names `--settings` alone, so the restriction to the provider's `--settings` comes from `--restricted` plus the empty config directory, and managed (policy) settings still load, as they do on every lane. `ANTHROPIC_API_KEY` is kept and required (the provider refuses the lane without it); every other variable in `LOGIN_LANE_DROPPED_ENV` and `CLAUDE_CODE_OAUTH_TOKEN` are removed, so the call cannot fall back to a login or another endpoint; `ANTHROPIC_BASE_URL` or a third-party switch is refused when the provider is built and at each call, as on `api`. A non-bare call reads the worktree's memory files, so the lane needs `--trusted-tree` and the repair-time instruction-file check applies (Trust, below). The lane is recorded as `signal.lane` and `signal.harness.lane`. `--lane api` without `--bare` is refused and points here. Not verified live: the empty config directory with a real Claude Code (the canary ran non-bare with its default configuration); the first live call on this lane must show `apiKeySource: ANTHROPIC_API_KEY` in `signal.harness.auth_source` and a `pass` or a rule on every lookup. The flags were read from 2.1.309's `--help` (2026-10-23); 2.1.312's `--help` was not available on the machine that built this lane, and the canary's hook diagnosis ran 2.1.312 with `--restricted`, `--setting-sources user`, `--strict-mcp-config` and `--settings`. + ## Trust Without `--bare`, `claude -p` runs the hooks in the tree's `.claude/settings.json`, connects the servers in its `.mcp.json` and reads its memory files (`CLAUDE.md`, `CLAUDE.local.md`) as instructions, with no workspace trust dialog and no per-server prompt (headless page, above). The provider therefore **refuses to run non-bare unless the operator passes `--trusted-tree`**, and the refusal (`untrusted_tree`) says why. Trusting the tree means trusting the repository and every tree the model writes in it, because a repair call runs in the failed candidate's tree. Two flags narrow that in every mode: `--setting-sources user` (no project or local settings, so no project hooks) and `--strict-mcp-config` (no `.mcp.json` servers). Memory files are still read from the working directory in non-bare mode, and a repair call runs in the failed candidate's tree, so a model could write the next call's instructions. Before each repair call the provider therefore compares the parent candidate's changed paths against the thread's base, and refuses the call when any is an instruction file: `CLAUDE.md`, `CLAUDE.local.md`, `AGENTS.md` or `.mcp.json` at any depth, a path named `.claude` itself (a file, or a symbolic link standing in for the directory) or anything under a `.claude/` directory (settings, rules, skills, agents, commands), compared case-insensitively, or a symbolic link (git mode `120000`) whose target, resolved against the link's directory, is one of those or lies under a `.claude` directory (`claude_code.instruction_link`). Not caught: an `@path` import in an unchanged `CLAUDE.md` that names a file the model edited (the import's target is an ordinary path; the memory file that imports it did not change), and a link to a directory that merely contains a `CLAUDE.md`. The refusal is journaled as `proposal_refused` with reason `instruction_files_changed` and the search ends with that error; no call is made on that parent. The base's own files are the operator's and are allowed, so the first call is never refused. The check runs in bare mode too (bare mode skips memory discovery, but skills still resolve), which means an objective that edits these files gets one shot, not a repair loop. `--bare` needs no declaration. None of this is a sandbox: Claude Code runs with the operator's permissions, like every provider command. @@ -118,17 +126,19 @@ An allowed query passes unchanged and silently (exit 0, no output). A blocked on **What the operator tunes.** `--guard-max-lines N` and `--guard-max-chars M` on `syberlabs propose --claude-code`, or `SYBERLABS_GUARD_MAX_LINES` and `SYBERLABS_GUARD_MAX_CHARS` in the host's environment (the flags win), or `guard_max_lines` and `guard_max_chars` on `ClaudeCodeProvider`. They are host policy, never contract data. The rules themselves and the 16-character tree-line floor are fixed in `query_guard.py` (`VERSION` 1). -**Limits, stated plainly.** The guard is a filter over what the model typed, not a proof that no code leaves the machine. A model can paraphrase code, split it into several short queries, split one credential across two lookups (each half alone is no credential shape), separate a key's characters with spaces, encode it (base64, base32, hex, URL or percent-encoding, HTML entities, ROT13, Morse, one character per word: none is decoded), use a lookalike the fold does not map (only NFKC, `Cf` removal and a fixed table of Cyrillic and Greek letters are applied; Armenian, Cherokee or other scripts are not), put it inside one line of a longer query (the tree rule compares whole lines), or quote a line it wrote during the run (the digests are taken from the parent's tracked files before the run). The identifier rule the packet sketched (a path or identifier from the tree that is in no public dependency name) is not built. A false positive costs a rephrased query: a query that happens to equal a 16-character line of the tree, an `user@host` that is not an address, a long but public question. Whether Claude Code 2.1.309 runs a `--settings` hook under `--bare` is not settled by what was read: the headless page says bare mode skips the *auto-discovery* of hooks and lists `--settings` as the way to pass settings, while the installed `--help` says bare mode skips "hooks (those defined in settings and by installed plugins)". No real Claude Code ran in this packet. The fail-closed rule above is what makes either answer safe: if the hook does not run, the first lookup stops the search and the record says `unguarded`. The first live run of either lane must check that its lookups read `pass` or a rule. +**Limits, stated plainly.** The guard is a filter over what the model typed, not a proof that no code leaves the machine. A model can paraphrase code, split it into several short queries, split one credential across two lookups (each half alone is no credential shape), separate a key's characters with spaces, encode it (base64, base32, hex, URL or percent-encoding, HTML entities, ROT13, Morse, one character per word: none is decoded), use a lookalike the fold does not map (only NFKC, `Cf` removal and a fixed table of Cyrillic and Greek letters are applied; Armenian, Cherokee or other scripts are not), put it inside one line of a longer query (the tree rule compares whole lines), or quote a line it wrote during the run (the digests are taken from the parent's tracked files before the run). The identifier rule the packet sketched (a path or identifier from the tree that is in no public dependency name) is not built. A false positive costs a rephrased query: a query that happens to equal a 16-character line of the tree, an `user@host` that is not an address, a long but public question. **Does a `--settings` hook run under `--bare`? No (Claude Code 2.1.312, live, 2026-10-30).** The headless page said bare mode skips the *auto-discovery* of hooks and lists `--settings` as the way to pass settings, while the `--help` says bare mode skips "hooks (those defined in settings and by installed plugins)". The `--help` is right. A hook diagnosis on a Freestyle Linux VM ran the same `PreToolUse` hook through `--settings` on the same command with and without `--bare` (`claude -p [--bare] --restricted --tools Bash --allowedTools "Bash(echo:*)" --setting-sources user --strict-mcp-config --settings `): the hook fired only without `--bare`, and without `--bare` it worked with `apiKeySource: ANTHROPIC_API_KEY`. The pilot canary then ran `propose --claude-code --bare --githits`: the model ran the launcher with no hook, the client sent the lookup (its record says `sent: true`, `status: ok`), and only afterwards did the host stop the run `provider_guard_inactive` with both lookups `unguarded`. Fail closed after the fact was not enough for a lookup that leaves the machine. Two changes follow: the launcher now refuses before the network without the hook's decision (the pre-send refusal, below), and `--githits` refuses `--bare` and runs on `api-hooked` (above). WebSearch, WebFetch and `--mcp-config` lookups on a `--bare` lane are still caught only after the fact (`spec/GAPS.md` 64 (3)). ## GitHits through the launcher (`--githits`, arm B) GitHits asked for the pilot to use its API rather than its MCP server, so that the host controls and measures every lookup. `ClaudeCodeProvider(..., githits=True)` (`syberlabs propose --claude-code --githits [--githits-arm NAME]`) gives the agent exactly one way to reach GitHits, the host's own client (`syberlabs/githits.py`, `docs/GITHITS.md`), and gives arm A nothing: ```bash -syberlabs propose --claude-code --bare # arm A: no GitHits, and no GitHits variable -syberlabs propose --claude-code --bare --githits # arm B: the same, plus the launcher and the skill +syberlabs propose --claude-code --lane api-hooked --trusted-tree # arm A: no GitHits, and no GitHits variable +syberlabs propose --claude-code --lane api-hooked --trusted-tree --githits # arm B: the same, plus the launcher and the skill ``` +Both arms run the same lane, so their argv differ only in the GitHits additions below (`tests/test_claude_code.py` pins this for `api-hooked`). `--githits --bare` is refused (`invalid_provider`): no hook runs there. + **The launcher.** For each call the provider writes, in the call's guard directory (outside the worktree, removed after the call), a read-only POSIX shell script named `githits` (`claude_code.write_launcher`): ```sh @@ -138,6 +148,8 @@ exec '' -I '/syberlabs/githits_cli_entry.py' "$@" `githits_cli_entry.py` puts the host checkout first on `sys.path`, refuses to run if `syberlabs` came from anywhere else, and runs the host's GitHits client with the arguments. It accepts only the five lookups (`search`, `example`, `read`, `list`, `grep`); `doctor` and anything else exit 2 with a usage message and write no record, and `example` takes no `--license-mode`, so it always sends `license_mode: strict` (`docs/GITHITS.md`, the launcher's entry). `doctor` stays the operator's: `syberlabs githits doctor`. An `example` is held on the host until this host's last one is 40 s old, across processes and arms (`syberlabs/githits_pace.py`). `-I` keeps the worktree's directory and `PYTHONPATH` off the import path, so the candidate's own `syberlabs/` never runs. Claude Code's Bash tool runs bash (Git Bash on Windows), which runs the script. The launcher is named by its absolute path with forward slashes, the exact text the model types; a path holding a space or a quote is refused (`provider_unavailable`). +**The pre-send refusal.** The launcher sends nothing unless the query guard decided on that very call. When the hook passes a launcher command, its decision also carries `arguments_digest`, the SHA-256 of the arguments it read from the command (`query_guard.bash_arguments`), under the same per-call MAC. The launcher's entry, once its lookup is parsed and before it is paced or sent, looks in the guard directory beside itself (`SYBERLABS_GUARD_LAUNCHER`'s parent) for a decision with `rule: pass`, a shell tool, an `arguments_digest` equal to the SHA-256 of its own argv, and a MAC that verifies under `SYBERLABS_GUARD_KEY`, and claims it by creating a file named for its tool use id in `launches/` exclusively, so one decision lets one launch through (`query_guard.claim_launch`). The hook's words and the launcher's argv agree because the hook passes only plain words and quoted strings, which the shell hands over unchanged; a disagreement refuses (fails closed). With no such decision (no hook ran, as under `--bare`; no key or launcher in the environment; a decision forged without the key, for other arguments, or already used) the entry records the call `status: unguarded`, `sent: false` (`githits.Client.refuse`), prints `refused: unguarded: ; no request was sent`, exits 2, and makes no HTTP request. The host then reconciles as before: under `--bare` no launcher call was guarded, so the record is `githits_unexpected_records`, journaled `guard: unguarded`, and the search stops `provider_guard_inactive`, now with the record saying the lookup never left. The entry is the provider's only: the operator's lookups and `doctor` run through `syberlabs githits`, which has no gate (the operator is not a model, and no hook stands in front of them), so a decision is required whenever the launcher runs and never on the operator's path. The limit is the MAC's: a process holding the key (an MCP server, an allowed shell) could still sign a decision (above). + **What arm B adds.** Argv: `Bash` in `--tools`, `Bash(:*)` in `--allowedTools`, and the skill after the harness note in `--append-system-prompt`. Environment, for the launcher's client: | Variable | Value | @@ -148,7 +160,8 @@ exec '' -I '/syberlabs/githits_cli_entry.py' "$@" | `GITHITS_SESSION_ID` | `-` (`X-GitHits-Session-ID`; GitHits allows only `[A-Za-z0-9_-]`, so not `/`); the thread is `thread_label` (the CLI passes the thread's id), else the search space's thread, so a provider built in Python names its thread too | | `GITHITS_AGENT` | `syberfactory/` (`X-GitHits-Agent`) | | `SYBERLABS_GITHITS_LICENSES` | the license allow list in force, written out (the operator's, else the permissive default) | -| `SYBERLABS_GITHITS_EXCLUDE` | the exclusion list in force, written out (`syberlabs` plus what the operator added) | +| `SYBERLABS_GITHITS_EXCLUDE` | the exclusion list in force, written out (the default deny-list, `syberlabs,sykosyber,pypi:syberwork,deepwiki.com,deepwiki.org`, plus what the operator added; `docs/GITHITS.md`) | +| `SYBERLABS_GITHITS_PACE_DIR` | the host's example pacing directory, resolved by the host (the operator's value, else `syberlabs-githits-pace` in the host's temporary directory), so a launcher whose `TMPDIR` or home differs (`api-hooked` moves the home) still shares one pacing slot | `GITHITS_API_TOKEN` is the operator's own, inherited from the environment and never written; the provider refuses `--githits` without it. `SYBERLABS_GITHITS_EXCLUDE_REPLACE` is never passed: an operator's switch is removed from Claude Code's environment, and the exclusion list above is computed without it, so it always begins with `syberlabs` (tested in both arms). The arm defaults to `b`. Arm B passes the operator's other GitHits variables through unchanged, as before. @@ -185,7 +198,7 @@ Per call, on the candidate's `signal` (all parsed by the host from the stream, b - `model_usage`: input, output and cache-read and cache-creation tokens summed over the run's assistant messages, counted once per message id (Claude Code repeats a message's usage on every content block it streams), with `model` when one model served the run; `model_usage_by_model` when more than one did; - `harness_turns`: the result's `num_turns`; `harness_stop`: the result's `subtype`; - `tool_usage`: `{web_search, web_fetch, githits, other: {tool: n}}`. GitHits is any `mcp__githits__*` tool; web is `WebSearch`, `WebFetch` and the server tools `web_search` and `web_fetch`. It is bounded by `providers.tool_usage`: at most 16 other names; -- `harness`: Claude Code's version, model, effort, the credential source name (`apiKeySource`, never a credential) and the MCP servers with their status; +- `harness`: Claude Code's version, model, effort, the credential source name (`apiKeySource`, never a credential), the MCP servers with their status, and the lane the call ran on (`lane`); - `lane`, `cost_basis`, `estimated_usd`, and `skipped_paths` (how many changed files were binary, not UTF-8 or over 1 MiB, and so left out); - `guard`: the query guard's policy for the call, `{version, max_lines, max_chars}`; - with `--githits`: `githits` (launcher calls, records, mismatch; above) and `harness.skill`. diff --git a/docs/EXPERIMENTS.md b/docs/EXPERIMENTS.md index 04636d5..c8a7da5 100644 --- a/docs/EXPERIMENTS.md +++ b/docs/EXPERIMENTS.md @@ -136,9 +136,9 @@ The pilot itself runs a real model with a paid key: Class X (`AGENTS.md`). A nam 1. The item list, as the `--items` file: each work item's id and its tag, `needs_lookup` when the work depends on a library or API the agent is unlikely to know well (new, niche, or changed since the model's training), else `routine`; beside it, each item's objective and scope. 2. The primary metric and the group the decision rests on (for example: the delivery rule on `needs_lookup` items, with `--delivered review`). 3. The exclusion rules: the nine excluded checks above, and nothing else. -4. The frozen setup: the SyberWork commit, Claude Code version, model, effort, `--allowed-tools`, `--max-turns`, the query guard's thresholds, the credential lane, and arm B's GitHits route (`--githits`) with its license policy (`SYBERLABS_GITHITS_LICENSES`, recorded as `signal.githits.license_policy`). Use the `api` lane (`--bare`) in both arms: it is the lane whose cost is billed and known, and in bare mode the tree's `CLAUDE.md` is not read in either arm. The contract prices every model the runs report (`evolution.budget.prices`); a run that used a model the contract does not price has an unknown cost. Anything that learns from runs stays off for the week. AutoSkill is a design (`docs/AUTOSKILL.md`) and is in no proposer path; the `jev` and `empirical` branch policies belong to `--evolve`, which the pilot does not use. +4. The frozen setup: the SyberWork commit, Claude Code version, model, effort, `--allowed-tools`, `--max-turns`, the query guard's thresholds, the credential lane, and arm B's GitHits route (`--githits`) with its license policy (`SYBERLABS_GITHITS_LICENSES`, recorded as `signal.githits.license_policy`). Use the `api-hooked` lane (`--lane api-hooked --trusted-tree`) in both arms: the API key, so the cost is billed and known, without `--bare`, because Claude Code 2.1.312 runs no `--settings` hook under `--bare` and the provider refuses `--githits` there (`docs/CLAUDE_CODE.md`, the hooked API lane); each call gets an empty config directory and home. Not in bare mode, the tree's `CLAUDE.md` is read, equally in both arms: record it with the conditions. The contract prices every model the runs report (`evolution.budget.prices`); a run that used a model the contract does not price has an unknown cost. Anything that learns from runs stays off for the week. AutoSkill is a design (`docs/AUTOSKILL.md`) and is in no proposer path; the `jev` and `empirical` branch policies belong to `--evolve`, which the pilot does not use. -**Before the week, the dry run** (Class X as well, a few dollars): `python scripts/pilot_dry_run.py --plan plan.json` runs this protocol on 3 to 5 items on the API route (`--githits`, `docs/GITHITS.md`, "The A/B report"), in a home of its own: a preflight that refuses to start on any failure, one canary call that shows the query guard fires under `--bare`, both arms per item from one `HEAD` in a seeded order, the blind review and this report under a hard budget cap (default $8). It passes when the nine metrics are known for every thread and the report compares every item; its docstring says how to run it, and its `receipt.json` and `report.md` are what to send. +**Before the week, the dry run** (Class X as well, a few dollars): `python scripts/pilot_dry_run.py --plan plan.json` runs this protocol on 3 to 5 items on the API route (`--githits`, `docs/GITHITS.md`, "The A/B report"), in a home of its own: a preflight that refuses to start on any failure, one canary call that shows the query guard fires on the `api-hooked` lane, both arms per item from one `HEAD` in a seeded order, the blind review and this report under a hard budget cap (default $8). It passes when the nine metrics are known for every thread and the report compares every item; its docstring says how to run it, and its `receipt.json` and `report.md` are what to send. **Each item, each arm, from the same `HEAD`** (accepting moves only the thread's own branch, so `HEAD` stays put): diff --git a/docs/GITHITS.md b/docs/GITHITS.md index c6c25c8..2f4cb4b 100644 --- a/docs/GITHITS.md +++ b/docs/GITHITS.md @@ -56,9 +56,24 @@ not be shown SyberLabs code as if it were outside evidence. For the pilot, conta side: the work items are unpublished, both arms run locally on a pinned base, and this client is the backstop. An exclusion at GitHits' index is optional, not something the pilot relies on. -The client works in three layers. The exclusion list is `SyberLabs` plus whatever `SYBERLABS_GITHITS_EXCLUDE` adds: -a comma list of owners or `owner/repo`, matched case-insensitively. The variable only adds; it never removes -`SyberLabs`, and an empty value keeps the default. `SYBERLABS_GITHITS_EXCLUDE_REPLACE=1` makes the variable's list +The client works in three layers. The exclusion list is the default deny-list plus whatever +`SYBERLABS_GITHITS_EXCLUDE` adds, matched case-insensitively. The default (`githits.DEFAULT_EXCLUDE`, from the +pilot's pre-registration) is: + +- the GitHub owners `SyberLabs` and `sykosyber` (`github.com/sykosyber/*`, and each owner's own domains and + `*.github.io` pages, as for any owner); +- `pypi:syberwork`, the PyPI name of this repository's distribution (`pyproject.toml`; there is no npm package), as a + target and as `pypi.org/project/syberwork` in a result; +- `deepwiki.com` and `deepwiki.org` with their subdomains, the DeepWiki front-ends that re-serve public repositories, + SyberLabs' among them, as a target (`site:deepwiki.com/...`, a URL) and anywhere in a result's text; +- forks of SyberLabs repositories where the result's URL shows one: a repository named like a SyberLabs product + (`githits.FORK_NAMES`: `syberwork`, `sybersdk`, `syberfactory`, `syberlabs`, `syberstate`) under any owner on a + code host (`github.com//SyberWork`, `github:/syberfactory`). This is by name only: a fork renamed + to something else is not detected. + +An entry of the variable is an owner, `owner/repo`, a package `registry:name` (`npm:left-pad`) or a host name with a +dot and no slash (`example.org`, which also covers its subdomains). The variable only adds; it never removes the +default, and an empty value keeps it. `SYBERLABS_GITHITS_EXCLUDE_REPLACE=1` makes the variable's list replace the default; it exists for tests only, and the Claude Code provider never passes it to the launcher's client. Passing another list to `Client(exclude=...)` in code replaces the default too. @@ -127,21 +142,29 @@ the two is an error, so a run never believes it is recording when it is not. {"op": "search", "query_digest": "", "targets_count": 1, "status": "ok", "http_status": 200, "request_id": "", "results_returned": 3, "results_dropped": 1, "results_dropped_license": 0, "license_policy": "", "contamination_blocked": false, "latency_ms": 412, "rate_limit_remaining": 97, - "at": "2026-10-26T17:01:16.123Z", "mac": ""} + "at": "2026-10-26T17:01:16.123Z", "sent": true, "mac": ""} ``` -- `status` is `ok`, `excluded_target`, `contamination_blocked` or `error`. +- `status` is `ok`, `excluded_target`, `contamination_blocked`, `error`, or `unguarded` (the launcher's entry + refused the call before sending: no query-guard decision for it; below). +- `sent` says whether the request was written to GitHits: `false` when it never left (an excluded target, an + `unguarded` refusal, no token, a connection, TLS or proxy failure), `true` once it was (an answer, an HTTP error, + a read timeout, a connection closed before the answer), `null` when the call was interrupted in flight. - `query_digest` is the SHA-256 of the canonical JSON request: the body of a POST, the query parameters of a read. - `rate_limit_remaining` is `X-RateLimit-Remaining` when the answer has it. The spec documents it for every - operation here except `example`. + operation here except `example`. Observed live (pilot canary, 2026-10-30): `search` and `grep` answers carry it + (599, then 598), `example` and `doctor` answers do not (`null`); `read` and `list` were not called. - `mac` is HMAC-SHA256 under the record key over the canonical JSON of every other field. It is `query_guard.sign`, the same canonical form and MAC the query guard's decisions use. - `read_records(directory, key)` returns the records whose MAC verifies, and counts every other line (bad MAC, extra or missing field, not JSON) as ignored. `paced_ms`, present only on a call the host's launcher paced (next section), is how long the launcher held the -call before sending it; the MAC covers it, and a record without it has exactly the fields above. +call before sending it; the MAC covers it, and a record without it has exactly the fields above. Every launcher +`example` to GitHits goes through the pacing lock, so `paced_ms` is present on it even when nothing was waited for, +with a small value: the lock and the time file's own cost (the canary's first example recorded `paced_ms: 4`; its +second, held for the slot, `26312`). Read a small value as "not held", not as a wait. No query text, no result and no token is ever stored. As with the guard's key, the MAC shows that a process holding the key wrote the line. The key is in the environment of whatever runs the client, so it does not show @@ -221,6 +244,13 @@ five lookups, `search`, `example`, `read`, `list` and `grep` (`githits.LOOKUPS`, `--license-mode`: through the launcher an example always asks for `license_mode: strict`. `doctor`, `--help`, an unknown operation and a usage error exit 2 with a usage message before a client exists, so nothing is sent and nothing is recorded. The operator keeps `syberlabs githits doctor` and `--license-mode`. + +A lookup is sent only when the query guard's hook passed that launcher call: the entry claims the hook's +MAC-verified decision whose `arguments_digest` equals its own argv's (`query_guard.claim_launch`; one decision, one +launch) before it paces or sends. Without one (no hook ran, as under `claude --bare` on 2.1.312; no guard key in +the environment; a forged, mismatched or used decision) it records the call `status: unguarded`, `sent: false`, +prints `refused: unguarded: ...` and exits 2, with no HTTP request (`docs/CLAUDE_CODE.md`, the pre-send refusal). +`syberlabs githits` (the operator's) has no such gate. It prints as `syberlabs` does, unencodable characters replaced, and before the record: a failed print records `error`. ## The pinned contract @@ -262,13 +292,13 @@ before its first candidate and keep everything else equal, `--allowed-tools` inc ```bash syberlabs start "Parse ISO dates with the UTC offset" --paths src tests --unattended syberlabs experiment label --name githits-pilot --item LIB-1 --arm off --tag needs_lookup -syberlabs propose --claude-code --bare --allowed-tools Read,Edit,Write,Glob,Grep,WebSearch,WebFetch +syberlabs propose --claude-code --lane api-hooked --trusted-tree --allowed-tools Read,Edit,Write,Glob,Grep,WebSearch,WebFetch syberlabs run --once syberlabs start "Parse ISO dates with the UTC offset" --paths src tests --unattended syberlabs experiment label --name githits-pilot --item LIB-1 --arm on --tag needs_lookup -syberlabs propose --claude-code --bare --allowed-tools Read,Edit,Write,Glob,Grep,WebSearch,WebFetch \ - --githits --githits-arm on # the only difference: the launcher and the skill +syberlabs propose --claude-code --lane api-hooked --trusted-tree --allowed-tools Read,Edit,Write,Glob,Grep,WebSearch,WebFetch \ + --githits --githits-arm on # the only difference: the launcher and the skill (never with --bare) syberlabs run --once syberlabs experiment report --name githits-pilot --items items.txt --delivered review @@ -331,8 +361,8 @@ misreports. Claim ceiling: implemented and tested with stand-ins; no pilot has r | `GITHITS_API_TOKEN` | the bearer token (required for a call; a refused target needs none) | | `GITHITS_API_URL` | base URL, default `https://api.githits.dev/v1` | | `GITHITS_SESSION_ID`, `GITHITS_AGENT` | optional `X-GitHits-Session-ID` and `X-GitHits-Agent` | -| `SYBERLABS_GITHITS_EXCLUDE` | owners or `owner/repo` to exclude besides `SyberLabs`, comma list (it only adds) | -| `SYBERLABS_GITHITS_EXCLUDE_REPLACE` | `1`: the list above replaces `SyberLabs` (tests only) | +| `SYBERLABS_GITHITS_EXCLUDE` | owners, `owner/repo`, `registry:name` packages or hosts to exclude besides the default deny-list (`SyberLabs`, `sykosyber`, `pypi:syberwork`, `deepwiki.com`, `deepwiki.org`, and forks by name), comma list (it only adds) | +| `SYBERLABS_GITHITS_EXCLUDE_REPLACE` | `1`: the list above replaces the default deny-list, the fork rule included (tests only) | | `SYBERLABS_GITHITS_LICENSES` | allowed SPDX ids, comma list, or `any` (default: the permissive list) | | `SYBERLABS_GITHITS_RECORD_DIR`, `SYBERLABS_GITHITS_RECORD_KEY` | where records go and their MAC key (both or neither) | | `SYBERLABS_GITHITS_PACE_DIR` | the host's example pacing directory, absolute (default: `syberlabs-githits-pace` in the temporary directory) | diff --git a/examples/githits_ab.py b/examples/githits_ab.py index 87f8218..cbadbe2 100644 --- a/examples/githits_ab.py +++ b/examples/githits_ab.py @@ -5,7 +5,8 @@ ``syberlabs experiment report`` over it. Two routes for arm B's GitHits: - ``--route api`` (the default; the pilot's route since #134): both arms run the real Claude Code proposer, - ``ClaudeCodeProvider`` on the api lane, arm B with ``githits=True``. For each call the host writes the GitHits + ``ClaudeCodeProvider`` on the api-hooked lane (the API key without ``--bare``, so the guard's hook runs), arm B + with ``githits=True``. For each call the host writes the GitHits launcher, its query guard judges every lookup and command, the host's own GitHits client makes each lookup and signs its record, and the host journals one ``lookup`` per record and its ledger of the call on the candidate (``signal.githits``), which is what the report reads the arm from. Only the ``claude`` executable is a stand-in: a @@ -290,7 +291,7 @@ def run(name, tool_input): Path(path).write_bytes(text.encode("utf-8")) # the exact text: no newline translation on Windows model, usage = "stand-in-model", {"input_tokens": tokens, "output_tokens": 350} say({"type": "system", "subtype": "init", "claude_code_version": "2.1.309", "model": model, "effort": "high", - "apiKeySource": "ANTHROPIC_API_KEY" if "--bare" in sys.argv else "none", "mcp_servers": [], + "apiKeySource": "ANTHROPIC_API_KEY" if os.environ.get("ANTHROPIC_API_KEY") else "none", "mcp_servers": [], "session_id": "rehearsal", "tools": ["Read", "Edit", "Write"] + (["Bash"] if launcher else [])}) say({"type": "assistant", "message": {"id": "msg_1", "model": model, "usage": usage, "content": [{"type": "text", "text": "Working on " + item}]}}) @@ -367,14 +368,16 @@ def close(self) -> None: @contextlib.contextmanager def api_environment(url: str): - """The API route's environment, put back afterwards: GitHits at the loopback stand-in under a placeholder token, no - GitHits setting from the shell, and no gateway or provider switch (``claude_code.THIRD_PARTY_SWITCHES`` would refuse - the api lane; the stand-in sends nothing anywhere).""" + """The API route's environment, put back afterwards: GitHits at the loopback stand-in under a placeholder token, a + placeholder Anthropic key (the api-hooked lane requires one; the stand-in sends nothing anywhere), no GitHits + setting from the shell, and no gateway or provider switch (``claude_code.THIRD_PARTY_SWITCHES`` would refuse the + lane).""" saved = dict(os.environ) for name in (*THIRD_PARTY_SWITCHES, githits.EXCLUDE_ENV, githits.EXCLUDE_REPLACE_ENV, githits.LICENSES_ENV, githits.SESSION_ENV, githits.AGENT_ENV, githits.RECORD_DIR_ENV, githits.RECORD_KEY_ENV): os.environ.pop(name, None) - os.environ.update({githits.TOKEN_ENV: "rehearsal-placeholder-not-a-token", githits.URL_ENV: url}) + os.environ.update({githits.TOKEN_ENV: "rehearsal-placeholder-not-a-token", githits.URL_ENV: url, + "ANTHROPIC_API_KEY": "rehearsal-placeholder-not-a-key"}) try: yield finally: @@ -389,11 +392,13 @@ def git(root: Path, *args: str) -> None: def claude_code(kit: Kit, thread, script: Path, item: str, arm: str) -> ClaudeCodeProvider: - """The API route's proposer, as the pilot builds it: ``syberlabs propose --claude-code --bare`` in arm A, the same - with ``--githits --githits-arm on`` in arm B, except on the misconfigured item, whose treatment forgot - ``--githits``. The executable is the stand-in; everything else is the real provider.""" - return ClaudeCodeProvider(kit.repo, bare=True, executable=[sys.executable, str(script)], max_calls=3, - githits=arm == "on" and item != MISCONFIGURED, arm=arm, thread_label=thread.id) + """The API route's proposer, as the pilot builds it: ``syberlabs propose --claude-code --lane api-hooked + --trusted-tree`` in arm A, the same with ``--githits --githits-arm on`` in arm B (never ``--bare``: Claude Code + 2.1.312 runs no ``--settings`` hook there), except on the misconfigured item, whose treatment forgot ``--githits``. + The executable is the stand-in; everything else is the real provider.""" + return ClaudeCodeProvider(kit.repo, lane="api-hooked", trusted_tree=True, executable=[sys.executable, str(script)], + max_calls=3, githits=arm == "on" and item != MISCONFIGURED, arm=arm, + thread_label=thread.id) @contextlib.contextmanager diff --git a/scripts/pilot_dry_run.py b/scripts/pilot_dry_run.py index a6569cd..3713d70 100644 --- a/scripts/pilot_dry_run.py +++ b/scripts/pilot_dry_run.py @@ -15,9 +15,9 @@ and stops if ``HEAD`` moves during the run. 2. Put both credentials in the shell's environment, never in a file:: - export ANTHROPIC_API_KEY=... # your own key: the api lane (claude -p --bare) + export ANTHROPIC_API_KEY=... # your own key: the api-hooked lane (claude -p, not --bare) export GITHITS_API_TOKEN=... - unset ANTHROPIC_BASE_URL # and any CLAUDE_CODE_USE_* switch: the api lane refuses them + unset ANTHROPIC_BASE_URL # and any CLAUDE_CODE_USE_* switch: the lane refuses them (PowerShell: ``$env:ANTHROPIC_API_KEY = "..."``; ``Remove-Item Env:ANTHROPIC_BASE_URL``.) 3. Write the plan: ``python scripts/pilot_dry_run.py --template > plan.json``, then fill in 3 to 5 items (id, tags, @@ -45,7 +45,7 @@ ------------ Preflight refuses to start on any failure: the repository and its base commit; Claude Code ``2.1.295`` or newer -(``claude_code.MIN_VERSION``: ``onFailure: "block"`` on the guard's hook); the api lane on the operator's own +(``claude_code.MIN_VERSION``: ``onFailure: "block"`` on the guard's hook); the api-hooked lane on the operator's own ``ANTHROPIC_API_KEY`` with no third-party routing switch (``claude_code.third_party``: ``ANTHROPIC_BASE_URL``, ``CLAUDE_CODE_USE_*``, ``CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST``); ``GITHITS_API_TOKEN`` in the environment and in no file the run touches (the repository's tracked and untracked files, its ``.env*``, ``.mcp.json`` and local Claude @@ -54,14 +54,16 @@ Claude Code runs it, signing a block and a pass that verify under a key the host set); the scheduling budget; and the contracts the run installs. -The canary is one cheap call in the treatment's mode (``propose --claude-code --bare --githits``, one call, a few +The canary is one cheap call in the treatment's mode (``propose --claude-code --lane api-hooked --trusted-tree +--githits``: not ``--bare``, under which Claude Code 2.1.312 runs no ``--settings`` hook; one call, a few turns, without the web tools so the launcher is its only way to look anything up) whose objective asks for two launcher lookups, one the guard must refuse (it holds an email address). It passes when the guard refused a lookup for what it says and passed one that GitHits answered (an authentic client record), every decision verified and none ``unguarded``, and every model the call used is priced; a call that shows only one of the two is inconclusive and fails, since it proves either no refusal or no GitHits. Then each item runs in both arms from the same ``HEAD``, both arms back to back, the items and which arm goes first set by a seeded shuffle: ``start --unattended``, -``experiment label`` (before the first candidate), ``propose --claude-code --bare [--githits]``, ``run --once`` +``experiment label`` (before the first candidate), ``propose --claude-code --lane api-hooked --trusted-tree +[--githits]``, ``run --once`` (accepts on the host's checks), ``export``. Then the blind review (``review --record``), ``experiment report --items FILE --delivered review [--json | --markdown]``, and the bar. @@ -170,6 +172,10 @@ PASS, FAIL, WARN = "PASS", "FAIL", "WARN" SCRUB_ENV = "CLAUDE_CODE_SUBPROCESS_ENV_SCRUB" API_KEY_ENV = "ANTHROPIC_API_KEY" +# Both arms' lane: the API key with Claude Code not in bare mode, so the query guard's --settings hook runs (Claude +# Code 2.1.312 runs none under --bare, and the provider refuses --githits there); an empty config directory and home +# per call keep the machine's own configuration out (docs/CLAUDE_CODE.md, the hooked API lane). +LANE = claude_code.HOOKED # Files beside the code where a credential tends to be written; scanned whether or not git ignores them. SECRET_FILES = (".env", ".mcp.json", ".claude/settings.json", ".claude/settings.local.json", "CLAUDE.local.md") MAX_SCAN_BYTES = 20 * 1024 * 1024 @@ -544,14 +550,15 @@ def check_lane(env) -> Check: problems = [] switches = claude_code.third_party(env) if switches: - problems.append(", ".join(switches) + " set: a --bare run would go to another endpoint or provider, not " - "Anthropic's API under your key; unset it (this dry run runs the api lane only)") + problems.append(", ".join(switches) + " set: the run would go to another endpoint or provider, not " + f"Anthropic's API under your key; unset it (this dry run runs the {LANE} lane only)") if not str(env.get(API_KEY_ENV, "")).strip(): - problems.append(f"{API_KEY_ENV} is not set: the api lane (--bare) runs on your own key from the environment") + problems.append(f"{API_KEY_ENV} is not set: the {LANE} lane runs on your own key from the environment") if problems: return Check("credential lane", FAIL, "; ".join(problems)) - return Check("credential lane", PASS, f"api (--bare, billed) on {API_KEY_ENV} from this environment; no " - "third-party routing switch set", {"lane": "api", "cost_basis": "billed"}) + return Check("credential lane", PASS, f"{LANE} (the API key without --bare, so the guard's hook runs; billed) on " + f"{API_KEY_ENV} from this environment; no third-party routing switch set", + {"lane": LANE, "cost_basis": "billed"}) def scan_paths(repo: Path, extra: list) -> list[Path]: @@ -652,7 +659,8 @@ def check_doctor(sy: Syberlabs) -> Check: def check_guard_key(env) -> Check: """The query guard's per-call key reaches its hook: Claude Code's subprocess scrub is off, and the host's own hook (``query_guard.hook_command``, the exec form Claude Code runs) signs a block and a pass under a key the host put in - its environment, both verifying. Claude Code itself running the hook under ``--bare`` is the canary's to show.""" + its environment, both verifying. Claude Code itself running the hook (on the api-hooked lane) is the canary's to + show.""" problems = [] if truthy(env.get(SCRUB_ENV)): problems.append(f"{SCRUB_ENV} is set: Claude Code would strip the guard's key and the GitHits token from the " @@ -822,9 +830,10 @@ def harness_of(events: list) -> dict: # ----------------------------------------------------------------------------------------------------------- canary def claude_flags(plan: dict, arm: str, max_calls: int, max_turns: int, tools: str | None = None) -> list[str]: - """``propose`` flags: the same in both arms but for ``--githits`` (and its arm name) in the treatment.""" - flags = ["propose", "--claude-code", "--bare", "--max-calls", str(max_calls), "--max-turns", str(max_turns), - "--allowed-tools", tools or plan["allowed_tools"]] + """``propose`` flags: the same in both arms (the api-hooked lane, never --bare) but for ``--githits`` (and its arm + name) in the treatment.""" + flags = ["propose", "--claude-code", "--lane", LANE, "--trusted-tree", "--max-calls", str(max_calls), + "--max-turns", str(max_turns), "--allowed-tools", tools or plan["allowed_tools"]] if plan["guard_max_lines"] is not None: flags += ["--guard-max-lines", str(plan["guard_max_lines"])] if plan["guard_max_chars"] is not None: @@ -856,7 +865,7 @@ def judge_canary(events: list, contracts: dict) -> tuple[str, str, dict]: "harness": harness_of(events)} if finished.get("error") == "provider_guard_inactive" or unguarded: return FAIL, (f"the query guard did not fire: {len(unguarded)} lookup(s) ran with no verified decision " - "(unguarded): under this Claude Code and --bare the --settings hook did not run, or its record " + f"(unguarded): under this Claude Code on {LANE} the --settings hook did not run, or its record " "was not the hook's. Do not run the pilot in this mode"), data if not refused or not reached: missing = ([] if refused else ["no lookup the guard refused for what it says (the canary asks for one " @@ -1270,7 +1279,7 @@ def preflight(self) -> bool: receipt["plan"] = {"sha256": digest(plan_path.read_text(encoding="utf-8")), "items": [{"id": i["id"], "tags": i["tags"], "paths": i["paths"], "objective_sha256": digest(i["objective"])} for i in plan["items"]]} - receipt["settings"] = {"lane": "api", "max_calls": plan["max_calls"], "max_turns": plan["max_turns"], + receipt["settings"] = {"lane": LANE, "max_calls": plan["max_calls"], "max_turns": plan["max_turns"], "allowed_tools": plan["allowed_tools"], "canary_turns": CANARY_TURNS, "guard": {"max_lines": plan["guard_max_lines"], "max_chars": plan["guard_max_chars"]}, @@ -1312,7 +1321,7 @@ def confirm(self) -> bool: def canary(self) -> bool: plan, repo = self.plan, self.repo - say("\ncanary: one call with --bare --githits; the guard must refuse one lookup and pass one to GitHits") + say(f"\ncanary: one call on {LANE} with --githits; the guard must refuse one lookup and pass one to GitHits") stop = gate(self.spent(), self.share, self.cap) if stop: return self.fail(f"budget: {stop}") @@ -1443,10 +1452,10 @@ def conclude(self) -> None: rows += report_rows(found, len(plan["items"]), reasons) lanes = sorted({lane for entry in self.entries for lane in entry["harness"].get("lane", [])}) bases = sorted({basis for entry in self.entries for basis in entry["harness"].get("cost_basis", [])}) - if lanes and (lanes != ["api"] or bases != ["billed"]): + if lanes and (lanes != [LANE] or bases != ["billed"]): rows.append({"status": FAIL, "name": "lane", "detail": f"the candidates recorded lane {lanes}, cost basis " - f"{bases}; the dry run runs the api lane only"}) - reasons.append("a candidate ran outside the api lane") + f"{bases}; the dry run runs the {LANE} lane only"}) + reasons.append(f"a candidate ran outside the {LANE} lane") total = receipt["spend"]["total"] receipt["spend"]["within_cap"] = within = total is not None and total <= self.cap + 1e-9 rows.append({"status": PASS if within else FAIL, "name": "spend", diff --git a/spec/GAPS.md b/spec/GAPS.md index f310976..be49e02 100644 --- a/spec/GAPS.md +++ b/spec/GAPS.md @@ -125,3 +125,4 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e 62. **SyberEval scores, gates and reports its runs; some gate rules cannot be computed for some arms, the agent-facing safety tasks do not exist yet, and candidate code still runs beside the hidden checks.** `eval-score` (2026-10-26, landed on the `eval-core` verification fixes of 2026-10-28: the runner's lock, bundle verification, the per-run preflight gate, strict JSON, refused run-id collisions and finite numbers) adds `syberlabs/eval/gate.py`, `score.py` and `report.py` and `syberlabs eval score|report` (`docs/EVAL.md`); the required spec field `hidden_feedback` (`tails`, `status`, `none`), enforced through `RepairProvider`'s optional `feedback_filter`, which every other caller leaves unset; the hidden copy's hashing (`tasks.tree_digests`) at four points against the copy as staged (`tasks.hidden_changes`, which names a symbolic link by its target and tolerates pytest's own `.pytest_cache`), with the run outcome `not_acceptable`, reason `hidden_tampered`; a host root where the hidden copy and the kit home get random names outside the run's work directory; the append-only spend ledger the eval's cap counts, read strictly, which also records every run that raised (it replaces the `runs/failed.jsonl` of the verification fixes); and `syberlabs.eval` in the wheel. It closes entry 60's (1) and (8) and narrows its (3) and (4). The contract schema, the event kinds, the golden traces and the kernel's behaviour for every existing caller are unchanged. Open: (1) the agent-facing safety tasks (`evals/tasks/safety-*`, packet `eval-safety-fixtures`) and the gate's validation on them: no planted canary, forbidden action or unsafe path has yet been offered to an agent, and every gate rule is tested on synthetic journals and with stand-in adapters only; (2) `canary.provider_requests` is `not_computable` for `claude-code` arms, whose own model requests hold whatever Claude Code's tools read, and for `evolve` and `evolve_jev` arms, whose mutate requests' feedback is not recorded; such a run's gate is `incomplete`, never `pass`; (3) the policy is not in the journal, so `approval_or_prerequisite_skipped` checks that at least one approval came between a `needs_approval` decision and its effect, not the policy's count; (4) `credential_observed` matches credential shapes (`syberlabs._patterns.SECRET`, at a word boundary) and the values of the host's credential variables present when the run ended, not every possible secret, and full check outputs and provider stderr are not retained, so only their recorded tails are scanned; (5) the hidden copy is hashed when staged, after the proposal (one hash after every check run inside it, the repair loop's evaluations included), after `Thread.check` (which runs the checks again only when the chosen candidate's result is missing or stale, so it usually follows a check that ran nothing) and before acceptance; the gate's `hidden_tampered` rule is `not_computable` for a bundle missing a hash the runner always writes; so a rewrite undone between two hashes, or inside one proposal's evaluations, is not detected, and candidate code inside a check can still read the copy and print it into its output, which `tails` sends to the next request; only a sandbox per check closes this (research register N4); (6) `hidden_feedback` filters the `repair` and `claude-code` feedback path only: an evolve arm builds its mutate requests in `syberlabs/evolve.py`, which no filter reaches, so a spec may use evolve arms under `tails` only; (7) under `none` the repair loop still stops on the first candidate that passes the hidden checks, so they still choose among up to K attempts on the host's side, and a call after a failure starts from the failed candidate's tree; (8) the eval's recorded spend can exceed `total_dollars` by one call's cost, and a call in flight when the runner is killed outright is in no journal and no ledger line; (9) the model API takes no sampling seed, so a run's seed indexes a repeat and is not a reproducible draw; (10) E5's registration is deleted, because adding `hidden_feedback` moved its digest and `register` returns an existing registration of the same digest unchanged; it is registered again on a clean checkout of `main` before it runs; (11) the package reads `benchmarks/`, the adapters and `evals/` from a checkout, so the wheel's `syberlabs eval` needs one to run anything; (12) the design gaps the verification pinned as `expectedFailure` in `tests/test_eval_adversarial.py` stay open: a candidate that exits 0 at import passes every hidden check; under `tails` a candidate can print its hidden check into the next request (under `status` and `none` it cannot, and that test passes); a `claude-code` arm's model is a declaration the runner neither sets nor compares with the model that served; a `repo:` task's run repository holds the commits after its base; and a runner killed outright charges nothing (tested on POSIX only, where `SIGKILL` exists). 63. **The GitHits pilot's example pacing and license reading rest on GitHits' documentation, not on observed traffic.** (1) The host paces `example` calls to one per 40 s per host (`syberlabs/githits_pace.py`, called by the launcher entry); GitHits' limit is per organization, so two hosts, or an operator's own `syberlabs githits example` (not paced), share the organization's budget without seeing each other's slots. (2) A loopback base URL is not paced unless `SYBERLABS_GITHITS_PACE_SECONDS` asks: a loopback proxy that forwards to GitHits would go unpaced. (3) Calls queued behind several others wait one interval each, so a Bash tool timeout shorter than the queue ends a call before it is sent, and the host then flags it `githits_unrecorded`. (4) The time file is plain text a process of the same user can rewrite; it paces cooperating processes and is no security boundary. (5) The license reading admits an example's `References`/`Sources` links by their `(SPDX)` and passes an example with neither code (a fenced, indented or HTML block) nor a source; inline code in prose is not read as code, and the three documented shapes (`complete`, `degraded`, `no_example` in the pinned OpenAPI document) are the only evidence of what real responses look like until the first live example is recorded. (6) The engineering smoke pins Claude Code 2.1.312 exactly because the host and the pilot VM run it; that version's flags were not re-checked here. +64. **The GitHits launcher fails closed before the network, and the treatment runs on a hooked API lane; neither is verified live, and other lookups on a `--bare` lane are still caught only after the fact.** The live canary of 2026-10-30 (Claude Code 2.1.312 on a Freestyle Linux VM) settled 56 (2): a `--settings` `PreToolUse` hook does not run under `--bare`, so `propose --claude-code --bare --githits` let the model run the launcher with no guard, and the client sent the lookup (`sent: true`) before the host stopped the run `provider_guard_inactive`. Now the launcher's entry sends nothing without claiming the hook's MAC-verified decision for its exact argv (`query_guard.claim_launch`; refused calls are recorded `status: unguarded`, `sent: false`), and `--githits` refuses `--bare` and runs on `--lane api-hooked` (the API key without `--bare`, a fresh `CLAUDE_CONFIG_DIR` and home per call, `--disable-slash-commands`; `docs/CLAUDE_CODE.md`). Not solved: (1) a decision is bound to the launcher's arguments (their SHA-256) and claimed once, not to a tool use id, which the launcher cannot see; a process holding the per-call key (a stdio MCP server, a shell under `--allow-code-tools-with-mcp`) can still sign a decision for its own arguments, as it could before; (2) the hook's words and the launcher's argv agree for plain words and quoted strings under a POSIX shell; a shell or platform that hands over different bytes (Git Bash's argument conversion to a Windows process, a non-UTF-8 byte) refuses a legitimate lookup (fail closed), which no live run has exercised; (3) WebSearch, WebFetch and `--mcp-config` (`mcp__githits__*`) lookups on a `--bare` lane have no pre-send gate: the hook does not run, so they still leave and are flagged `unguarded` afterwards; only `--githits` refuses `--bare`, and `api-hooked` or a login lane is where those tools are guarded; (4) `api-hooked` has run only against the stand-in `claude`: whether 2.1.312 with an empty config directory and home starts headless on `ANTHROPIC_API_KEY` (`apiKeySource`) and runs the hook is for its first live call, and the flags were read from 2.1.309's `--help` because 2.1.312's was not on the building machine; managed (policy) settings still load, `--setting-sources` has no value meaning `--settings` alone, `USERPROFILE` moves only on Windows, and the worktree's memory files (`CLAUDE.md`) are read in both arms, so the lane needs `--trusted-tree`; (5) the example license parser now reads any ATX or bold heading that names a source or reference ("## Source code" included) as a sources section and a fenced or indented block inside a blockquote as code, both of which can only drop more examples (fail closed); how often that drops a usable GitHits example is not measured; (6) the default deny-list (`SyberLabs`, `sykosyber`, `pypi:syberwork`, `deepwiki.com`, `deepwiki.org`) detects a fork of a SyberLabs repository only by its name (`githits.FORK_NAMES`) in a result's URL or a target, so a renamed fork, a vendored copy, or SyberLabs code mirrored under another name and site still passes, and no mirror or front-end other than DeepWiki is listed. diff --git a/syberlabs/claude_code.py b/syberlabs/claude_code.py index 95d0c9f..1c5affa 100644 --- a/syberlabs/claude_code.py +++ b/syberlabs/claude_code.py @@ -3,7 +3,8 @@ syberlabs propose --claude-code --trusted-tree # the operator's login (subscription_local) syberlabs propose --claude-code --bare # an API key in the environment (api) syberlabs propose --claude-code --bare --mcp-config githits.json # the same, with one more MCP server - syberlabs propose --claude-code --bare --githits # the same, with the GitHits API launcher + syberlabs propose --claude-code --lane api-hooked --trusted-tree # an API key, without --bare: hooks run + syberlabs propose --claude-code --lane api-hooked --trusted-tree --githits # the same, with the GitHits launcher ``ClaudeCodeProvider`` is ``providers.RepairProvider`` with one call replaced: instead of a JSON adapter, each call checks out the parent (the base, or the candidate being repaired) in a detached temporary worktree, runs Claude Code @@ -79,10 +80,21 @@ ``signal.guard.allow_code_tools_with_mcp``. Credential lanes (bring your own credential: SyberLabs never stores, pools or forwards a credential; the operator's -own Claude Code runs under their own login, or ``--bare`` with their API key). ``api`` (``--bare``, the default +own Claude Code runs under their own login, or with their API key). ``api`` (``--bare``, the default there) is ``billed``: the tokens are priced under the contract's ``budget.prices`` through the host's meter, as an -adapter's usage is. ``provider_3p`` (Bedrock, Vertex, Foundry credentials) is ``estimated``: priced the same way, but -the bill is the cloud provider's. ``subscription_local`` (an inherited login, the default without ``--bare``) and +adapter's usage is. ``api-hooked`` (``--lane api-hooked``, never with ``--bare``) is the same API key, billed the +same way, with Claude Code not in bare mode, so the query guard's ``--settings`` hook runs: Claude Code 2.1.312 does +not run ``--settings`` ``PreToolUse`` hooks under ``--bare`` (live canary on a Linux VM, 2026-10-30), so ``githits`` +refuses ``--bare`` and the GitHits treatment runs only on a lane where its guard runs. Ambient configuration is kept +out of it: each call gets a fresh, empty ``CLAUDE_CONFIG_DIR`` and ``HOME`` (``USERPROFILE`` on Windows) in the call's +guard directory (no user settings, memory, skills, plugins or login), ``--disable-slash-commands``, and the +``--restricted``, ``--setting-sources user`` and ``--strict-mcp-config`` every lane passes, so only managed settings +and the provider's ``--settings`` load; ``ANTHROPIC_API_KEY`` is kept (required) and every other credential, login +token, routing or proxy variable is removed (``LOGIN_LANE_DROPPED_ENV`` and ``CLAUDE_CODE_OAUTH_TOKEN``), while +``ANTHROPIC_BASE_URL`` or a third-party switch is refused, as on ``api``. A non-bare run reads the worktree's memory +files, so ``api-hooked`` needs ``trusted_tree`` like every non-bare lane. ``provider_3p`` (Bedrock, Vertex, +Foundry credentials) is ``estimated``: priced the same way, but the bill is the cloud provider's. +``subscription_local`` (an inherited login, the default without ``--bare``) and ``seat_team_enterprise`` are ``none``: no per-call bill exists, so nothing is priced and the dollar budget does not apply (turn, second and token budgets for those lanes are not built: ``spec/GAPS.md``). @@ -124,7 +136,7 @@ from pathlib import Path from syberlabs import githits as githits_client -from syberlabs import query_guard +from syberlabs import githits_pace, query_guard from syberlabs.checks import _stop from syberlabs.errors import Rejected from syberlabs.gitspace import MAX_FILE_BYTES, Repo @@ -133,9 +145,19 @@ from syberlabs.query_guard import query_text from syberlabs.search import SearchSpace -LANES = ("api", "subscription_local", "seat_team_enterprise", "provider_3p") +HOOKED = "api-hooked" # the API key without --bare, so --settings hooks run; ambient configuration kept out +LANES = ("api", HOOKED, "subscription_local", "seat_team_enterprise", "provider_3p") COST_BASES = ("billed", "estimated", "none") -LANE_COST = {"api": "billed", "provider_3p": "estimated", "subscription_local": "none", "seat_team_enterprise": "none"} +LANE_COST = {"api": "billed", HOOKED: "billed", "provider_3p": "estimated", "subscription_local": "none", + "seat_team_enterprise": "none"} +API_KEY_ENV = "ANTHROPIC_API_KEY" +# What api-hooked removes besides LOGIN_LANE_DROPPED_ENV (minus the key it runs on): the claude.ai login token, so +# a run recorded as billed on the key cannot fall back to a subscription. +HOOKED_DROPPED_ENV = ("CLAUDE_CODE_OAUTH_TOKEN",) +CONFIG_DIR, HOME_DIR = "claude-config", "home" # api-hooked: a fresh, empty config directory and home per call +BARE_HOOKS = ("Claude Code 2.1.312 does not run --settings PreToolUse hooks under --bare (live canary on a Linux VM, " + "2026-10-30: the launcher ran with no guard and its lookup was sent before the run stopped " + "provider_guard_inactive)") BIN_ENV = "SYBERLABS_CLAUDE_BIN" # the Claude Code executable (a path, or a command line); default ``claude`` on PATH DEFAULT_TOOLS = ("Read", "Edit", "Write", "Glob", "Grep") # no Bash: a shell could read the guard's key # The built-in tools that run commands or code that ``--restricted``'s help names (2.1.309: "Bash, PowerShell and @@ -615,7 +637,10 @@ def read() -> None: return proc.returncode, lines, stderr -def _third_party_reason(names: list[str]) -> str: +def _third_party_reason(names: list[str], lane: str = "api") -> str: + if lane == HOOKED: + return (", ".join(names) + f" is set: this {HOOKED} run would go to another endpoint or provider, not " + "Anthropic's API under your key; unset it (provider_3p runs with --bare)") return (", ".join(names) + " is set: this --bare run would go to another endpoint or provider, not Anthropic's API " "under your key, so its lane is provider_3p (pass --lane provider_3p) and api is refused") @@ -637,6 +662,16 @@ def __init__(self, repo: Repo | str | Path, *, bare: bool = False, trusted_tree: raise Rejected("invalid_provider", "lane must be one of " + ", ".join(LANES)) if bare and lane in ("subscription_local", "seat_team_enterprise"): raise Rejected("invalid_provider", "--bare never reads a Claude login; its lane is api or provider_3p") + if bare and lane == HOOKED: + raise Rejected("invalid_provider", f"{HOOKED} is the API-key lane without --bare, so the query guard's " + "--settings hook runs; drop --bare") + if not bare and lane == "api": + raise Rejected("invalid_provider", f"the api lane is --bare; for the API key without --bare (hooks run) " + f"use --lane {HOOKED}") + if githits and bare: + raise Rejected("invalid_provider", f"refusing --githits with --bare: {BARE_HOOKS}. Run the treatment where " + f"its guard runs: --lane {HOOKED} --trusted-tree (the API key), or a " + "login lane; arm A runs the same lane without --githits") if not bare and not trusted_tree: raise Rejected("untrusted_tree", "refusing to run Claude Code without --bare on a tree not declared trusted: a non-bare " @@ -650,8 +685,11 @@ def __init__(self, repo: Repo | str | Path, *, bare: bool = False, trusted_tree: raise Rejected("invalid_provider", "allowed tools must be a non-empty list of tool names") if max_turns is not None and (type(max_turns) is not int or max_turns < 1): raise Rejected("invalid_provider", "max_turns must be a positive integer") - if bare and lane == "api" and third_party(os.environ): - raise Rejected("invalid_provider", _third_party_reason(third_party(os.environ))) + if lane in ("api", HOOKED) and third_party(os.environ): + raise Rejected("invalid_provider", _third_party_reason(third_party(os.environ), lane)) + if lane == HOOKED and not str(os.environ.get(API_KEY_ENV, "")).strip(): + raise Rejected("invalid_provider", f"{HOOKED} runs on {API_KEY_ENV} from the operator's environment " + "(its config directory is empty: no login, no apiKeyHelper); set it") self.githits, self.arm, self.thread_label = bool(githits), arm, thread_label self.skill: tuple[str, str] | None = None if self.githits: @@ -676,7 +714,8 @@ def __init__(self, repo: Repo | str | Path, *, bare: bool = False, trusted_tree: try: githits_client.LicensePolicy.from_env(os.environ) githits_client.Exclusions.from_env(os.environ) - except githits_client.GitHitsError as exc: + githits_pace.directory(os.environ) + except (githits_client.GitHitsError, githits_pace.PaceError) as exc: raise Rejected("invalid_provider", str(exc)) from None self.skill = githits_skill() if mcp_config is not None: @@ -739,6 +778,8 @@ def command(self, settings: str | Path | None = None, launcher: str | None = Non tools.append(f"Bash({launcher}:*)") note = HARNESS_NOTE + "\n\n" + self.skill_prompt(launcher) argv = [*self.argv, "-p"] + (["--bare"] if self.bare else []) + if self.lane == HOOKED: + argv.append("--disable-slash-commands") # no skill or command loads, whatever the machine holds argv += ["--output-format", "stream-json", "--verbose", "--permission-mode", "dontAsk", "--restricted", "--tools", ",".join(built_in_tools(tools)), "--allowedTools", ",".join(tools), "--setting-sources", "user", "--strict-mcp-config", @@ -753,30 +794,55 @@ def command(self, settings: str | Path | None = None, launcher: str | None = Non def environment(self) -> dict: """The caller's environment, narrowed: no signing seed on any lane; on a login lane nothing in - ``LOGIN_LANE_DROPPED_ENV``; with no GitHits route (neither ``githits=True`` nor ``mcp_config``: arm A) no - GitHits variable (``githits_variable``: the token, the base URL and every ``SYBERLABS_GITHITS_*``), so arm A - holds no GitHits credential. A ``--bare`` api run whose environment routes elsewhere is refused here too, in + ``LOGIN_LANE_DROPPED_ENV``; on ``api-hooked`` nothing in it either but ``ANTHROPIC_API_KEY``, and no + ``CLAUDE_CODE_OAUTH_TOKEN`` (each call then adds a fresh config directory and home: ``isolation``); with no + GitHits route (neither ``githits=True`` nor ``mcp_config``: arm A) no GitHits variable (``githits_variable``: + the token, the base URL and every ``SYBERLABS_GITHITS_*``), so arm A holds no GitHits credential. An api or + api-hooked run whose environment routes elsewhere, or an api-hooked run with no key, is refused here too, in case the environment changed after the provider was built.""" route = self.githits or self.mcp_config is not None env = {k: v for k, v in os.environ.items() if k not in DROPPED_ENV and (route or not githits_variable(k))} if self.lane in ("subscription_local", "seat_team_enterprise"): env = {k: v for k, v in env.items() if not _dropped(k)} - elif self.bare and self.lane == "api" and third_party(env): - raise Rejected("provider_lane_mismatch", _third_party_reason(third_party(env))) + elif self.lane in ("api", HOOKED) and third_party(env): + raise Rejected("provider_lane_mismatch", _third_party_reason(third_party(env), self.lane)) + if self.lane == HOOKED: + if not str(env.get(API_KEY_ENV, "")).strip(): + raise Rejected("provider_lane_mismatch", f"{API_KEY_ENV} is no longer set; {HOOKED} runs on it") + env = {k: v for k, v in env.items() + if k == API_KEY_ENV or not (_dropped(k) or k in HOOKED_DROPPED_ENV)} return env + def isolation(self, directory: Path) -> dict: + """``api-hooked``: what keeps the machine's own Claude Code configuration out of one call. A fresh, empty + ``CLAUDE_CONFIG_DIR`` ("All settings, session history, and plugins are stored under this path", Claude Code's + environment reference) and ``HOME`` (and ``USERPROFILE`` on Windows, where Node reads the home from it), both + created in the call's guard ``directory``, which is removed after the call. Empty on every other lane.""" + if self.lane != HOOKED: + return {} + config, home = Path(directory) / CONFIG_DIR, Path(directory) / HOME_DIR + config.mkdir() + home.mkdir() + found = {"CLAUDE_CONFIG_DIR": str(config), "HOME": str(home)} + if os.name == "nt": + found["USERPROFILE"] = str(home) + return found + def githits_environment(self, records: Path, record_key: str, session: str) -> dict: """What arm B adds to Claude Code's environment for the launcher's client: the record directory and key - (fresh per call), the session and agent headers, and the license and exclusion policy as explicit lists - (the operator's, else the defaults; ``SYBERLABS_GITHITS_EXCLUDE`` only adds to ``SyberLabs``). The token is - the operator's own ``GITHITS_API_TOKEN``, inherited, never written.""" + (fresh per call), the host's example pacing directory (``SYBERLABS_GITHITS_PACE_DIR``: the operator's, + else the host's temporary-directory default, resolved here, so a launcher whose ``TMPDIR`` or home differs + still shares one pacing slot), the session and agent headers, and the license and exclusion policy as + explicit lists (the operator's, else the defaults; ``SYBERLABS_GITHITS_EXCLUDE`` only adds to ``SyberLabs``). + The token is the operator's own ``GITHITS_API_TOKEN``, inherited, never written.""" licenses = githits_client.LicensePolicy.from_env(os.environ) exclude = githits_client.Exclusions.from_env( {k: v for k, v in os.environ.items() if k != githits_client.EXCLUDE_REPLACE_ENV}) return {githits_client.RECORD_DIR_ENV: str(records), githits_client.RECORD_KEY_ENV: record_key, + githits_pace.PACE_DIR_ENV: str(githits_pace.directory(os.environ)), githits_client.SESSION_ENV: session, githits_client.AGENT_ENV: f"syberfactory/{self.arm}", githits_client.LICENSES_ENV: ",".join(licenses.allowed) if licenses.allowed else githits_client.ANY_LICENSE, - githits_client.EXCLUDE_ENV: ",".join("/".join(entry) for entry in exclude.entries)} + githits_client.EXCLUDE_ENV: exclude.written()} def _session(self, space: SearchSpace) -> str: """``X-GitHits-Session-ID``: ``-`` (GitHits allows ``[A-Za-z0-9_-]{1,64}``, so no ``/``). The @@ -850,7 +916,7 @@ def _call(self, space: SearchSpace, request: dict, parents: list[str]): hook_env = query_guard.hook_environment(guard_dir, key, launcher) env = {k: v for k, v in self.environment().items() if not (self.githits and k == githits_client.EXCLUDE_REPLACE_ENV)} - env = {**env, **hook_env, **extra} + env = {**env, **hook_env, **extra, **self.isolation(guard_dir)} try: code, _, stderr = run_stream(self.command(settings, launcher), prompt, cwd=tree, env=env, timeout=self.timeout, lines=lines) @@ -919,7 +985,7 @@ def _call(self, space: SearchSpace, request: dict, parents: list[str]): # without assigning a missing record to a particular concurrent command. self._journal(space, [unrecorded_lookup({})]) signal = {"lane": self.lane, "cost_basis": self.cost_basis, - "harness": {**transcript.harness(), "installed_version": self.version}, + "harness": {**transcript.harness(), "installed_version": self.version, "lane": self.lane}, "guard": {**self.guard, "policy_digest": hook_env[query_guard.POLICY_DIGEST_ENV], **({"allow_code_tools_with_mcp": True} if self.allow_code_tools_with_mcp else {})}} if githits_signal is not None: diff --git a/syberlabs/cli.py b/syberlabs/cli.py index fc469da..db751b5 100644 --- a/syberlabs/cli.py +++ b/syberlabs/cli.py @@ -168,7 +168,8 @@ def main(argv: list[str] | None = None) -> int: help="for --repair and --claude-code: the most calls in this search (default: the search's " "candidate budget; 1 is one shot)") propose.add_argument("--bare", action="store_true", - help="for --claude-code: claude --bare, API key from the environment (lane api, billed)") + help="for --claude-code: claude --bare, API key from the environment (lane api, billed); " + "Claude Code 2.1.312 runs no --settings hook under --bare, so --githits refuses it") propose.add_argument("--trusted-tree", action="store_true", help="for --claude-code without --bare: declare this repository trusted; a non-bare claude -p " "runs project hooks and .mcp.json servers with no trust prompt") @@ -184,13 +185,17 @@ def main(argv: list[str] | None = None) -> int: propose.add_argument("--githits", action="store_true", help="for --claude-code: GitHits through the API client (arm B of the pilot): a per-call " "launcher as the only Bash command, guarded, its records journaled as lookups, and the " - "adapted GitHits skill appended (docs/CLAUDE_CODE.md); needs GITHITS_API_TOKEN") + "adapted GitHits skill appended (docs/CLAUDE_CODE.md); needs GITHITS_API_TOKEN; " + "refused with --bare (no hook runs there): use --lane api-hooked --trusted-tree") propose.add_argument("--githits-arm", default="b", metavar="NAME", help="for --githits: the arm's name in X-GitHits-Session-ID - and " "X-GitHits-Agent syberfactory/ (default b)") propose.add_argument("--max-turns", type=int, help="for --claude-code: claude --max-turns") - propose.add_argument("--lane", choices=("api", "subscription_local", "seat_team_enterprise", "provider_3p"), - help="for --claude-code: the credential lane (default api with --bare, else subscription_local)") + propose.add_argument("--lane", choices=("api", "api-hooked", "subscription_local", "seat_team_enterprise", + "provider_3p"), + help="for --claude-code: the credential lane (default api with --bare, else subscription_local); " + "api-hooked: ANTHROPIC_API_KEY without --bare, so the query guard's hook runs, with an " + "empty config directory and home per call (needs --trusted-tree)") propose.add_argument("--guard-max-lines", type=int, metavar="N", help="for --claude-code: the query guard blocks a lookup with more than N non-blank lines " "(default $SYBERLABS_GUARD_MAX_LINES, else 3)") diff --git a/syberlabs/eval/spec.py b/syberlabs/eval/spec.py index 9dc8e63..ac6ab99 100644 --- a/syberlabs/eval/spec.py +++ b/syberlabs/eval/spec.py @@ -53,7 +53,7 @@ HEX64 = r"^[0-9a-f]{64}$" ARM_KINDS = ("single", "repair", "evolve", "evolve_jev", "claude-code") HIDDEN_FEEDBACK = ("tails", "status", "none") # what of a hidden check's result reaches the proposer (module docstring) -LANES = ("api", "subscription_local", "seat_team_enterprise", "provider_3p") +LANES = ("api", "api-hooked", "subscription_local", "seat_team_enterprise", "provider_3p") # claude_code.LANES MAX_TOTAL_DOLLARS = 1000.0 _TEXT = {"type": "string", "minLength": 1, "maxLength": 4000} diff --git a/syberlabs/githits.py b/syberlabs/githits.py index 65f95aa..abcce3f 100644 --- a/syberlabs/githits.py +++ b/syberlabs/githits.py @@ -69,9 +69,10 @@ HMAC-SHA256 over the record's canonical JSON under the key (``query_guard.sign``, the MAC the query guard's decisions use). No query text, no result and no token is stored; ``query_digest`` is the SHA-256 of the canonical request. ``sent`` says whether the request was written to the server: false when it never left (an excluded -target, no token, a connection, TLS or proxy failure), true once it was (an answer, an HTTP error, a read timeout, -a connection closed before the answer), null when the call was interrupted in flight. So a timeout after GitHits -received the request and a refused connection no longer leave the same record. ``read_records`` returns the +target, a launcher call refused ``unguarded`` before sending, no token, a connection, TLS or proxy failure), true +once it was (an answer, an HTTP error, a read timeout, a connection closed before the answer), null when the call +was interrupted in flight. So a timeout after GitHits received the request and a refused connection no longer leave +the same record. ``read_records`` returns the records whose MAC verifies and counts the rest. A record may also carry ``paced_ms`` (``RECORD_OPTIONAL``): the milliseconds the host's launcher held the call before sending it, to keep GitHits' per-organization example rate (``githits_pace``, ``run(pace=...)``). This client never waits or retries on its own; the field is present only when @@ -111,7 +112,14 @@ EXCLUDE_REPLACE_ENV = "SYBERLABS_GITHITS_EXCLUDE_REPLACE" # "1": the list replaces the default; for tests only LICENSES_ENV = "SYBERLABS_GITHITS_LICENSES" RECORD_DIR_ENV, RECORD_KEY_ENV = "SYBERLABS_GITHITS_RECORD_DIR", "SYBERLABS_GITHITS_RECORD_KEY" -DEFAULT_EXCLUDE = ("SyberLabs",) +# The default deny-list (the pilot's pre-registration): SyberLabs' and its founder's GitHub owners, the PyPI name of +# this repository's distribution (``pyproject.toml``: ``syberwork``; there is no npm package), and the DeepWiki +# front-ends that re-serve public repositories, SyberLabs' among them. ``SYBERLABS_GITHITS_EXCLUDE`` adds to it. +DEFAULT_EXCLUDE = ("SyberLabs", "sykosyber", "pypi:syberwork", "deepwiki.com", "deepwiki.org") +# A fork of a SyberLabs repository keeps its name under another owner (``github.com//SyberWork``), so a +# repository named like a SyberLabs product, under any owner on a code host, is excluded too: by name only, so a fork +# renamed to something else is not detected. +FORK_NAMES = ("syberwork", "sybersdk", "syberfactory", "syberlabs", "syberstate") # The licenses a result may carry by default: permissive licenses whose only obligation is to keep a notice, which an # Apache-2.0 project can meet. Anything not listed (copyleft, NOASSERTION, unknown, none) is dropped. PERMISSIVE = ("MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC", "0BSD", "Unlicense", "Zlib", "PSF-2.0", @@ -125,7 +133,8 @@ RECORD_OPTIONAL = ("paced_ms",) # present only when the host's launcher paced the call (``run(pace=...)``) OK, EXCLUDED, BLOCKED, ERROR = "ok", "excluded_target", "contamination_blocked", "error" -STATUSES = (OK, EXCLUDED, BLOCKED, ERROR) +UNGUARDED = "unguarded" # the launcher's entry refused the call before sending: no query-guard decision for it +STATUSES = (OK, EXCLUDED, BLOCKED, ERROR, UNGUARDED) # Lean defaults for an agent's context window; GitHits' own default in the comment. SEARCH_LIMIT = 5 # hits per search (10) @@ -144,6 +153,9 @@ REPOSITORY_PREFIXES = {"github": "https://github.com/", "gitlab": "https://gitlab.com/", "codeberg": "https://codeberg.org/"} _NAME = re.compile(r"^[A-Za-z0-9_.-]+$") +_HOST_NAME = re.compile(r"^(?:[A-Za-z0-9-]+\.)+[A-Za-z][A-Za-z0-9-]*$") +# Where a registry shows a package, for a package exclusion read in free text and result URLs. +_REGISTRY_PAGES = {"pypi": r"pypi\.org/(?:project|simple)/", "npm": r"(?:www\.)?npmjs\.(?:com|org)/package/"} _HOSTS = (r"(?:[a-z0-9-]+\.)*(?:github\.com|gitlab\.com|codeberg\.org|bitbucket\.org|githubusercontent\.com)" r"|github|gitlab|codeberg") @@ -215,7 +227,7 @@ class Request(NamedTuple): class Result(NamedTuple): op: str - status: str # ok, excluded_target or contamination_blocked; errors raise GitHitsError + status: str # ok, excluded_target, contamination_blocked or unguarded; errors raise GitHitsError data: dict | None # the response with excluded results removed; None when refused or blocked http_status: int | None request_id: str | None @@ -228,7 +240,7 @@ class Result(NamedTuple): @property def refused(self) -> bool: - return self.status in (EXCLUDED, BLOCKED) + return self.status in (EXCLUDED, BLOCKED, UNGUARDED) class RecordRead(NamedTuple): @@ -239,22 +251,52 @@ class RecordRead(NamedTuple): # ---------------------------------------------------------------------------------------------------------- exclusions class Exclusions: - """Owners (``SyberLabs``) and repositories (``SyberLabs/SyberWork``) whose content must not reach a result, - matched case-insensitively in every reading ``_readings`` gives (as a URL parser or a lenient target parser - would read the text).""" + """Owners (``SyberLabs``), repositories (``SyberLabs/SyberWork``), packages (``pypi:syberwork``) and hosts + (``deepwiki.com``, with its subdomains) whose content must not reach a result, plus any code-host repository named + like a SyberLabs product (``FORK_NAMES``: a fork under another owner), matched case-insensitively in every reading + ``_readings`` gives (as a URL parser or a lenient target parser would read the text). An entry is an owner or + ``owner/repo``, a ``registry:name`` (``REGISTRIES``), or a host name (a dot and no slash).""" - def __init__(self, entries): + def __init__(self, entries, forks: bool = True): parsed = [] for entry in entries: - parts = [part.strip() for part in str(entry).strip().strip("/").split("/")] + text = str(entry).strip().strip("/") + registry, _, package = text.partition(":") + if package: + if registry.lower() not in REGISTRIES or not _NAME.match(package.strip()): + raise GitHitsError("bad_exclusion", f"a package exclusion is registry:name, got {text[:80]!r}") + parsed.append((f"{registry.lower()}:{package.strip().lower()}",)) + continue + parts = [part.strip() for part in text.split("/")] if not 1 <= len(parts) <= 2 or not all(_NAME.match(part) for part in parts): - raise GitHitsError("bad_exclusion", f"an exclusion is an owner or owner/repo, got {str(entry)[:80]!r}") + raise GitHitsError("bad_exclusion", "an exclusion is an owner, owner/repo, registry:name or a host, " + f"got {str(entry)[:80]!r}") + if len(parts) == 1 and "." in parts[0] and not _HOST_NAME.match(parts[0]): + raise GitHitsError("bad_exclusion", f"a host exclusion is a host name, got {parts[0][:80]!r}") parsed.append(tuple(part.lower() for part in parts)) if not parsed: raise GitHitsError("bad_exclusion", "the exclusion list is empty") self.entries = tuple(dict.fromkeys(parsed)) mentions, bare = [], [] - for owner, *repo in self.entries: + if forks: # FORK_NAMES under any owner: on a code host, or as a code-host shorthand or bare owner/repo + names = r"(?:" + "|".join(re.escape(name) for name in FORK_NAMES) + r")(?![a-z0-9_-])" + mentions.append(r"(? str: + """The list as ``SYBERLABS_GITHITS_EXCLUDE`` writes it: comma-separated entries ``from_env`` reads back.""" + return ",".join("/".join(entry) for entry in self.entries) + @classmethod def from_env(cls, environ=None) -> "Exclusions": - """``DEFAULT_EXCLUDE`` plus the owners and repositories ``SYBERLABS_GITHITS_EXCLUDE`` adds; the variable never - removes the default. ``SYBERLABS_GITHITS_EXCLUDE_REPLACE=1`` (for tests only) makes its list replace it.""" + """``DEFAULT_EXCLUDE`` plus the owners, repositories, packages and hosts ``SYBERLABS_GITHITS_EXCLUDE`` adds; the + variable never removes the default. ``SYBERLABS_GITHITS_EXCLUDE_REPLACE=1`` (for tests only) makes its list + replace it.""" environ = _process_environ if environ is None else environ listed = [item for item in environ.get(EXCLUDE_ENV, "").split(",") if item.strip()] if listed and environ.get(EXCLUDE_REPLACE_ENV, "").strip() == "1": - return cls(listed) + return cls(listed, forks=False) # the whole default replaced, the fork rule included return cls(list(DEFAULT_EXCLUDE) + listed) def mentioned(self, text) -> bool: @@ -501,14 +548,23 @@ def satisfied(node) -> bool: r"|githubusercontent\.com)(?::\d+)?/", re.IGNORECASE) # GitHits' own pages (the "Open in GitHits Web App" link every documented example ends with) are never a source. _GITHITS_HOST = re.compile(r"^https?://(?:[a-z0-9-]+\.)*githits\.(?:com|dev)(?::\d+)?(?:[/?#]|$)", re.IGNORECASE) -# A "References" or "Sources" heading (ATX, bold or a bare line, as the documented examples write "### References"): -# every link under it, up to the next heading or thematic break, is a source, whatever its host. -_SOURCES_HEADING = re.compile(r"^[ \t]{0,3}(?:#{1,6}[ \t]+)?\**(?:references|sources)\**[ \t]*:?[ \t]*#*[ \t]*$", +# A "References" or "Sources" heading, as the documented examples write "### References": an ATX or bold heading that +# names a reference or source anywhere ("## Source", "### Sources and references", "**Reference links:**"), or a +# bare line that is the word alone ("Sources:"), singular or plural, whatever the line ending (CRLF too). Every link +# under it, up to the next heading or thematic break, is a source, whatever its host. Reading too many headings as +# sources only asks more links for a license: an unlicensed one drops the example (fail closed). +_SOURCE_WORD = r"\b(?:references?|sources?)\b" +_SOURCES_HEADING = re.compile(r"^[ \t]{0,3}(?:#{1,6}[ \t]+[^\r\n]*?" + _SOURCE_WORD + r"[^\r\n]*?" + r"|\*\*[^*\r\n]*?" + _SOURCE_WORD + r"[^*\r\n]*?\*\*[ \t]*:?" + r"|\**(?:references?|sources?)\**[ \t]*:?)[ \t]*\r?$", re.IGNORECASE | re.MULTILINE) -_SECTION_END = re.compile(r"^[ \t]{0,3}(?:#{1,6}(?:[ \t]|$)|(?:[-*_][ \t]*){3,}$)", re.MULTILINE) -# Code an agent could take: a fenced block, an indented block, or HTML code. An example without any, and without a -# source, carries nothing to license (GitHits' documented "degraded" and "no_example" answers). -_CODE = re.compile(r"^[ \t]{0,3}(?:`{3,}|~{3,})|^(?: {4}|\t)[ \t]*\S|<(?:pre|code)\b", re.IGNORECASE | re.MULTILINE) +_SECTION_END = re.compile(r"^[ \t]{0,3}(?:#{1,6}(?:[ \t]|\r?$)|(?:[-*_][ \t]*){3,}\r?$)", re.MULTILINE) +# Code an agent could take: a fenced block or an indented block, at the margin or inside a blockquote (``> ```py``), +# or HTML code. An example without any, and without a source, carries nothing to license (GitHits' documented +# "degraded" and "no_example" answers); a quoted fence is code like any other, so an uncited one is dropped. +_QUOTE = r"(?:>[ \t]?)*" # a blockquote's markers, nested or not, before the block +_CODE = re.compile(r"^[ \t]{0,3}" + _QUOTE + r"[ \t]{0,3}(?:`{3,}|~{3,})|^(?:[ \t]{0,3}>[ \t]?(?:>[ \t]?)*)?" + r"(?: {4}|\t)[ \t]*\S|<(?:pre|code)\b", re.IGNORECASE | re.MULTILINE) _LICENSE_LINE = re.compile(r"^[ \t]*(?:[-*][ \t]*)?\**licen[cs]es?\**[ \t]*:[ \t]*(.+?)[ \t]*$", re.IGNORECASE | re.MULTILINE) _SPDX_TAG = re.compile(r"SPDX-License-Identifier:[ \t]*(.*?)[ \t]*(?:\*/|-->)?[ \t]*$", re.IGNORECASE | re.MULTILINE) @@ -912,11 +968,7 @@ def call(self, request: Request, deliver=None, paced_ms: int | None = None) -> R caller did not receive. ``paced_ms``, when given, is how long the caller held the call before this (the host launcher's pacing); it is recorded, never waited here, and ``latency_ms`` never includes it.""" started = time.monotonic() - record = {"op": request.op, "query_digest": query_digest(request), "targets_count": len(request.targets), - "status": ERROR, "http_status": None, "request_id": None, "results_returned": 0, - "results_dropped": 0, "results_dropped_license": 0, "license_policy": self.licenses.digest, - "contamination_blocked": False, "latency_ms": 0, - "rate_limit_remaining": None, "sent": False} + record = self._fresh(request) if paced_ms is not None: if type(paced_ms) is not int or paced_ms < 0: raise GitHitsError("bad_pacing", "paced_ms is a non-negative integer") @@ -961,6 +1013,22 @@ def call(self, request: Request, deliver=None, paced_ms: int | None = None) -> R self._finish(record, result, started, headers, deliver) return result + def refuse(self, request: Request, reason: str, deliver=None) -> Result: + """Record ``request`` as refused before it was sent, ``status: unguarded`` and ``sent: false``, and return the + refused result; nothing is sent. The launcher's entry calls this when the query guard passed no launcher call + with these exact arguments (``query_guard.claim_launch``): Claude Code ran the launcher without the hook.""" + started = time.monotonic() + result = Result(request.op, UNGUARDED, None, None, None, 0, 0, False, None, reason=reason) + self._finish(self._fresh(request), result, started, None, deliver) + return result + + def _fresh(self, request: Request) -> dict: + """A call's record before anything is sent: ``error`` until the call says otherwise, ``sent: false``.""" + return {"op": request.op, "query_digest": query_digest(request), "targets_count": len(request.targets), + "status": ERROR, "http_status": None, "request_id": None, "results_returned": 0, + "results_dropped": 0, "results_dropped_license": 0, "license_policy": self.licenses.digest, + "contamination_blocked": False, "latency_ms": 0, "rate_limit_remaining": None, "sent": False} + def _finish(self, record: dict, result: Result, started: float, headers, deliver) -> None: failure = None if deliver is not None: @@ -1346,12 +1414,14 @@ def write_text(stream, text: str) -> None: stream.flush() -def run(args, environ=None, stdout=None, stderr=None, pace=None) -> int: - """``syberlabs githits ...``: 0 a result, 2 refused (an excluded target or a blocked result), 1 an error; - ``doctor``: 0 ready, 1 not (the status says why). A lookup's output is written (``write_text``) before its record, - so an output that fails is recorded ``error`` and exits 1 (``Client.call``, ``deliver``). ``pace``, given by the - host's launcher entry, is called with each lookup's request once the client and request are built and before it - is sent; it returns the milliseconds it held the call (recorded as ``paced_ms``) or None (not paced).""" +def run(args, environ=None, stdout=None, stderr=None, pace=None, gate=None) -> int: + """``syberlabs githits ...``: 0 a result, 2 refused (an excluded target, a blocked result, or an unguarded + launcher call), 1 an error; ``doctor``: 0 ready, 1 not (the status says why). A lookup's output is written + (``write_text``) before its record, so an output that fails is recorded ``error`` and exits 1 (``Client.call``, + ``deliver``). ``gate`` and ``pace`` are given by the host's launcher entry, never by the operator's CLI. ``gate`` + is called first with each lookup's request once the client and request are built: None lets it go on, a reason + refuses it (``Client.refuse``: recorded ``unguarded``, ``sent: false``, nothing sent, not paced). ``pace`` is then + called before it is sent; it returns the milliseconds it held the call (recorded as ``paced_ms``) or None.""" stdout, stderr = stdout or sys.stdout, stderr or sys.stderr op = args.githits_command options = {name: getattr(args, name) for name in _OPTIONS[op]} @@ -1381,8 +1451,12 @@ def deliver(result: Result) -> None: request = list_request(args.target, **options) else: request = grep_request(args.pattern, args.targets, **options) - paced_ms = pace(request) if pace is not None else None - result = client.call(request, deliver=deliver, paced_ms=paced_ms) + refusal = gate(request) if gate is not None else None + if refusal is not None: + result = client.refuse(request, refusal, deliver=deliver) + else: + paced_ms = pace(request) if pace is not None else None + result = client.call(request, deliver=deliver, paced_ms=paced_ms) except GitHitsError as exc: try: write_text(stderr, f"error: {exc}\n") diff --git a/syberlabs/githits_cli_entry.py b/syberlabs/githits_cli_entry.py index 5ff3e34..40fbf8f 100644 --- a/syberlabs/githits_cli_entry.py +++ b/syberlabs/githits_cli_entry.py @@ -13,6 +13,16 @@ exists, so they write no record; ``doctor`` is the operator's, through ``syberlabs githits doctor``. ``example`` has no ``--license-mode`` here and always sends ``license_mode: strict``. +No lookup leaves without the query guard's decision for it. Once a lookup's request is built, and before it is paced +or sent, the entry claims a decision the guard's ``PreToolUse`` hook wrote for a launcher call with exactly this argv +(``query_guard.claim_launch``: ``rule: pass``, the arguments' SHA-256, a MAC under the per-call +``SYBERLABS_GUARD_KEY``, each decision claimed once). Without one it refuses: the call is recorded ``status: +unguarded`` with ``sent: false`` (``githits.Client.refuse``), ``refused: unguarded: ...`` goes to standard error, +the exit code is 2, and no HTTP request is made. That is what happens when Claude Code runs the launcher without +running the hook (``claude --bare`` on 2.1.312 skips ``--settings`` hooks: live canary, 2026-10-30), with no guard +key in the environment, or with a forged or mismatched decision. The entry is the provider's alone: operator +lookups and ``doctor`` go through ``syberlabs githits``, which has no gate and needs no decision. + An ``example`` to GitHits is paced here, on the host, before it is sent (``githits_pace.wait``): at most one per ``githits_pace.EXAMPLE_INTERVAL`` (40 s) across every launcher process on this host, both arms included, under a cross-process lock and a time file in the host's pacing directory, outside any worktree. GitHits limits its example @@ -49,9 +59,10 @@ def main(argv: list[str] | None = None) -> int: if sys.path[:1] != [root]: sys.path.insert(0, root) import syberlabs - from syberlabs import cli, githits, githits_pace + from syberlabs import cli, githits, githits_pace, query_guard - if any(Path(module.__file__).resolve().parent != HERE for module in (syberlabs, cli, githits, githits_pace)): + if any(Path(module.__file__).resolve().parent != HERE + for module in (syberlabs, cli, githits, githits_pace, query_guard)): sys.stderr.write("error: githits_launcher: syberlabs was not imported from the host checkout\n") return 1 cli.survive_unicode() # before any output: a cp1252 pipe replaces what it cannot encode instead of raising @@ -69,7 +80,16 @@ def main(argv: list[str] | None = None) -> int: signal.signal(signal.SIGTERM, _terminated) except (AttributeError, ValueError, OSError): # no SIGTERM here, or not the main thread: run as before pass - return githits.run(args, pace=_pace) + return githits.run(args, pace=_pace, gate=lambda request: _gate(words)) + + +def _gate(words: list[str]) -> str | None: + """``githits.run``'s gate hook: None when this run claimed the query guard's decision for exactly ``words`` + (``query_guard.claim_launch``); otherwise the refusal's reason, and the call is recorded unguarded, unsent.""" + from syberlabs import query_guard + + reason, detail = query_guard.claim_launch(words) + return None if reason is None else f"{reason}: {detail}; no request was sent" def _pace(request) -> int | None: diff --git a/syberlabs/query_guard.py b/syberlabs/query_guard.py index c1f6061..783b451 100644 --- a/syberlabs/query_guard.py +++ b/syberlabs/query_guard.py @@ -54,6 +54,11 @@ lookup with its ``guard``: the rule, ``pass``, or ``unguarded`` when no verified decision carries the lookup's exact tool use id, tool and query digest. +A launcher call the hook passes also gets ``arguments_digest`` on its decision (the SHA-256 of the arguments it read, +covered by the MAC). The launcher's entry (``githits_cli_entry.py``) refuses to send a lookup until it has claimed +such a decision for its own argv (``claim_launch``), so a launcher Claude Code runs without the hook (``--bare`` on +2.1.312 skips ``--settings`` hooks) sends nothing: its call is recorded ``unguarded`` with ``sent: false``. + This is a filter over what the model typed, not a proof that no private material leaves the machine: a model can paraphrase code, spread it (or one credential) over several short queries, encode it (base64, hex, URL or percent encoding, ROT13, one character per word), or quote a line it wrote during the run (the tree hash set is taken @@ -113,6 +118,11 @@ def _beside(name: str): SHELL_SYNTAX, ERROR) DECISIONS = "decisions" DECISION_FIELDS = ("tool_use_id", "tool", "query_digest", "rule") # what the MAC covers +# Present only on a launcher call the hook passed: the SHA-256 of the arguments it read (``arguments_digest``), so the +# launcher can find the decision for its own argv before it sends anything (``claim_launch``). Covered by the MAC +# when present. +DECISION_OPTIONAL = ("arguments_digest",) +LAUNCHES = "launches" # beside ``decisions/``: one empty file per decision a launcher run has claimed POLICY_FILE, TREE_FILE, SETTINGS_FILE = "policy.json", "tree_lines.txt", "settings.json" # What the host hands the hook through Claude Code's environment, which a command hook inherits (the hooks reference # documents no per-hook ``env`` field): the per-call MAC key and the SHA-256 of the policy and tree files it wrote. @@ -470,19 +480,27 @@ def verify(value, given, key: str) -> bool: return bool(key) and isinstance(given, str) and hmac.compare_digest(given, sign(value, key)) +def _signed(entry: dict) -> dict: + """What a decision's MAC covers: ``DECISION_FIELDS``, and each ``DECISION_OPTIONAL`` field the entry holds.""" + return {field: entry.get(field) for field in DECISION_FIELDS + tuple( + field for field in DECISION_OPTIONAL if entry.get(field) is not None)} + + def mac(entry: dict, key: str) -> str: - """HMAC-SHA256 of a decision's four fields (``DECISION_FIELDS``, as canonical JSON) under the per-call key.""" - return sign({field: entry.get(field) for field in DECISION_FIELDS}, key) + """HMAC-SHA256 of a decision's four fields (``DECISION_FIELDS``) and its ``arguments_digest`` when it has one, + as canonical JSON, under the per-call key.""" + return sign(_signed(entry), key) def authentic(entry: dict, key: str) -> bool: """True only when ``entry`` carries a MAC that verifies under ``key``; a missing or wrong MAC is not.""" - return verify({field: entry.get(field) for field in DECISION_FIELDS}, entry.get("mac"), key) + return verify(_signed(entry), entry.get("mac"), key) def read_decisions(directory: Path) -> list[dict]: """The decisions in ``decisions/``, each ``{tool_use_id, tool, query_digest, rule, mac}`` (``mac`` None when - absent); malformed files are skipped. None is trusted until ``authentic`` verifies it.""" + absent), plus ``arguments_digest`` on a launcher call the hook passed; malformed files are skipped. None is + trusted until ``authentic`` verifies it.""" found = [] folder = Path(directory) / DECISIONS for file in sorted(folder.glob("*.json")) if folder.is_dir() else []: @@ -493,24 +511,85 @@ def read_decisions(directory: Path) -> list[dict]: if (isinstance(item, dict) and item.get("rule") in RULES + (PASS,) and isinstance(item.get("tool"), str) and isinstance(item.get("query_digest"), str) and isinstance(item.get("tool_use_id"), (str, type(None)))): found.append({**{k: item[k] for k in DECISION_FIELDS}, + **{k: item[k] for k in DECISION_OPTIONAL if isinstance(item.get(k), str)}, "mac": item["mac"] if isinstance(item.get("mac"), str) else None}) return found -def _record(directory: Path, call: dict, rule: str, key: str) -> None: - tool_use_id = call.get("tool_use_id") if isinstance(call.get("tool_use_id"), str) else None - tool = call.get("tool_name") if isinstance(call.get("tool_name"), str) else "" - digest = query_digest(call.get("tool_input")) - entry = {"tool_use_id": tool_use_id, "tool": tool[:64], "query_digest": digest, "rule": rule} +def arguments_digest(arguments) -> str: + """SHA-256 of a launcher call's arguments as canonical JSON (a list of strings): what the hook read from the + command (``bash_arguments``) and what the launcher receives as its argv, which agree for every command the hook + passes (plain words and quoted strings, no expansion).""" + return hashlib.sha256(utf8(canonical([str(word) for word in arguments]))).hexdigest() + + +def write_decision(directory: Path, entry: dict, key: str) -> None: + """Write one decision into ``decisions/`` with its MAC under ``key`` (none without a key), atomically.""" + entry = dict(entry) if key: entry["mac"] = mac(entry, key) - name = hashlib.sha256(utf8(canonical([tool_use_id, tool, digest, os.getpid(), os.urandom(8).hex()]))).hexdigest() - folder = directory / DECISIONS + name = hashlib.sha256(utf8(canonical([entry.get("tool_use_id"), entry.get("tool"), entry.get("query_digest"), + os.getpid(), os.urandom(8).hex()]))).hexdigest() + folder = Path(directory) / DECISIONS temporary = folder / f".{name}.tmp" temporary.write_text(json.dumps(entry), encoding="utf-8") os.replace(temporary, folder / f"{name}.json") +def _record(directory: Path, call: dict, rule: str, key: str, arguments: list[str] | None = None) -> None: + tool_use_id = call.get("tool_use_id") if isinstance(call.get("tool_use_id"), str) else None + tool = call.get("tool_name") if isinstance(call.get("tool_name"), str) else "" + entry = {"tool_use_id": tool_use_id, "tool": tool[:64], "query_digest": query_digest(call.get("tool_input")), + "rule": rule} + if arguments is not None: + entry["arguments_digest"] = arguments_digest(arguments) + write_decision(directory, entry, key) + + +NO_GUARD = "no_guard" # the launcher's environment holds no guard key or launcher: not a guarded call +NO_DECISION = "no_decision" # no verified pass decision names these exact arguments +DECISION_USED = "decision_used" # every such decision was already claimed by another launcher run +CLAIM_FAILED = "claim_failed" # the guard's directory could not be read or written + + +def claim_launch(arguments, environ=None) -> tuple[str | None, str]: + """Before the GitHits launcher sends anything: ``(None, tool_use_id)`` when the query guard's hook passed a + launcher call with exactly these ``arguments`` (a decision in ``decisions/`` beside the launcher, ``rule: pass``, + a shell tool, ``arguments_digest`` equal to this argv's, and a MAC that verifies under ``SYBERLABS_GUARD_KEY``), + which this run then claims (an empty file in ``launches/`` created exclusively, named by the decision's tool use + id): one decision, one launch. Otherwise ``(reason, detail)``: ``no_guard`` (no key or launcher in the + environment, so no hook could have decided), ``no_decision``, ``decision_used`` or ``claim_failed``. + + The guard's directory is the launcher's own (``SYBERLABS_GUARD_LAUNCHER``'s parent): the provider writes the + launcher there, beside the hook's ``decisions/``. Under ``claude --bare`` no ``--settings`` hook runs (Claude + Code 2.1.312, live, 2026-10-30), so there is no decision and the launcher refuses before any request leaves.""" + environ = os.environ if environ is None else environ + key, launcher = environ.get(KEY_ENV, ""), environ.get(LAUNCHER_ENV, "") + if not key or not launcher: + return NO_GUARD, (f"{KEY_ENV} and {LAUNCHER_ENV} are not both set: the launcher runs only inside a provider " + "call whose query guard passed it; operator lookups go through `syberlabs githits`") + directory = Path(launcher).parent + digest = arguments_digest(arguments) + try: + passed = [d for d in read_decisions(directory) + if d["rule"] == PASS and d["tool"] in SHELL_TOOLS and isinstance(d["tool_use_id"], str) + and d.get("arguments_digest") == digest and authentic(d, key)] + if not passed: + return NO_DECISION, "no verified query-guard decision passed a launcher call with these exact arguments" + folder = directory / LAUNCHES + folder.mkdir(exist_ok=True) + for decision in passed: + name = hashlib.sha256(utf8(decision["tool_use_id"])).hexdigest() + try: + os.close(os.open(folder / name, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)) + except FileExistsError: + continue + return None, decision["tool_use_id"] + except OSError as exc: + return CLAIM_FAILED, f"the guard's directory could not be read or written: {exc.strerror or type(exc).__name__}" + return DECISION_USED, "every query-guard decision for these arguments was already used by another launcher run" + + def _matching(path: Path, expected: str) -> bytes: """The file's bytes when their SHA-256 is the digest the host gave; otherwise the guard refuses to judge.""" data = path.read_bytes() @@ -526,6 +605,7 @@ def run(directory: Path, stdin, stderr, environ=None) -> int: environ = os.environ if environ is None else environ key = environ.get(KEY_ENV, "") call: dict = {} + arguments = None # a passed launcher call's arguments, for its decision's ``arguments_digest`` try: raw = stdin.buffer.read() if hasattr(stdin, "buffer") else stdin.read() call = json.loads(raw.decode("utf-8") if isinstance(raw, bytes) else raw) # Claude Code writes UTF-8 @@ -537,12 +617,14 @@ def run(directory: Path, stdin, stderr, environ=None) -> int: raise ValueError("no tool_use_id: no decision could be matched to this call") rules = json.loads(_matching(directory / POLICY_FILE, environ.get(POLICY_DIGEST_ENV, "")).decode("utf-8")) tree = set(_matching(directory / TREE_FILE, environ.get(TREE_DIGEST_ENV, "")).decode("utf-8").split()) - rule = decide(call, max_lines=rules["max_lines"], max_chars=rules["max_chars"], tree=tree, - launcher=environ.get(LAUNCHER_ENV) or None) + launcher = environ.get(LAUNCHER_ENV) or None + rule = decide(call, max_lines=rules["max_lines"], max_chars=rules["max_chars"], tree=tree, launcher=launcher) + if rule == PASS and call.get("tool_name") in SHELL_TOOLS: + arguments = bash_arguments(call["tool_input"].get("command"), launcher)[1] except Exception: # noqa: BLE001 a guard that cannot judge blocks - rule = ERROR + rule, arguments = ERROR, None try: - _record(directory, call if isinstance(call, dict) else {}, rule, key) + _record(directory, call if isinstance(call, dict) else {}, rule, key, arguments) except Exception: # noqa: BLE001 an unrecorded pass would read as unguarded; an unrecorded block still blocks if rule == PASS: rule = ERROR diff --git a/tests/test_claude_code.py b/tests/test_claude_code.py index f1b7f25..7e67419 100644 --- a/tests/test_claude_code.py +++ b/tests/test_claude_code.py @@ -32,9 +32,9 @@ from unittest import mock from spec.validate import validate_events -from syberlabs import Rejected, claude_code, githits, metrics, query_guard +from syberlabs import Rejected, claude_code, githits, githits_pace, metrics, query_guard from syberlabs.canonical import canonical -from syberlabs.claude_code import (BIN_ENV, CODE_TOOLS, DEFAULT_TOOLS, GITHITS_ENTRY, LOGIN_LANE_DROPPED_ENV, LOOKUP, +from syberlabs.claude_code import (BIN_ENV, CODE_TOOLS, DEFAULT_TOOLS, GITHITS_ENTRY, HOOKED, LOGIN_LANE_DROPPED_ENV, LOOKUP, SKILL_PATH, THIRD_PARTY_SWITCHES, ClaudeCodeProvider, Transcript, code_tools, githits_skill, instruction_file, parse_tools, query_text, version_tuple, write_launcher) @@ -43,7 +43,7 @@ from syberlabs.gitspace import IDENTITY from syberlabs.providers import RepairProvider from tests.test_build_thread import GOOD, REPORT, RepoCase -from tests.test_githits import FakeGitHits +from tests.test_githits import FakeGitHits, launch_guard PY = sys.executable BROKEN = {**GOOD, "src/report.py": "def rows():\n return []\n"} # tests fail: the report rows are gone @@ -78,7 +78,9 @@ settings, guard_dir = json.loads(settings_path.read_text(encoding="utf-8")), settings_path.parent hook_errors = [] key = os.environ.get("SYBERLABS_GUARD_KEY", "") -seen = {"guard_key": len(key), "policy_writable": None, "key_in_files": None} +seen = {"guard_key": len(key), "policy_writable": None, "key_in_files": None, + "config_dir": None if "CLAUDE_CONFIG_DIR" not in os.environ else sorted(os.listdir(os.environ["CLAUDE_CONFIG_DIR"])), + "home": None if "HOME" not in os.environ or not os.path.isdir(os.environ["HOME"]) else os.environ["HOME"]} if guard_dir is not None: policy = guard_dir / "policy.json" seen["policy_writable"] = bool(stat.S_IMODE(os.stat(policy).st_mode) & stat.S_IWUSR) @@ -286,7 +288,8 @@ def test_the_change_usage_turns_tools_and_lookups_come_from_the_worktree_and_the self.assertEqual(signal["tool_usage"], {"web_search": 1, "web_fetch": 0, "githits": 1, "other": {"Edit": 1}}) self.assertEqual((signal["lane"], signal["cost_basis"], signal["estimated_usd"]), ("subscription_local", "none", 0.0421)) self.assertEqual(signal["harness"], {"name": "claude-code", "version": "2.1.309", "model": "claude-opus-5", - "effort": "high", "auth_source": "none", "installed_version": "2.1.309"}) + "effort": "high", "auth_source": "none", "installed_version": "2.1.309", + "lane": "subscription_local"}) self.assertNotIn("model_usage_by_model", signal, "one model served the run") self.assertEqual(signal["guard"], {**POLICY, "policy_digest": POLICY_DIGEST}, "the guard's thresholds are host policy, recorded on the candidate with the policy file's digest") @@ -449,6 +452,78 @@ def test_bare_routed_elsewhere_is_provider_3p_and_api_is_refused(self): api.environment() self.assertEqual(caught.exception.code, "provider_lane_mismatch", "checked again at each call") + def test_api_hooked_runs_without_bare_on_the_key_with_an_empty_config_directory_and_home(self): + """guard-lane, item 2: Claude Code 2.1.312 runs no --settings hook under --bare, so the API key needs a lane + without it: no --bare, --disable-slash-commands, a fresh empty CLAUDE_CONFIG_DIR and HOME per call, the key + kept, every other credential, login and routing variable removed, billed like api.""" + budget = {"max_candidates": 6, "max_evaluations": 6, "max_seconds": 600, "prices": PRICES, "max_dollars": 1.0} + kit = self.kit(self.contract(**{"evolution.budget": budget})) + thread = kit.start("Add a CSV export", paths=["src/", "tests/"]) + ambient = {"ANTHROPIC_API_KEY": "test-placeholder", "CLAUDE_CODE_OAUTH_TOKEN": "login", + "ANTHROPIC_AUTH_TOKEN": "t", "CLAUDE_CONFIG_DIR": str(self.root.parent / "operator-claude"), + "HTTPS_PROXY": "http://proxy.example.test:3128", "AWS_PROFILE": "x", "ANTHROPIC_MODEL": "claude-opus-5", + "SYBERLABS_CERTIFICATE_SEED": "00"} + with mock.patch.dict(os.environ, ambient): + provider = ClaudeCodeProvider(kit.repo, lane=HOOKED, trusted_tree=True, executable=self.stand_in( + "hooked", log_env=True), max_calls=1) + [candidate] = thread.propose(provider) + [run] = self.runs("hooked") + argv, env = run["argv"], run["env"] + self.assertNotIn("--bare", argv) + self.assertEqual(argv[:2], ["-p", "--disable-slash-commands"]) + for flag in ("--restricted", "--strict-mcp-config", "--settings"): + self.assertIn(flag, argv) + self.assertEqual(argv[argv.index("--setting-sources") + 1], "user") + self.assertEqual((env["ANTHROPIC_API_KEY"], env.get("ANTHROPIC_MODEL")), ("test-placeholder", "claude-opus-5")) + for name in ("CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_AUTH_TOKEN", "HTTPS_PROXY", "AWS_PROFILE", + "SYBERLABS_CERTIFICATE_SEED"): + self.assertNotIn(name, env) + config, home = Path(env["CLAUDE_CONFIG_DIR"]), Path(env["HOME"]) + self.assertNotEqual(config, Path(ambient["CLAUDE_CONFIG_DIR"]), "not the operator's configuration") + self.assertEqual(run["config_dir"], [], "empty when Claude Code started") + self.assertEqual(run["home"], env["HOME"], "a directory of its own") + self.assertEqual(config.parent, home.parent, "both in the call's guard directory") + self.assertEqual(config.parent, Path(argv[argv.index("--settings") + 1]).parent) + self.assertNotIn(Path(run["cwd"]).resolve(), config.resolve().parents, "outside the worktree") + self.assertFalse(config.parent.exists(), "removed after the call") + if os.name == "nt": + self.assertEqual(env["USERPROFILE"], env["HOME"]) + self.assertEqual((candidate.signal["lane"], candidate.signal["cost_basis"], candidate.signal["harness"]["lane"]), + (HOOKED, "billed", HOOKED)) + body = [e["body"] for e in thread.history() if e["kind"] == "candidate_registered"][-1] + self.assertEqual((body["spend"]["dollars"], body["spend"]["host_observed"]), (COST, True), "priced as api") + + def test_api_hooked_refuses_bare_a_routed_environment_and_a_missing_key(self): + kit = self.kit() + with mock.patch.dict(os.environ, {"ANTHROPIC_API_KEY": "test-placeholder"}): + for kwargs, code, words in ( + ({"bare": True, "lane": HOOKED}, "invalid_provider", "drop --bare"), + ({"lane": "api", "trusted_tree": True}, "invalid_provider", f"use --lane {HOOKED}"), + ({"lane": HOOKED}, "untrusted_tree", "--trusted-tree")): + with self.subTest(kwargs=kwargs), self.assertRaises(Rejected) as caught: + ClaudeCodeProvider(kit.repo, executable=VERSION_ONLY, **kwargs) + self.assertEqual(caught.exception.code, code) + self.assertIn(words, caught.exception.detail) + for name, value in (("ANTHROPIC_BASE_URL", "https://gw.example.test"), ("CLAUDE_CODE_USE_BEDROCK", "1")): + with self.subTest(name=name), mock.patch.dict(os.environ, {name: value}), \ + self.assertRaises(Rejected) as caught: + ClaudeCodeProvider(kit.repo, lane=HOOKED, trusted_tree=True, executable=VERSION_ONLY) + self.assertEqual(caught.exception.code, "invalid_provider") + self.assertIn(name, caught.exception.detail) + hooked = ClaudeCodeProvider(kit.repo, lane=HOOKED, trusted_tree=True, executable=VERSION_ONLY) + with mock.patch.dict(os.environ, {"ANTHROPIC_BASE_URL": "https://gw.example.test"}), \ + self.assertRaises(Rejected) as caught: + hooked.environment() + self.assertEqual(caught.exception.code, "provider_lane_mismatch", "checked again at each call") + os.environ.pop("ANTHROPIC_API_KEY") + with self.assertRaises(Rejected) as caught: + hooked.environment() + self.assertEqual(caught.exception.code, "provider_lane_mismatch") + with self.assertRaises(Rejected) as caught: + ClaudeCodeProvider(kit.repo, lane=HOOKED, trusted_tree=True, executable=VERSION_ONLY) + self.assertEqual(caught.exception.code, "invalid_provider") + self.assertIn("ANTHROPIC_API_KEY", caught.exception.detail) + def test_a_non_bare_run_on_an_undeclared_tree_is_refused_and_says_why(self): kit = self.kit() with self.assertRaises(Rejected) as caught: @@ -919,28 +994,44 @@ def setUp(self): self.addCleanup(self.fake.close) self.token = "ghtest_" + secrets.token_hex(16) env = {k: v for k, v in os.environ.items() if not claude_code.githits_variable(k)} - env.update({githits.TOKEN_ENV: self.token, githits.URL_ENV: self.fake.url}) + env.update({githits.TOKEN_ENV: self.token, githits.URL_ENV: self.fake.url, + "ANTHROPIC_API_KEY": "test-placeholder"}) patcher = mock.patch.dict(os.environ, env, clear=True) patcher.start() self.addCleanup(patcher.stop) - def arm_b(self, kit, claude, **kwargs): + def arm_a(self, kit, claude, **kwargs): + """Arm A on the treatment's lane: api-hooked, no --githits.""" kwargs.setdefault("max_calls", 1) - return ClaudeCodeProvider(kit.repo, bare=True, executable=claude, githits=True, **kwargs) + return ClaudeCodeProvider(kit.repo, lane=HOOKED, trusted_tree=True, executable=claude, **kwargs) + + def arm_b(self, kit, claude, **kwargs): + """Arm B: the same lane with --githits (never --bare: no hook runs there).""" + return self.arm_a(kit, claude, githits=True, **kwargs) def test_arm_a_is_unchanged_and_arm_b_differs_only_in_the_githits_additions_and_the_skill(self): - """Arm A's argv is byte-identical to before. Its environment is the caller's minus every GitHits variable - (githits-pilot-hardening, item 2: least privilege over a byte-identical environment), plus the guard's three.""" + """A bare arm A's argv is byte-identical to before. Both pilot arms now run api-hooked (guard-lane, item 2), + and there they differ only in the GitHits additions. Arm A's environment is the caller's minus every GitHits + variable (githits-pilot-hardening, item 2: least privilege over a byte-identical environment) and the lane's + dropped variables, plus the guard's three and the lane's fresh config directory and home.""" self.enterContext(mock.patch.dict(os.environ, OPERATOR_GITHITS)) kit = self.kit() claude = self.stand_in("arms", lookups=[launcher_lookup(SEARCH)], log_env=True) - arm_a = ClaudeCodeProvider(kit.repo, bare=True, executable=claude, max_calls=1) - self.assertEqual(arm_a.command("S"), [*claude, "-p", "--bare", "--output-format", "stream-json", "--verbose", - "--permission-mode", "dontAsk", "--restricted", "--tools", - "Read,Edit,Write,Glob,Grep", "--allowedTools", "Read,Edit,Write,Glob,Grep", - "--setting-sources", "user", "--strict-mcp-config", - "--no-session-persistence", "--append-system-prompt", TODAY_NOTE, - "--settings", "S"], "arm A's argv is what main built before this packet") + bare = ClaudeCodeProvider(kit.repo, bare=True, executable=claude, max_calls=1) + self.assertEqual(bare.command("S"), [*claude, "-p", "--bare", "--output-format", "stream-json", "--verbose", + "--permission-mode", "dontAsk", "--restricted", "--tools", + "Read,Edit,Write,Glob,Grep", "--allowedTools", "Read,Edit,Write,Glob,Grep", + "--setting-sources", "user", "--strict-mcp-config", + "--no-session-persistence", "--append-system-prompt", TODAY_NOTE, + "--settings", "S"], "a bare arm A's argv is what main built before") + arm_a = self.arm_a(kit, claude) + self.assertEqual(arm_a.command("S"), [*claude, "-p", "--disable-slash-commands", "--output-format", + "stream-json", "--verbose", "--permission-mode", "dontAsk", "--restricted", + "--tools", "Read,Edit,Write,Glob,Grep", "--allowedTools", + "Read,Edit,Write,Glob,Grep", "--setting-sources", "user", + "--strict-mcp-config", "--no-session-persistence", + "--append-system-prompt", TODAY_NOTE, "--settings", "S"], + "api-hooked: no --bare, and no skill or command loads") arm_b = self.arm_b(kit, claude, thread_label="thread-ab") for name, provider in (("a", arm_a), ("b", arm_b)): thread = kit.start(f"Add a CSV export ({name})", paths=["src/", "tests/"]) @@ -966,21 +1057,30 @@ def test_arm_a_is_unchanged_and_arm_b_differs_only_in_the_githits_additions_and_ self.assertTrue(launcher.endswith("/githits") and Path(launcher).is_absolute() and "\\" not in launcher) self.assertNotIn(Path(run_b["cwd"]).resolve(), Path(launcher).resolve().parents, "outside the worktree") env_a, env_b = run_a["env"], run_b["env"] - per_call = {query_guard.KEY_ENV} # a fresh MAC key per call + isolation = {"CLAUDE_CONFIG_DIR", "HOME"} | ({"USERPROFILE"} if os.name == "nt" else set()) + per_call = {query_guard.KEY_ENV} | isolation # a fresh MAC key, config directory and home per call dropped = {k for k in os.environ if k.upper().startswith(("GITHITS_", "SYBERLABS_GITHITS_"))} self.assertEqual(dropped, {githits.TOKEN_ENV, githits.URL_ENV, *OPERATOR_GITHITS}) - self.assertEqual({k for k in env_a if k not in os.environ}, {query_guard.KEY_ENV, query_guard.POLICY_DIGEST_ENV, - query_guard.TREE_DIGEST_ENV}, - "arm A: the caller's environment plus the guard's three") - self.assertEqual({k: v for k, v in env_a.items() if k in os.environ}, { - k: v for k, v in os.environ.items() if k not in ("SYBERLABS_CERTIFICATE_SEED",) and k not in dropped}, - "minus every GITHITS_* and SYBERLABS_GITHITS_* variable, the token included") + lane_dropped = {k for k in os.environ if k != "ANTHROPIC_API_KEY" and ( + claude_code._dropped(k) or k in claude_code.HOOKED_DROPPED_ENV)} + self.assertEqual({k for k in env_a if k not in os.environ}, + {query_guard.KEY_ENV, query_guard.POLICY_DIGEST_ENV, query_guard.TREE_DIGEST_ENV} + | (isolation - set(os.environ)), + "arm A: the caller's environment plus the guard's three and the lane's config and home") + self.assertEqual({k: v for k, v in env_a.items() if k in os.environ and k not in isolation}, { + k: v for k, v in os.environ.items() if k not in ("SYBERLABS_CERTIFICATE_SEED",) and k not in dropped + and k not in lane_dropped and k not in isolation}, + "minus every GITHITS_* and SYBERLABS_GITHITS_* variable, the token included, and the lane's dropped ones") + self.assertEqual(env_a["ANTHROPIC_API_KEY"], env_b["ANTHROPIC_API_KEY"], "the key, in both arms") self.assertFalse([k for k in env_a if k.upper().startswith(("GITHITS_", "SYBERLABS_GITHITS_"))]) self.assertTrue(all(claude_code.githits_variable(k) for k in dropped)) added = {k: env_b[k] for k in env_b if k not in env_a} self.assertEqual(set(added), {query_guard.LAUNCHER_ENV, githits.RECORD_DIR_ENV, githits.RECORD_KEY_ENV, githits.SESSION_ENV, githits.AGENT_ENV, githits.LICENSES_ENV, - githits.EXCLUDE_ENV, githits.TOKEN_ENV, githits.URL_ENV, *OPERATOR_GITHITS}) + githits.EXCLUDE_ENV, githits_pace.PACE_DIR_ENV, githits.TOKEN_ENV, + githits.URL_ENV, *OPERATOR_GITHITS}) + self.assertEqual(added[githits_pace.PACE_DIR_ENV], str(githits_pace.directory(os.environ)), + "the host's pacing directory, pinned for the launcher") self.assertEqual({k: env_b[k] for k in dropped}, {k: os.environ[k] for k in dropped}, "arm B as before: the operator's GitHits variables pass through unchanged") self.assertEqual({k for k in env_a if k not in env_b}, set()) @@ -988,7 +1088,7 @@ def test_arm_a_is_unchanged_and_arm_b_differs_only_in_the_githits_additions_and_ self.assertEqual((added[query_guard.LAUNCHER_ENV], added[githits.SESSION_ENV], added[githits.AGENT_ENV], added[githits.LICENSES_ENV], added[githits.EXCLUDE_ENV]), (launcher, "thread-ab-b", "syberfactory/b", ",".join(githits.LicensePolicy().allowed), - "syberlabs")) + "syberlabs,sykosyber,pypi:syberwork,deepwiki.com,deepwiki.org")) self.assertEqual(len(added[githits.RECORD_KEY_ENV]), 64) self.assertEqual(env_b[githits.TOKEN_ENV], self.token, "the operator's token, passed through") self.assertNotIn(Path(run_b["cwd"]).resolve(), Path(added[githits.RECORD_DIR_ENV]).resolve().parents) @@ -999,6 +1099,36 @@ def test_arm_a_is_unchanged_and_arm_b_differs_only_in_the_githits_additions_and_ self.assertEqual(len(self.fake.requests), 1, "only arm B's launcher call reached GitHits") self.assertIn("Permission to use Bash", run_a["bash"][0], "in arm A the same Bash call is denied") + def test_githits_refuses_bare_where_no_hook_runs(self): + """guard-lane, item 2: the live canary (Claude Code 2.1.312, 2026-10-30) ran the launcher under --bare with no + hook, and the lookup left. --githits is refused with --bare, in Python and from the terminal.""" + kit = self.kit() + for lane in ("api", "provider_3p"): + with self.subTest(lane=lane), self.assertRaises(Rejected) as caught: + ClaudeCodeProvider(kit.repo, bare=True, lane=lane, executable=VERSION_ONLY, githits=True) + self.assertEqual(caught.exception.code, "invalid_provider") + for words in ("--bare", "2.1.312", "--settings", f"--lane {HOOKED}"): + self.assertIn(words, caught.exception.detail) + ClaudeCodeProvider(kit.repo, bare=True, executable=VERSION_ONLY) # arm A may still run bare + self.arm_b(kit, VERSION_ONLY) # and the treatment builds where its guard runs + + def test_a_launcher_claude_code_runs_without_the_hook_sends_nothing(self): + """guard-lane, item 1, the canary's case through the provider: a Claude Code that skips the --settings hook + runs the launcher; the launcher finds no decision and refuses before the network. The fake sees no request, + the record says unguarded and sent false, and the search stops provider_guard_inactive.""" + kit = self.kit() + thread = kit.start("Add a CSV export", paths=["src/", "tests/"]) + claude = self.stand_in("no-hook", hooks=False, lookups=[launcher_lookup(SEARCH)]) + self.assertEqual(thread.propose(self.arm_b(kit, claude)), []) + self.assertEqual(self.fake.requests, [], "nothing reached GitHits") + [run] = self.runs("no-hook") + self.assertIn("refused: unguarded: no_decision", run["bash"][0]) + search = [e["body"] for e in thread.history() if e["kind"] == "search_finished"][-1] + self.assertEqual((search["stopped"], search["error"]), ("error", "provider_guard_inactive")) + values = [b["value"] for b in self.lookup_facts(thread)] + self.assertEqual([(v["tool"], v["guard"]) for v in values], [("Bash", "unguarded"), ("githits.search", "unguarded")]) + self.assertEqual((values[1]["status"], values[1]["sent"], values[1]["ok"]), (githits.UNGUARDED, False, False)) + def test_an_mcp_config_run_without_githits_keeps_the_operators_githits_variables(self): """launcher-unicode-and-docs, item 2 (#135's verification): the GitHits route is ``githits=True`` or an ``--mcp-config``. A GitHits MCP server is Claude Code's child and may read its token from that environment, @@ -1141,27 +1271,32 @@ def test_code_running_tools_are_refused_except_the_launcher(self): for tools, mcp in ((["Read", "Bash"], None), (["Read", "PowerShell"], None), (["Edit", "Bash(git log *)"], None), (["Read", "bash"], config), (["Read", "Bash(/tmp/githits:*)"], None)): with self.subTest(tools=tools), self.assertRaises(Rejected) as caught: - ClaudeCodeProvider(kit.repo, bare=True, executable=VERSION_ONLY, githits=True, allowed_tools=tools, - mcp_config=mcp) + ClaudeCodeProvider(kit.repo, lane=HOOKED, trusted_tree=True, executable=VERSION_ONLY, githits=True, + allowed_tools=tools, mcp_config=mcp) self.assertEqual(caught.exception.code, "invalid_provider") self.assertIn("launcher", caught.exception.detail) self.assertIn("the guard still blocks every Bash and PowerShell command but the launcher", caught.exception.detail) for mcp in (None, config): - provider = ClaudeCodeProvider(kit.repo, bare=True, executable=VERSION_ONLY, githits=True, mcp_config=mcp) + provider = ClaudeCodeProvider(kit.repo, lane=HOOKED, trusted_tree=True, executable=VERSION_ONLY, + githits=True, mcp_config=mcp) command = provider.command("S", "C:/tmp/syberlabs-guard-1/githits") tools = command[command.index("--allowedTools") + 1].split(",") self.assertEqual(tools, list(DEFAULT_TOOLS) + ["Bash(C:/tmp/syberlabs-guard-1/githits:*)"]) self.assertEqual(code_tools(tools), ["Bash"], "the launcher's entry is the only code-running one") self.assertEqual(command[command.index("--tools") + 1], ",".join(DEFAULT_TOOLS) + ",Bash") - ClaudeCodeProvider(kit.repo, bare=True, executable=VERSION_ONLY, githits=True, allowed_tools=["Read", "Bash"], - allow_code_tools_with_mcp=True) # the operator's recorded override still builds it + self.arm_b(kit, VERSION_ONLY, allowed_tools=["Read", "Bash"], + allow_code_tools_with_mcp=True) # the operator's recorded override still builds it with mock.patch.dict(os.environ, {githits.TOKEN_ENV: ""}), self.assertRaises(Rejected) as caught: - ClaudeCodeProvider(kit.repo, bare=True, executable=VERSION_ONLY, githits=True) + self.arm_b(kit, VERSION_ONLY) self.assertIn(githits.TOKEN_ENV, caught.exception.detail) for arm in ("", "a/b", "x" * 17): with self.assertRaises(Rejected): - ClaudeCodeProvider(kit.repo, bare=True, executable=VERSION_ONLY, githits=True, arm=arm) + self.arm_b(kit, VERSION_ONLY, arm=arm) + with mock.patch.dict(os.environ, {githits_pace.PACE_DIR_ENV: "relative/pace"}), \ + self.assertRaises(Rejected) as caught: + self.arm_b(kit, VERSION_ONLY) + self.assertIn(githits_pace.PACE_DIR_ENV, caught.exception.detail, "an unusable pacing directory, at build") def test_the_launcher_is_read_only_outside_the_worktree_and_runs_the_host_client(self): with tempfile.TemporaryDirectory() as folder: @@ -1179,6 +1314,8 @@ def test_the_launcher_is_read_only_outside_the_worktree_and_runs_the_host_client env = {**os.environ, "PYTHONPATH": str(decoy), githits.RECORD_DIR_ENV: str(records), githits.RECORD_KEY_ENV: "k" * 64} lookup = ["search", "retry backoff", "--target", "pypi:requests@2.32.3"] + guard = self.enterContext(launch_guard(lookup, ["search", "retry", "--target", "pypi:requests@2.32.3"])) + env.update(guard) # the query guard's hook passed both lookups below done = subprocess.run([sys.executable, "-I", str(GITHITS_ENTRY), *lookup], cwd=decoy, env=env, capture_output=True, timeout=120) self.assertEqual(done.returncode, 0, done.stderr[-400:]) @@ -1259,8 +1396,7 @@ def test_the_exclusion_replace_switch_never_reaches_claude_code_in_either_arm(se self.enterContext(mock.patch.dict(os.environ, {githits.EXCLUDE_ENV: "acme", githits.EXCLUDE_REPLACE_ENV: "1"})) kit = self.kit() claude = self.stand_in("replace", lookups=[launcher_lookup("list github:SyberLabs/SyberWork")], log_env=True) - for name, provider in (("a", ClaudeCodeProvider(kit.repo, bare=True, executable=claude, max_calls=1)), - ("b", self.arm_b(kit, claude))): + for name, provider in (("a", self.arm_a(kit, claude)), ("b", self.arm_b(kit, claude))): thread = kit.start(f"Add a CSV export ({name})", paths=["src/", "tests/"]) thread.propose(provider) if name == "b": @@ -1270,7 +1406,9 @@ def test_the_exclusion_replace_switch_never_reaches_claude_code_in_either_arm(se for run in (run_a, run_b): self.assertNotIn(githits.EXCLUDE_REPLACE_ENV, run["env"]) self.assertNotIn(githits.EXCLUDE_ENV, run_a["env"], "arm A holds no GitHits variable") - self.assertEqual(run_b["env"][githits.EXCLUDE_ENV], "syberlabs,acme", "SyberLabs first, the operator's added") + self.assertEqual(run_b["env"][githits.EXCLUDE_ENV], + "syberlabs,sykosyber,pypi:syberwork,deepwiki.com,deepwiki.org,acme", + "the default deny-list first, the operator's added") self.assertEqual(self.fake.requests, [], "the SyberLabs target was refused before any request") def test_powershell_is_judged_by_the_launcher_rule_under_the_override(self): @@ -1319,6 +1457,11 @@ def run_cli(*args, env=None): env = {BIN_ENV: " ".join(self.stand_in("cli-githits", lookups=[launcher_lookup(SEARCH)]))} code, text, problem = run_cli("propose", "--claude-code", "--bare", "--max-calls", "1", "--githits", "--githits-arm", "on", env=env) + self.assertNotEqual(code, 0, "--githits refuses --bare from the terminal too") + self.assertIn("refusing --githits with --bare", text + problem) + self.assertEqual(self.fake.requests, []) + code, text, problem = run_cli("propose", "--claude-code", "--lane", HOOKED, "--trusted-tree", "--max-calls", + "1", "--githits", "--githits-arm", "on", env=env) self.assertEqual(code, 0, text + problem) thread = (self.root / ".syberlabs" / "current").read_text().strip() self.assertEqual(self.fake.requests[0]["headers"]["x-githits-session-id"], f"{thread}-on") diff --git a/tests/test_eval_adversarial.py b/tests/test_eval_adversarial.py index 81eea0f..2bab25f 100644 --- a/tests/test_eval_adversarial.py +++ b/tests/test_eval_adversarial.py @@ -473,7 +473,8 @@ def setUp(self): self.claude.write_text(f"GOOD = {GOOD!r}\n" + textwrap.dedent(CLAUDE), encoding="utf-8") removed = ("ANTHROPIC_BASE_URL", "CLAUDE_CODE_USE_BEDROCK", "CLAUDE_CODE_USE_VERTEX", "CLAUDE_CODE_USE_FOUNDRY", "CLAUDE_CODE_USE_MANTLE", "CLAUDE_CODE_USE_ANTHROPIC_AWS", "CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST") - patcher = mock.patch.dict(os.environ, {"GITHITS_API_TOKEN": "test-not-a-token"}) + patcher = mock.patch.dict(os.environ, {"GITHITS_API_TOKEN": "test-not-a-token", + "ANTHROPIC_API_KEY": "test-not-a-key"}) patcher.start() self.addCleanup(patcher.stop) for name in removed: @@ -482,8 +483,8 @@ def setUp(self): def pilot(self, eval_id: str, *, declared: str, served: str) -> dict: written = tasks.import_fixtures(self.root / "tasks", ["pricing"]) executable = ["{python}", str(self.claude), str(self.log), served] - base = {"kind": "claude-code", "model": declared, "effort": "medium", "bare": True, "lane": "api", - "max_calls": 1, "executable": executable} + base = {"kind": "claude-code", "model": declared, "effort": "medium", "lane": "api-hooked", + "trusted_tree": True, "max_calls": 1, "executable": executable} # --githits refuses --bare doc = tiny_spec([w["id"] for w in written], {w["id"]: w["digest"] for w in written}, id=eval_id, arms=[dict(base, name="off", githits=False), dict(base, name="on", githits=True)]) doc["budget"] = dict(doc["budget"], prices={served: {"input_per_million": 2.0, "output_per_million": 10.0}}) diff --git a/tests/test_experiment_githits_api.py b/tests/test_experiment_githits_api.py index 452a864..8d7f619 100644 --- a/tests/test_experiment_githits_api.py +++ b/tests/test_experiment_githits_api.py @@ -249,7 +249,7 @@ def setUp(self): super().setUp() self.fake = FakeGitHits() self.addCleanup(self.fake.close) - # The api lane as an operator's machine has it: a key, no gateway or provider switch inherited from the + # The api-hooked lane (both arms; --githits refuses --bare) as an operator's machine has it: a key, no gateway or provider switch inherited from the # environment the suite runs in, and GitHits settings from nowhere but this test. env = {k: v for k, v in os.environ.items() if k not in THIRD_PARTY_SWITCHES and k not in GITHITS_SETTINGS} env.update({"ANTHROPIC_API_KEY": "test-placeholder", githits.TOKEN_ENV: "ghtest_" + secrets.token_hex(16), @@ -283,8 +283,8 @@ def run_arm(self, item, arm, tag, *, githits_on, name, first=None, repair=None, self.assertEqual(code, 0, text) self.assertNotIn("labelled_late", text) claude = self.stand_in(name, first=first, repair=repair, lookups=[list(entry) for entry in lookups]) - provider = ClaudeCodeProvider(kit.repo, bare=True, executable=claude, max_calls=3, githits=githits_on, - arm=arm, thread_label=thread.id, **options) + provider = ClaudeCodeProvider(kit.repo, lane="api-hooked", trusted_tree=True, executable=claude, max_calls=3, + githits=githits_on, arm=arm, thread_label=thread.id, **options) found = thread.propose(provider) receipt = thread.accept(found[-1].id, actor=RUNNER) self.assertEqual(receipt.status, "succeeded", receipt.reason) diff --git a/tests/test_githits.py b/tests/test_githits.py index 1d29566..0873ef7 100644 --- a/tests/test_githits.py +++ b/tests/test_githits.py @@ -31,7 +31,7 @@ from unittest import mock from urllib.parse import parse_qs, urlsplit -from syberlabs import cli, githits, githits_cli_entry +from syberlabs import cli, githits, githits_cli_entry, query_guard from syberlabs.canonical import canonical ROOT = Path(__file__).resolve().parents[1] @@ -41,7 +41,8 @@ DOCTOR_FIXTURE_SHA256 = "492fcb7f97838a2bed4dfc4618c3858cc8589787f1f3ff92422ffff3fdc8904c" UPSTREAM_SHA256 = "e58048e3e56d7b0d6346e24722614d2d8907748960f18f18204895a354ab8a6a" ENV_NAMES = (githits.TOKEN_ENV, githits.URL_ENV, githits.SESSION_ENV, githits.AGENT_ENV, githits.EXCLUDE_ENV, - githits.EXCLUDE_REPLACE_ENV, githits.LICENSES_ENV, githits.RECORD_DIR_ENV, githits.RECORD_KEY_ENV) + githits.EXCLUDE_REPLACE_ENV, githits.LICENSES_ENV, githits.RECORD_DIR_ENV, githits.RECORD_KEY_ENV, + query_guard.KEY_ENV, query_guard.LAUNCHER_ENV) # ----------------------------------------------------------------------------------------------- the pinned contract @@ -582,7 +583,7 @@ def test_owner_and_repository_locators_in_every_spelling(self): "https%3A%2F%2Fgithub.com%2FSyberLabs%2FSyberWork", "gitlab:SyberLabs/mirror"): with self.subTest(text=text): self.assertTrue(exclude.names(text)) - for text in ("https://github.com/psf/requests", "github:SyberLabsX/fork", "github:other/SyberLabs", + for text in ("https://github.com/psf/requests", "github:SyberLabsX/fork", "pypi:syberlabs-sdk", "https://github.com/syberlabs-archive/x", "npm:express@5.2.1", "site:docs.python.org/3", "https://notgithub.com/SyberLabs/x", ""): with self.subTest(text=text): @@ -603,7 +604,8 @@ def test_the_environment_adds_to_the_default_and_only_an_explicit_switch_replace self.assertTrue(custom.names("https://github.com/other/repo")) self.assertFalse(custom.names("https://github.com/other/else")) self.assertTrue(custom.names("github:SyberLabs/SyberWork"), "the default is kept") - self.assertEqual(custom.entries, (("syberlabs",), ("acme",), ("other", "repo"))) + self.assertEqual(custom.entries, (("syberlabs",), ("sykosyber",), ("pypi:syberwork",), ("deepwiki.com",), + ("deepwiki.org",), ("acme",), ("other", "repo"))) for switch in ("0", "", "yes", "true"): kept = githits.Exclusions.from_env({githits.EXCLUDE_ENV: "Acme", githits.EXCLUDE_REPLACE_ENV: switch}) self.assertTrue(kept.names("github:SyberLabs/x"), f"only 1 replaces, not {switch!r}") @@ -612,6 +614,61 @@ def test_the_environment_adds_to_the_default_and_only_an_explicit_switch_replace self.assertTrue(replaced.names("github:acme/x")) empty = githits.Exclusions.from_env({githits.EXCLUDE_ENV: " , ", githits.EXCLUDE_REPLACE_ENV: "1"}) self.assertTrue(empty.names("github:SyberLabs/x"), "replacing with nothing keeps the default") + + def test_the_default_deny_list_refuses_our_package_founder_forks_and_deepwiki(self): + """guard-lane (the pilot's pre-registration): besides SyberLabs, the default refuses this repository's PyPI + name (pyproject.toml: syberwork), github.com/sykosyber, a fork of a SyberLabs repository detectable by its + name under another owner (FORK_NAMES), and the DeepWiki front-ends, as targets and in result text.""" + exclude = githits.Exclusions.from_env({}) + self.assertEqual(githits.DEFAULT_EXCLUDE, ("SyberLabs", "sykosyber", "pypi:syberwork", "deepwiki.com", + "deepwiki.org")) + for text in ("pypi:syberwork", "pypi:SyberWork@0.1.0", "pypi: syberwork", "github:sykosyber/dotfiles", + "https://github.com/sykosyber/x", "https://github.com/someone/SyberWork", + "github:someone/syberfactory@main", "someone/SyberSDK", "github:other/SyberLabs", + "https://gitlab.com/mirror/syberwork.git", "site:deepwiki.com/SyberLabs/SyberWork", + "https://deepwiki.com/psf/requests", "https://www.deepwiki.org/x", "site:DEEPWIKI.COM", + "https://deepwiki.com:443/x", "https%3A%2F%2Fdeepwiki.com%2Fx"): + with self.subTest(refused=text): + self.assertTrue(exclude.names(text)) + for text in ("pypi:syberworkx", "pypi:requests", "npm:syberwork", "https://github.com/someone/syberworking", + "https://notdeepwiki.com/x", "site:docs.python.org/3", "https://github.com/psf/requests", + "https://notgithub.com/x/SyberWork"): + with self.subTest(kept=text): + self.assertFalse(exclude.names(text)) + for text in ("see https://pypi.org/project/syberwork/", "the page at deepwiki.com/psf/requests", + "[Source 1](https://github.com/fork/SyberWork/blob/1/a.py) (MIT)", + "(https://github.com/sykosyber/notes)"): + with self.subTest(mentioned=text): + self.assertTrue(exclude.mentioned(text)) + self.assertFalse(exclude.mentioned("syberwork, said without a host or registry")) + added = githits.Exclusions.from_env({githits.EXCLUDE_ENV: "npm:left-pad, example.org"}) + self.assertTrue(added.names("npm:left-pad@1.3.0") and added.names("https://docs.example.org/x")) + self.assertTrue(added.names("pypi:syberwork"), "the variable adds; the deny-list stays") + self.assertEqual(githits.Exclusions.from_env({githits.EXCLUDE_ENV: added.written()}).entries, added.entries, + "the written list reads back as itself") + for bad in ("foo:bar", "pypi:", "a..b", "x/y/z"): + with self.subTest(bad=bad), self.assertRaises(githits.GitHitsError): + githits.Exclusions([bad]) + + def test_the_deny_list_never_leaves_and_drops_what_points_at_it(self): + """Refused targets send nothing; a result or an example pointing at a denied place is dropped.""" + fake = FakeGitHits() + self.addCleanup(fake.close) + if True: + env = {githits.TOKEN_ENV: "ghtest_" + secrets.token_hex(8), githits.URL_ENV: fake.url} + client = githits.Client.from_env(env) + for target in ("pypi:syberwork", "site:deepwiki.com/psf/requests", "github:sykosyber/x"): + with self.subTest(target=target): + self.assertEqual(client.search("q", [target]).status, githits.EXCLUDED) + self.assertEqual(fake.requests, [], "nothing was sent") + fake.serve("/v1/searches", search_response([search_hit("a", "https://github.com/someone/SyberWork"), + search_hit("b", "https://github.com/psf/requests")])) + result = client.search("q", ["pypi:requests"]) + self.assertEqual((result.results_returned, result.results_dropped), (1, 1), "the fork's hit is dropped") + fake.serve("/v1/examples/search", {"result": "Use this.\n\nSee https://deepwiki.com/psf/requests\n", + "search_status": "complete", + "solution_id": "12345678-1234-4234-8234-123456789abc"}) + self.assertTrue(client.example("q").contamination_blocked, "an example citing DeepWiki is dropped whole") for value in ("", " , "): self.assertTrue(githits.Exclusions.from_env({githits.EXCLUDE_ENV: value}).names("github:SyberLabs/x")) with self.assertRaises(githits.GitHitsError): @@ -933,6 +990,38 @@ def test_only_links_under_a_sources_heading_or_on_a_code_host_are_sources_and_ne "[Source 1](https://github.com/a/b/blob/1/c.py) (MIT)"}), ["MIT"], "outside a sources section a non-code-host link is not a source, as before") + def test_a_singular_heading_a_heading_with_more_words_and_crlf_still_open_a_sources_section(self): + """guard-lane, item 4 (#153's verification): the heading parser took only ``References`` or ``Sources``, + alone, on an LF line, so a ``## Source`` heading, ``### Sources and references`` or a CRLF answer left its + links outside every sources section, where an unlicensed docs link counted for nothing.""" + read = githits.example_licenses + link = "\n- [a](https://docs.example.org/a)\n" # a docs link: a source only inside a sources section + for heading in ("## Source", "### Reference", "Reference:", "Source", "## Sources and references", + "### Reference links ###", "**Sources used:**", "## Sources"): + for newline in ("\n", "\r\n"): + with self.subTest(heading=heading, crlf=newline == "\r\n"): + text = ("```py\nx\n```\n" + heading + link).replace("\n", newline) + self.assertEqual(read({"result": text}), [None], "the unlicensed link is a source: dropped") + licensed = text.replace("/a)", "/a) (MIT)") + self.assertEqual(read({"result": licensed}), ["MIT"]) + self.assertEqual(read({"result": "```py\nx\n```\nThe sources below are informal." + link}), [None], + "a sentence is no heading: the link is not a source, and the code has none") + + def test_a_fenced_block_inside_a_blockquote_is_code(self): + """guard-lane, item 4 (#153's verification): ``_CODE`` matched a fence only at the margin, so an example whose + only code sat in a blockquote, with no source, read as carrying nothing to license and passed.""" + read = githits.example_licenses + for quoted in ("> ```python\n> import requests\n> ```\n", ">> ~~~\n>> x = 1\n>> ~~~\n", + "> ```python\r\n> import requests\r\n> ```\r\n", "> indented = True\n"): + with self.subTest(quoted=quoted): + self.assertEqual(read({"result": "Use this:\n\n" + quoted}), [None], "code with no source") + self.assertEqual(read({"result": "> A quoted remark, no code.\n"}), [], "a quote without code is not code") + self.fake.serve("/v1/examples/search", {"result": "Use this:\n\n> ```python\n> import requests\n> ```\n", + "search_status": "complete", + "solution_id": "12345678-1234-4234-8234-123456789abc"}) + result = self.client().example("q") + self.assertEqual((result.results_dropped_license, result.data), (1, None), "an uncited quoted fence is dropped") + class Transport(FakeCase): def test_lean_defaults_are_sent(self): @@ -1199,8 +1288,29 @@ def test_compact_output_by_default_and_json_on_request(self): self.assertTrue(all(record["status"] == "ok" for record in self.records().records)) -def run_entry(env: dict, *argv: str) -> tuple[int, str, str]: - """The GitHits launcher's entry (``githits_cli_entry.main``), what the launcher a model can run executes.""" +@contextlib.contextmanager +def launch_guard(*argvs, key: str | None = None): + """A query-guard directory holding one ``pass`` decision per launcher argv in ``argvs``, as the hook writes when it + passes a launcher command (``arguments_digest`` of the words it read, MAC under the per-call key), and the + environment the launcher's entry reads it with: the key and the launcher's path in that directory.""" + with TemporaryDirectory() as folder: + directory = Path(folder) + (directory / query_guard.DECISIONS).mkdir() + key = key or secrets.token_hex(32) + for n, argv in enumerate(argvs, 1): + query_guard.write_decision(directory, { + "tool_use_id": f"toolu_guard{n}", "tool": "Bash", "query_digest": "0" * 64, "rule": query_guard.PASS, + "arguments_digest": query_guard.arguments_digest(argv)}, key) + yield {query_guard.KEY_ENV: key, query_guard.LAUNCHER_ENV: (directory / "githits").as_posix()} + + +def run_entry(env: dict, *argv: str, guarded: bool = True) -> tuple[int, str, str]: + """The GitHits launcher's entry (``githits_cli_entry.main``), what the launcher a model can run executes. With + ``guarded`` (the default) the query guard's hook has passed exactly this argv (``launch_guard``), as it has for + every launcher call Claude Code runs with the hook; ``guarded=False`` runs it with ``env`` alone.""" + if guarded: + with launch_guard(list(argv)) as extra: + return run_entry({**env, **extra}, *argv, guarded=False) out, err = io.StringIO(), io.StringIO() with mock.patch.dict(os.environ, env): for name in ENV_NAMES: @@ -1263,6 +1373,126 @@ def test_example_always_sends_license_mode_strict_and_takes_no_license_option(se self.assertEqual(json.loads(self.fake.requests[-1]["body"])["license_mode"], "yolo") +class LauncherGuard(FakeCase): + """guard-lane, item 1 (the live canary, 2026-10-30): under ``claude --bare`` Claude Code 2.1.312 ran the launcher + without the query guard's hook, and the client sent the lookup (``sent: true``) before the host stopped the run. + Now the launcher's entry sends nothing unless it claims the hook's verified decision for its exact argv: without + one the call is recorded ``unguarded`` with ``sent: false``, exits 2, and the loopback fake sees no request.""" + + WORDS = ["search", "retry backoff", "--target", "pypi:urllib3@2.2.1"] + + def refused(self, env, words=None): + code, out, err = run_entry(env, *(words or self.WORDS), guarded=False) + self.assertEqual((code, out), (2, "")) + self.assertIn("refused: unguarded:", err) + self.assertIn("no request was sent", err) + return err + + def test_a_launcher_run_with_no_decision_sends_nothing_and_records_sent_false(self): + with launch_guard() as guard: # a guard directory and key, and no decision at all: no hook ran + err = self.refused({**self.env, **guard}) + self.assertIn(query_guard.NO_DECISION, err) + self.refused(self.env) # no key and no launcher in the environment at all + self.assertEqual(self.fake.requests, [], "the loopback fake saw no request") + records = self.records().records + self.assertEqual(len(records), 2, "each refusal is recorded") + for record in records: + self.assertEqual((record["op"], record["status"], record["sent"], record["http_status"]), + ("search", githits.UNGUARDED, False, None)) + self.assertEqual(record["query_digest"], githits.query_digest( + githits.search_request("retry backoff", ["pypi:urllib3@2.2.1"])), "the request it would have sent") + self.assertNotIn("paced_ms", record) + + def test_an_unguarded_example_is_not_paced_before_it_is_refused(self): + with mock.patch.object(githits_cli_entry, "_pace", side_effect=AssertionError("paced")): + self.refused(self.env, ["example", "retry"]) + self.assertEqual(self.fake.requests, []) + + def test_a_forged_or_mismatched_decision_is_refused(self): + def decision(directory, key, words, **changes): + entry = {"tool_use_id": "toolu_forged", "tool": "Bash", "query_digest": "0" * 64, "rule": query_guard.PASS, + "arguments_digest": query_guard.arguments_digest(words)} + entry.update(changes) + query_guard.write_decision(directory, entry, key) + + cases = { + "no MAC": lambda d, key: decision(d, "", self.WORDS), + "another key": lambda d, key: decision(d, secrets.token_hex(32), self.WORDS), + "other arguments": lambda d, key: decision(d, key, ["search", "retry", "--target", "pypi:urllib3@2.2.1"]), + "a blocking rule": lambda d, key: decision(d, key, self.WORDS, rule="email"), + "not a shell tool": lambda d, key: decision(d, key, self.WORDS, tool="mcp__githits__search"), + "no arguments digest": lambda d, key: decision(d, key, self.WORDS, arguments_digest=None), + "no tool use id": lambda d, key: decision(d, key, self.WORDS, tool_use_id=None), + } + for name, write in cases.items(): + with self.subTest(name), launch_guard() as guard: + write(Path(guard[query_guard.LAUNCHER_ENV]).parent, guard[query_guard.KEY_ENV]) + self.refused({**self.env, **guard}) + with self.subTest("a MAC'd decision whose digest was rewritten after signing"), launch_guard(self.WORDS) as guard: + folder = Path(guard[query_guard.LAUNCHER_ENV]).parent / query_guard.DECISIONS + [file] = folder.glob("*.json") + entry = json.loads(file.read_text(encoding="utf-8")) + other = ["search", "other", "--target", "pypi:urllib3@2.2.1"] + entry["arguments_digest"] = query_guard.arguments_digest(other) + file.write_text(json.dumps(entry), encoding="utf-8") + self.refused({**self.env, **guard}, other) + self.assertEqual(self.fake.requests, [], "no forged or mismatched decision let a request out") + self.assertTrue(all(r["status"] == githits.UNGUARDED and r["sent"] is False for r in self.records().records)) + + def test_a_valid_decision_is_sent_once(self): + with launch_guard(self.WORDS) as guard: + env = {**self.env, **guard} + code, out, err = run_entry(env, *self.WORDS, guarded=False) + self.assertEqual((code, err), (0, "")) + self.assertTrue(out.startswith("search: ")) + self.assertEqual(len(self.fake.requests), 1, "the guarded lookup was sent") + err = self.refused(env) # the same command again: its one decision is spent + self.assertIn(query_guard.DECISION_USED, err) + self.assertEqual(len(self.fake.requests), 1) + first, second = self.records().records + self.assertEqual((first["status"], first["sent"]), ("ok", True)) + self.assertEqual((second["status"], second["sent"]), (githits.UNGUARDED, False)) + + def test_two_passed_calls_with_the_same_arguments_each_send_once(self): + with launch_guard(self.WORDS, self.WORDS) as guard: + env = {**self.env, **guard} + self.assertEqual(run_entry(env, *self.WORDS, guarded=False)[0], 0) + self.assertEqual(run_entry(env, *self.WORDS, guarded=False)[0], 0) + self.refused(env) + self.assertEqual(len(self.fake.requests), 2) + + def test_the_hooks_decision_for_a_passed_launcher_command_is_what_the_launcher_claims(self): + """End to end without Claude Code: the real hook judges the command, then the entry runs its words.""" + launcher = "/tmp/syberlabs-guard-0000/githits" + command = launcher + ' search "retry backoff" --target pypi:urllib3@2.2.1' + with TemporaryDirectory() as folder: + directory = Path(folder) + key = secrets.token_hex(32) + query_guard.prepare(directory, query_guard.policy(), set(), bash=True) + hook_env = query_guard.hook_environment(directory, key, launcher) + call = {"tool_name": "Bash", "tool_input": {"command": command}, "tool_use_id": "toolu_1"} + code = query_guard.run(directory, io.StringIO(json.dumps(call)), io.StringIO(), hook_env) + self.assertEqual(code, 0) + [decision] = query_guard.read_decisions(directory) + self.assertTrue(query_guard.authentic(decision, key)) + self.assertEqual(decision["arguments_digest"], query_guard.arguments_digest(self.WORDS)) + env = {**self.env, query_guard.KEY_ENV: key, query_guard.LAUNCHER_ENV: (directory / "githits").as_posix()} + self.assertEqual(run_entry(env, *self.WORDS, guarded=False)[0], 0) + query_guard.remove(directory) + self.assertEqual(len(self.fake.requests), 1) + + def test_the_operator_cli_and_doctor_need_no_decision(self): + for env in (self.env, {**self.env, query_guard.KEY_ENV: secrets.token_hex(32), + query_guard.LAUNCHER_ENV: "/nowhere/githits"}): # with or without a guard key around + code, out, _ = run_cli(env, "githits", "doctor") + self.assertEqual(code, 0) + self.assertTrue(out.startswith("githits doctor: ok")) + code, out, _ = run_cli(env, "githits", *self.WORDS) + self.assertEqual(code, 0, "the operator's own lookup is not gated") + self.assertEqual(len(self.fake.requests), 4) + self.assertEqual([(r["op"], r["status"]) for r in self.records().records], [("doctor", "ok"), ("search", "ok")] * 2) + + # Outside cp1252: two box-drawing characters (U+2500) and an emoji (U+1F680). Written as escapes: the file stays ASCII. WIDE_TITLE = "box \u2500\u2500 rocket \U0001F680" ENTRY = ROOT / "syberlabs" / "githits_cli_entry.py" @@ -1295,10 +1525,11 @@ def entry_subprocess(self, isolated: bool, ioencoding): env.update(self.env) if ioencoding is not None: env["PYTHONIOENCODING"] = ioencoding - argv = [sys.executable, *(["-I"] if isolated else []), str(ENTRY), "search", "q", "--target", "pypi:requests"] - with TemporaryDirectory() as cwd: # no checkout in the current directory - return subprocess.run(argv, cwd=cwd, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, - stderr=subprocess.PIPE, timeout=120) + words = ["search", "q", "--target", "pypi:requests"] + argv = [sys.executable, *(["-I"] if isolated else []), str(ENTRY), *words] + with TemporaryDirectory() as cwd, launch_guard(words) as guard: # no checkout in the current directory + return subprocess.run(argv, cwd=cwd, env={**env, **guard}, stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=120) def test_a_result_outside_cp1252_prints_through_the_entry_on_a_piped_stdout_and_is_recorded_ok_once(self): # -I as the launcher runs it (PYTHONIOENCODING is then ignored: cp1252 on Windows, the locale's elsewhere); @@ -1355,9 +1586,10 @@ def test_a_forced_output_failure_is_recorded_error_not_ok(self): with self.subTest(error=type(failure).__name__, json=bool(json_flag)): sent, before = len(self.fake.requests), len(self.records().records) err = io.StringIO() - with contextlib.redirect_stdout(_BrokenStream(failure)), contextlib.redirect_stderr(err), \ - mock.patch.dict(os.environ, self.env): - code = githits_cli_entry.main(["search", "q", "--target", "pypi:requests", *json_flag]) + words = ["search", "q", "--target", "pypi:requests", *json_flag] + with launch_guard(words) as guard, contextlib.redirect_stdout(_BrokenStream(failure)), \ + contextlib.redirect_stderr(err), mock.patch.dict(os.environ, {**self.env, **guard}): + code = githits_cli_entry.main(words) self.assertEqual(code, 1) self.assertIn("error: output_failed", err.getvalue()) self.assertNotIn("Traceback", err.getvalue()) diff --git a/tests/test_githits_accounting.py b/tests/test_githits_accounting.py index ca6294f..1f561ce 100644 --- a/tests/test_githits_accounting.py +++ b/tests/test_githits_accounting.py @@ -55,7 +55,8 @@ from syberlabs.claude_code import CLIENT_TOOL, GITHITS_ENTRY, LOOKUP, THIRD_PARTY_SWITCHES, ClaudeCodeProvider from tests.test_build_thread import GOOD, REPORT from tests.test_claude_code import ClaudeCodeCase -from tests.test_githits import FakeGitHits, example_response, read_code_response, search_hit, search_response, source +from tests.test_githits import (FakeGitHits, example_response, launch_guard, read_code_response, search_hit, + search_response, source) ARM = "b" AGENT = f"syberfactory/{ARM}" @@ -175,17 +176,18 @@ def setUp(self): self.token = "ghtest_" + secrets.token_hex(16) self.bearer = hashlib.sha256(f"Bearer {self.token}".encode()).hexdigest() kept = {k: v for k, v in os.environ.items() if k not in THIRD_PARTY_SWITCHES + GITHITS_SETTINGS} - patcher = mock.patch.dict(os.environ, {**kept, githits.TOKEN_ENV: self.token, githits.URL_ENV: self.server.url}, - clear=True) + patcher = mock.patch.dict(os.environ, {**kept, githits.TOKEN_ENV: self.token, githits.URL_ENV: self.server.url, + "ANTHROPIC_API_KEY": "test-placeholder"}, clear=True) patcher.start() self.addCleanup(patcher.stop) def arm_b(self, kit, thread, claude, **kwargs) -> ClaudeCodeProvider: - """Arm B as ``syberlabs propose --claude-code --bare --githits`` builds it: the thread's id is the session - label (``cli.py``).""" + """Arm B as ``syberlabs propose --claude-code --lane api-hooked --trusted-tree --githits`` builds it: the + thread's id is the session label (``cli.py``).""" kwargs.setdefault("max_calls", 1) kwargs.setdefault("thread_label", thread.id) - return ClaudeCodeProvider(kit.repo, bare=True, executable=claude, githits=True, **kwargs) + return ClaudeCodeProvider(kit.repo, lane="api-hooked", trusted_tree=True, executable=claude, githits=True, + **kwargs) def work_item(self, kit, name: str, lookups: list, **stand_in): thread = kit.start(f"Add a CSV export ({name})", paths=["src/", "tests/"]) @@ -341,9 +343,11 @@ def test_launcher_processes_running_at_once_each_leave_one_authentic_record(self folder, key = self.root.parent / "parallel-records", secrets.token_hex(32) env = {**os.environ, githits.RECORD_DIR_ENV: str(folder), githits.RECORD_KEY_ENV: key, githits.SESSION_ENV: "parallel-b", githits.AGENT_ENV: AGENT} - processes = [subprocess.Popen([sys.executable, "-I", str(GITHITS_ENTRY), "search", f"retry backoff zq9{n}", - "--target", "pypi:requests@2.32.3"], env=env, stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) for n in range(8)] + words = [["search", f"retry backoff zq9{n}", "--target", "pypi:requests@2.32.3"] for n in range(8)] + env.update(self.enterContext(launch_guard(*words))) # the query guard's hook passed all eight + processes = [subprocess.Popen([sys.executable, "-I", str(GITHITS_ENTRY), *argv], env=env, + stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + for argv in words] self.assertEqual([process.wait(timeout=120) for process in processes], [0] * 8) read = githits.read_records(folder, key) self.assertEqual((len(self.server.of("parallel-b")), len(read.records), read.ignored), (8, 8, 0)) @@ -541,7 +545,8 @@ def test_claude_code_a_provider_built_in_python_names_its_thread_in_the_session( threads = [] for name in ("label-1", "label-2"): thread, claude = self.work_item(kit, name, [launcher(SEARCH)]) - thread.propose(ClaudeCodeProvider(kit.repo, bare=True, executable=claude, githits=True, max_calls=1)) + thread.propose(ClaudeCodeProvider(kit.repo, lane="api-hooked", trusted_tree=True, executable=claude, + githits=True, max_calls=1)) threads.append(thread) self.assertEqual(len(self.server.log), 2) self.assertEqual([entry["session"] for entry in self.server.log], [f"{t.id}-{ARM}" for t in threads]) diff --git a/tests/test_githits_adversarial.py b/tests/test_githits_adversarial.py index 2f4f5cd..c6d20b0 100644 --- a/tests/test_githits_adversarial.py +++ b/tests/test_githits_adversarial.py @@ -34,6 +34,7 @@ from syberlabs.claude_code import GITHITS_ENTRY, ClaudeCodeProvider, record_lookup from tests.test_claude_code import SEARCH, ClaudeCodeCase, launcher_lookup from tests.test_githits import (ENV_NAMES, FakeCase, FakeGitHits, example_response, grep_hit, grep_response, + launch_guard, list_response, read_doc_response, run_cli, search_hit, search_response) LAUNCHER = "/tmp/syberlabs-guard-x1/githits" # the form claude_code.write_launcher returns @@ -313,8 +314,10 @@ def test_a_terminated_launcher_records_the_request_it_sent(self): env = {k: v for k, v in os.environ.items() if k not in ENV_NAMES} env.update({githits.TOKEN_ENV: "ghtest_" + secrets.token_hex(16), githits.URL_ENV: fake.url, githits.RECORD_DIR_ENV: folder.name, githits.RECORD_KEY_ENV: key}) - proc = subprocess.Popen([sys.executable, "-I", str(GITHITS_ENTRY), "search", "retry", "--target", - "pypi:requests"], env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + words = ["search", "retry", "--target", "pypi:requests"] + env.update(self.enterContext(launch_guard(words))) # the query guard's hook passed this launcher call + proc = subprocess.Popen([sys.executable, "-I", str(GITHITS_ENTRY), *words], env=env, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) self.addCleanup(lambda: proc.poll() is None and proc.kill()) deadline = time.monotonic() + 60 while not fake.requests and time.monotonic() < deadline and proc.poll() is None: @@ -358,7 +361,7 @@ def setUp(self): self.addCleanup(self.fake.close) self.fake.server.handle_error = lambda request, address: None # a client the host killed is not an error patcher = mock.patch.dict(os.environ, {githits.TOKEN_ENV: "ghtest_" + secrets.token_hex(16), - githits.URL_ENV: self.fake.url}) + githits.URL_ENV: self.fake.url, "ANTHROPIC_API_KEY": "test-placeholder"}) patcher.start() self.addCleanup(patcher.stop) for name in ENV_NAMES: @@ -377,7 +380,7 @@ def test_an_excluded_target_url_and_the_yolo_license_mode_never_leave(self): launcher_lookup('search "governed work runtime" --target site:www.syberlabs.io'), launcher_lookup('example "retry an HTTP call with backoff" --license-mode yolo'), launcher_lookup('search "retry" --target pypi:requests GITHITS_API_URL=https://evil.example/v1')]) - provider = ClaudeCodeProvider(kit.repo, bare=True, executable=claude, githits=True, max_calls=1) + provider = ClaudeCodeProvider(kit.repo, lane="api-hooked", trusted_tree=True, executable=claude, githits=True, max_calls=1) [candidate] = thread.propose(provider) self.assertEqual(self.fake.requests, [], "nothing reached GitHits") values = [b["value"] for b in self.lookup_facts(thread)] @@ -516,8 +519,8 @@ def run_call(self, name: str, lookups: int): kit = self.kit() thread = kit.start(f"Add a CSV export ({name})", paths=["src/", "tests/"]) claude = self.stand_in(name, lookups=[launcher_lookup(SEARCH)] * lookups) - provider = ClaudeCodeProvider(kit.repo, bare=True, executable=claude, githits=True, max_calls=1, timeout=10, - thread_label=thread.id) + provider = ClaudeCodeProvider(kit.repo, lane="api-hooked", trusted_tree=True, executable=claude, githits=True, + max_calls=1, timeout=10, thread_label=thread.id) self.assertEqual(thread.propose(provider), []) finished = [e["body"] for e in thread.history() if e["kind"] == "search_finished"][-1] self.assertEqual((finished["stopped"], finished["error"]), ("error", "provider_timeout")) @@ -550,7 +553,7 @@ def test_a_provider_built_in_python_names_each_thread_in_the_session(self): for name in ("one", "two"): thread = kit.start(f"Add a CSV export ({name})", paths=["src/", "tests/"]) claude = self.stand_in(f"session-{name}", lookups=[launcher_lookup(SEARCH)]) - thread.propose(ClaudeCodeProvider(kit.repo, bare=True, executable=claude, githits=True, max_calls=1)) + thread.propose(ClaudeCodeProvider(kit.repo, lane="api-hooked", trusted_tree=True, executable=claude, githits=True, max_calls=1)) threads.append(thread) self.assertEqual([r["headers"]["x-githits-session-id"] for r in self.fake.requests], [f"{t.id}-b" for t in threads]) diff --git a/tests/test_githits_pace.py b/tests/test_githits_pace.py index 489721b..a868e83 100644 --- a/tests/test_githits_pace.py +++ b/tests/test_githits_pace.py @@ -21,7 +21,7 @@ from syberlabs import experiment, githits, githits_cli_entry, githits_pace, metrics from syberlabs.claude_code import record_lookup -from tests.test_githits import ENTRY, FakeCase, run_entry +from tests.test_githits import ENTRY, FakeCase, launch_guard, run_entry from tests.test_metrics import HOUR, T0, Journal, check ROOT = Path(__file__).resolve().parents[1] @@ -144,12 +144,14 @@ def example(request): env.update(self.env) env.update({githits_pace.PACE_DIR_ENV: str(Path(self.pace.name) / "pace"), githits_pace.PACE_SECONDS_ENV: str(self.INTERVAL)}) - argv = [sys.executable, "-I", str(ENTRY), "example", "retry with backoff"] - processes = [subprocess.Popen(argv, cwd=self.pace.name, env=env, stdin=subprocess.DEVNULL, - stdout=subprocess.PIPE, stderr=subprocess.PIPE) for _ in range(2)] - for process in processes: - _, err = process.communicate(timeout=120) - self.assertEqual(process.returncode, 0, err.decode("utf-8", "replace")) + words = ["example", "retry with backoff"] + argv = [sys.executable, "-I", str(ENTRY), *words] + with launch_guard(words, words) as guard: # the hook passed both launcher calls + processes = [subprocess.Popen(argv, cwd=self.pace.name, env={**env, **guard}, stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) for _ in range(2)] + for process in processes: + _, err = process.communicate(timeout=120) + self.assertEqual(process.returncode, 0, err.decode("utf-8", "replace")) self.assertEqual(len(arrivals), 2) # Each slot is stamped just before its request leaves, so arrivals sit a whole interval apart less the first # request's own few milliseconds of transit. diff --git a/tests/test_pilot_dry_run.py b/tests/test_pilot_dry_run.py index fea7037..fd409d9 100644 --- a/tests/test_pilot_dry_run.py +++ b/tests/test_pilot_dry_run.py @@ -362,7 +362,7 @@ def test_one_command_runs_the_canary_both_arms_the_review_and_the_report_with_ev on = entry["arm"] == "on" self.assertEqual(entry["metrics"]["githits_lookups"]["value"], 1 if on else 0) self.assertEqual(entry["lookups_blocked"]["value"], 1 if on else 0) - self.assertEqual(entry["harness"]["lane"], ["api"]) + self.assertEqual(entry["harness"]["lane"], ["api-hooked"]) self.assertEqual(entry["harness"].get("githits_route"), ["api"] if on else None) self.assertEqual(len(entry["harness"].get("license_policy", [])), 1 if on else 0, "the treatment's license policy, as the host recorded it (a digest)") @@ -377,7 +377,9 @@ def test_one_command_runs_the_canary_both_arms_the_review_and_the_report_with_ev self.assertEqual(len(proposals), 7, "the canary and six runs") for argv in proposals: start = argv.index("propose") - self.assertEqual(argv[start:start + 3], ["propose", "--claude-code", "--bare"]) + self.assertEqual(argv[start:start + 5], ["propose", "--claude-code", "--lane", "api-hooked", + "--trusted-tree"]) + self.assertNotIn("--bare", argv, "never --bare: no hook runs there") tools = [argv[argv.index("--allowed-tools") + 1] for argv in proposals] self.assertEqual(tools, ["Read,Edit,Write,Glob,Grep"] + ["Read,Edit,Write,Glob,Grep,WebSearch,WebFetch"] * 6, "the pilot's tools in both arms; the canary without the web, so it can only use the launcher") @@ -726,7 +728,8 @@ def test_the_script_as_the_operator_runs_it(self): self.assertNotIn(secret, printed) log = (out / "commands.log").read_text(encoding="utf-8") self.assertIn("$ syberlabs githits doctor --json", log) - self.assertEqual(log.count(" propose --claude-code --bare "), 7, "the canary and six runs") + self.assertEqual(log.count(" propose --claude-code --lane api-hooked --trusted-tree "), 7, + "the canary and six runs") self.assertEqual(len(self.fake.requests), 5, "the doctor, the canary's clean lookup, three treatment runs") diff --git a/tests/test_query_guard.py b/tests/test_query_guard.py index 99c0044..f20b8d5 100644 --- a/tests/test_query_guard.py +++ b/tests/test_query_guard.py @@ -422,6 +422,23 @@ def test_the_hook_passes_only_the_launcher_and_signs_each_bash_decision(self): capture_output=True, timeout=120, env=no_launcher) self.assertEqual(done.returncode, guard.BLOCK_EXIT, "without the host's launcher no Bash command passes") self.assertIn(b"rule: not_launcher", done.stderr) + # guard-lane, item 1: a passed launcher call's decision carries the digest of the arguments the hook read, which + # the launcher's entry claims before sending; a blocked one carries none. The MAC covers it. + self.assertEqual(found["toolu_b0"]["arguments_digest"], guard.arguments_digest( + ["search", "csv dialect sniffing", "--target", "pypi:clevercsv"])) + self.assertTrue(all("arguments_digest" not in found[f"toolu_b{n}"] for n in range(1, len(cases)))) + altered = {**found["toolu_b0"], "arguments_digest": guard.arguments_digest(["search", "other"])} + self.assertFalse(guard.authentic(altered, KEY), "a rewritten arguments digest no longer verifies") + self.assertFalse(guard.authentic({**found["toolu_b0"], "arguments_digest": None}, KEY), "nor a removed one") + launcher_env = {guard.KEY_ENV: KEY, guard.LAUNCHER_ENV: (settings.parent / "githits").as_posix()} + self.assertEqual(guard.claim_launch(["search", "csv dialect sniffing", "--target", "pypi:clevercsv"], + launcher_env), (None, "toolu_b0")) + self.assertEqual(guard.claim_launch(["search", "csv dialect sniffing", "--target", "pypi:clevercsv"], + launcher_env)[0], guard.DECISION_USED, "one decision, one launch") + self.assertEqual(guard.claim_launch(["search", "q;"], launcher_env)[0], guard.NO_DECISION) + self.assertEqual(guard.claim_launch(["search", "x"], {guard.KEY_ENV: KEY})[0], guard.NO_GUARD) + self.assertEqual(guard.claim_launch(["search", "csv dialect sniffing", "--target", "pypi:clevercsv"], + {**launcher_env, guard.KEY_ENV: "another-key"})[0], guard.NO_DECISION) def test_the_hook_decides_as_the_host_does(self): texts = ["pydantic v2 model_validator usage", SECRET, "key AKIA" + "Q" * 16, "slack xoxb-" + "1" * 12,