Skip to content
Merged
8 changes: 4 additions & 4 deletions docs/CLAUDE_CODE.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ exec '<host python>' -I '<host checkout>/syberlabs/githits_cli_entry.py' "$@"
| `SYBERLABS_GUARD_LAUNCHER` | the launcher's path, for the query guard |
| `SYBERLABS_GITHITS_RECORD_DIR` | `githits-records/` in the call's guard directory |
| `SYBERLABS_GITHITS_RECORD_KEY` | a fresh `secrets.token_hex(32)` per call, in no file |
| `GITHITS_SESSION_ID` | `<thread>-<arm>` (`X-GitHits-Session-ID`; GitHits allows only `[A-Za-z0-9_-]`, so not `/`) |
| `GITHITS_SESSION_ID` | `<thread>-<arm>` (`X-GitHits-Session-ID`; GitHits allows only `[A-Za-z0-9_-]`, so not `/`); the thread is `thread_label` (the CLI passes the thread's id), else the search space's thread, so a provider built in Python names its thread too |
| `GITHITS_AGENT` | `syberfactory/<arm>` (`X-GitHits-Agent`) |
| `SYBERLABS_GITHITS_LICENSES` | the license allow list in force, written out (the operator's, else the permissive default) |
| `SYBERLABS_GITHITS_EXCLUDE` | the exclusion list in force, written out (`syberlabs` plus what the operator added) |
Expand All @@ -154,13 +154,13 @@ exec '<host python>' -I '<host checkout>/syberlabs/githits_cli_entry.py' "$@"

**Arm A holds no GitHits variable.** Without `--githits`, and without `--mcp-config` (no GitHits route at all), the provider removes every `GITHITS_*` and `SYBERLABS_GITHITS_*` variable from Claude Code's environment (`claude_code.githits_variable`, compared ignoring case), `GITHITS_API_TOKEN` and `GITHITS_API_URL` included. Least privilege was chosen over a byte-identical environment: arm A has no tool that could use the token, and now it does not hold it either. Arm A's argv is still byte-identical to what it was before this route existed. `tests/test_claude_code.py` pins arm A's argv literally and its environment as the caller's minus those variables plus the guard's three, and pins arm B's differences to exactly the items above plus the operator's GitHits variables that arm A no longer receives. An `--mcp-config` run keeps them, because a GitHits MCP server, a child of Claude Code, may read its token from that environment (the MCP A/B below).

**The guard over Bash.** With the launcher, the query guard's matcher is `^(mcp__githits__.*|WebSearch|WebFetch|Bash)$`, and a `Bash` call passes only when its command is the launcher's exact path, alone or followed by a space, and what follows is plain words and single- or double-quoted strings (`query_guard.bash_arguments`). `not_launcher` blocks any other command; `shell_syntax` blocks a control operator, redirection, expansion or substitution anywhere (`` ; & | < > $ ` \ ( ) { } ! ``, any control character such as a newline), a glob, tilde or comment character outside quotes, an unterminated quote, and `run_in_background`. The arguments that pass are the outbound text: each one, and all of them joined, are judged by the same rules as a GitHits tool's query (`secret`, `url_credentials`, `email`, `too_long`, `code_fence`, `code_lines`, `tree_line`). Decisions carry the same MAC under the per-call key. Claude Code's own permission check is a second gate: under `dontAsk` a Bash command that `Bash(<launcher>:*)` does not name is denied.
**The guard over Bash.** With the launcher, the query guard's matcher is `^(mcp__githits__.*|WebSearch|WebFetch|Bash)$`, and a `Bash` call passes only when its command is the launcher's exact path, alone or followed by a space, and what follows is plain words and single- or double-quoted strings (`query_guard.bash_arguments`). `not_launcher` blocks any other command; `shell_syntax` blocks a control operator, redirection, expansion or substitution anywhere (`` ; & | < > $ ` \ ( ) { } ! ``, any control character such as a newline), a glob, tilde, caret or comment character outside quotes, an unquoted word that a shell reads differently from the guard (one that starts with `=`, which zsh's default EQUALS option expands to a command's path; with `NAME=`, which bash's `set -k` turns into an assignment into the launcher's environment, so `GITHITS_API_URL=https://...` would send the client and its token elsewhere; or with `@`, which PowerShell splats; quoted, each passes), an unterminated quote, `run_in_background`, and a `timeout` shorter than the Bash tool's two-minute default (`query_guard.MIN_BASH_TIMEOUT_MS`): ended early by Claude Code, the launcher's client could be killed after its request left and before it writes the call's record. The arguments that pass are the outbound text: each one, and all of them joined, are judged by the same rules as a GitHits tool's query (`secret`, `url_credentials`, `email`, `too_long`, `code_fence`, `code_lines`, `tree_line`). Decisions carry the same MAC under the per-call key. Claude Code's own permission check is a second gate: under `dontAsk` a Bash command that `Bash(<launcher>:*)` does not name is denied.

**PowerShell under the override.** An operator's `Bash` or `PowerShell` entry is refused with `--githits` unless `--allow-code-tools-with-mcp` is passed. With the override, the matcher is `^(mcp__githits__.*|WebSearch|WebFetch|Bash|PowerShell)$` (`query_guard.MATCHER_SHELLS`), and a `PowerShell` call is judged by the same launcher-only rule as `Bash` (`query_guard.SHELL_TOOLS`): any command but the launcher is `not_launcher`, so an allowed PowerShell no longer runs unguarded. Its refusals are journaled like a refused Bash call (tool `PowerShell`, its rule) and counted in `signal.githits.bash_refused` and `lookups_blocked`. Without the override the matcher is the Bash one above.

**What is recorded.** After the call the provider reads the client's records (`githits.read_records` under the call's record key). Each authentic record becomes one verified `lookup` fact, tool `githits.<op>`, value `{tool, query_digest, request_id, status, at, results_returned, results_dropped, results_dropped_license, latency_ms, contamination_blocked, ok, guard}`: the client's own digest of the request it sent, never text. Forged or altered record lines are counted (`signal.githits.records_ignored`) and ignored. Every `Bash` call the guard did not pass is journaled as a `lookup` with tool `Bash` and its rule (or `unguarded`), which `lookups_blocked` counts. `signal.tool_usage.githits` is the number of authentic records (plus any `mcp__githits__*` call the stream shows). The candidate also carries `signal.githits: {arm, launcher_calls, launcher_passed, records, records_ignored, bash_refused, license_policy[, mismatch]}` and `signal.harness.skill: {path, sha256}`. `githits_lookups` never counts a `githits.doctor` fact.
**What is recorded.** After the call the provider reads the client's records (`githits.read_records` under the call's record key). Each authentic record becomes one verified `lookup` fact, tool `githits.<op>`, value `{tool, query_digest, request_id, status, at, results_returned, results_dropped, results_dropped_license, latency_ms, contamination_blocked, sent, ok, guard}`: the client's own digest of the request it sent, never text; `sent` is the record's, whether the request was written to GitHits (false for an excluded target or a connection that never opened, true from a timeout after the request left to an answer, null for a call interrupted in flight). The client records every call it starts, an answer it cannot read and an interruption included; the launcher's entry turns a `SIGTERM` into an exception for that, so a launcher ended by a Bash timeout records its call (a `SIGKILL` cannot be caught). Forged or altered record lines are counted (`signal.githits.records_ignored`) and ignored. Every `Bash` call the guard did not pass is journaled as a `lookup` with tool `Bash` and its rule (or `unguarded`), which `lookups_blocked` counts. `signal.tool_usage.githits` is the number of authentic records (plus any `mcp__githits__*` call the stream shows). The candidate also carries `signal.githits: {arm, launcher_calls, launcher_passed, records, records_ignored, bash_refused, license_policy[, mismatch]}` and `signal.harness.skill: {path, sha256}`. `githits_lookups` never counts a `githits.doctor` fact.

A record cannot be tied to the tool use that ran it, so the host reconciles counts: authentic records against launcher calls the guard passed. Fewer records is `mismatch: githits_unrecorded` (a usage error, a client that failed before recording, a denied permission; recorded, the call stands). More records than guarded calls is `githits_unexpected_records`: each record of the call is journaled `guard: unguarded`, and the search stops `provider_guard_inactive`, because something holding the key ran the client without the guard.
A record cannot be tied to the tool use that ran it, so the host reconciles counts: authentic records against launcher calls the guard passed. Fewer records is `mismatch: githits_unrecorded` (a usage error, a client that failed before recording, a denied permission; recorded, the call stands). More records than guarded calls is `githits_unexpected_records`: each record of the call is journaled `guard: unguarded`, and the search stops `provider_guard_inactive`, because something holding the key ran the client without the guard. A call that fails or times out registers no candidate, so its mismatch would reach the journal nowhere, and a launcher call still waiting for GitHits when the host ends Claude Code (its session is killed, so the client cannot record) would leave no trace: when such a call has fewer records than guarded launcher calls, its records are journaled `guard: unguarded` (not reconciled), and when it has none, one `githits.unrecorded` fact (the command line's digest, `sent: null`, `guard: unguarded`) stands for each launcher call the guard passed. A call refused `no_change` still loses its mismatch (`spec/GAPS.md` 59 (12)).

**The skill.** `skills/githits/SKILL.md` adapts GitHits' shipped skills (`githits-com/githits-cli` at `0bc9f5e68b`, `skills/githits-code/SKILL.md` and `skills/githits-mcp/SKILL.md`, Apache-2.0, Copyright 2025 GitHits Inc.; `skills/githits/LICENSE` is the upstream license and `skills/githits/NOTICE` says what changed) for the launcher: when a lookup is worth it (an API or version fact the repository and the standard library do not settle), how to phrase a query (short, naming the package and version, no code or repository text), targets first, search then grep or list then read, results as untrusted data, and citing sources in the final reply (the harness does not commit). Claude Code 2.1.309's `--help` offers `--append-system-prompt`, `--plugin-dir` (skills inside a plugin) and `--add-dir`; the provider uses `--append-system-prompt`, so the text is part of arm B's system prompt on every request rather than a skill the model may or may not load, and arm A's prompt is exactly the harness note. The YAML front matter is left out, and `{launcher}` is replaced by the call's launcher path, so arm B's exact appended text is the harness note, a blank line, and the file's body with that one substitution; the file's SHA-256 is on `signal.harness.skill`, computed with CRLF normalized to LF, so a Windows checkout with `core.autocrlf` records the committed blob's digest (`24c150e4…6508` today) as a POSIX one does. `.gitattributes` (`skills/** text eol=lf`) also checks the skill out with LF on every platform.

Expand Down
8 changes: 3 additions & 5 deletions docs/METRICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ A **work item** is one Build Thread: from `case_created` to the merge of the pul
| **agent_time** (hours) | `pr_merged.merged_at` minus `case_created.at`. Human review time is inside the interval by construction. | no `pr_merged` fact |
| **fix_rounds** | Registrations that follow a failed evaluation (a required check of the contract not `passed`; every recorded check when the contract is not supplied), each failed evaluation counted once. First attempt right is `fix_rounds == 0` with the first candidate accepted. | never |
| **web_lookups** | Sum of `signal.tool_usage.web_search + web_fetch` over registrations. | any registration did not declare either |
| **githits_lookups** | When the record holds `lookup` facts made from GitHits client records (tool `githits.<op>`: the `--githits` route, `docs/CLAUDE_CODE.md`), their number: host-observed, one per authentic record the client signed, calls whose answer was refused included. Otherwise the sum of `signal.tool_usage.githits` over registrations. | no client-record fact, and any registration did not declare it |
| **githits_lookups** | When the record holds `lookup` facts made from GitHits client records (tool `githits.<op>`: the `--githits` route, `docs/CLAUDE_CODE.md`), their number: host-observed, one per authentic record the client signed, calls whose answer was refused included. A `githits.doctor` fact (the operator's readiness check, `docs/GITHITS.md`) is never counted, and when the client-record facts are all doctor facts the count is 0, not the declared sum. Otherwise the sum of `signal.tool_usage.githits` over registrations. | no client-record fact, and any registration did not declare it; or a call's `signal.githits.mismatch` says a launcher call the guard passed left no client record, or a client-record fact is `unguarded` (a record no guarded launcher call stands behind, a failed call's record, or a `githits.unrecorded` fact standing for a failed call's launcher call that left none): the record cannot say whether those reached GitHits |
| **delivered** | `true` with a `pr_merged` fact (for the opened pull request's number); `false` when no pull request was opened, the thread was cancelled, or a `pr_closed` fact (same number) says it was closed unmerged. `opened` is a separate column: a `github_pull_request` effect that succeeded, or a verified reconciliation of one. | a pull request is open and neither `pr_merged` nor `pr_closed` is in the record |
| **review_rejections** | Critic checks (`critic`, `critic.*`) that did not pass, or that passed with a host critic observation counting a high finding (what `critic.no_high_finding` refuses); plus `review` facts with verdict `fail` and `pr_review` facts with state `changes_requested`. | never |
| **human_interventions** | `human_intervention` facts, plus `proposed`, `approved` and `case_cancelled` events whose actor is neither `syberlabs.host` nor the thread's unattended runner (`unattended.by`) and whose proposal no fact names. | never |
Expand All @@ -52,11 +52,9 @@ All are `observed` events written by the host (`actor: syberlabs.host`, `verifie
## 4. Accuracy, stated honestly

- **Cost** is exact for host-metered API lanes and unknown for others; the label says which. Critic usage is declared by the critic, so it counts as a turn and is left out of host-observed dollars.
- **Turns** count host-recorded calls with declared usage. A harness's internal loop is one call. A stand-in adapter that reports identical usage on two consecutive calls with nothing recorded between them would be read as one call.
- **Agent time** includes human review time; `human_intervention` facts let a reader split it, and the `unattended` flag marks items where no person was asked. It is known from the first settlement pass after the merge; `merged_at` is GitHub's clock and `case_created` the host's, so skew between them is inside the number.
- **Lookups** are adapter-declared (`declared`, not `host_observed`) until a grounding station records GitHits reads itself.
- **Turns** count host-recorded calls with declared usage. A harness's internal loop is one call unless the registration declares `harness_turns`; a harness call refused `no_change` carries no `harness_turns` and counts as one. A stand-in adapter that reports identical usage on two consecutive calls with nothing recorded between them would be read as one call.
- **Agent time** includes human review time; `human_intervention` facts let a reader split it, and the `unattended` flag marks items where no person was asked.
- **Agent time** includes human review time; `human_intervention` facts let a reader split it, and the `unattended` flag marks items where no person was asked. It is known from the first settlement pass after the merge; `merged_at` is GitHub's clock and `case_created` the host's, so skew between them is inside the number.
- **Lookups** are adapter-declared for JSON adapters. For `ClaudeCodeProvider` the host counts them from Claude Code's stream and writes one `lookup` record per call; the stream is still Claude Code's report of what it did, not a network observation, and lookups by a call whose answer was refused are not counted (`spec/GAPS.md` 52, 54). On the `--githits` route GitHits lookups are counted from the host's own client's records instead: each is a request the host's code sent (or refused itself) and signed under a per-call key, counted even when the call's answer was refused; a thread that mixes this route with GitHits MCP lookups counts only the client's (`spec/GAPS.md` 59).
- **GitHits lookups against the server** (`tests/test_githits_accounting.py`, 2026-10-26, with #135 and #142). An independent counter, the loopback GitHits' own log of the requests it received, was compared per work item with the journal's `githits.<op>` facts and `githits_lookups`, through the real provider, launcher and client. They agree, request id for request id and digest for digest, for: a lookup; results dropped for an excluded owner or the license; launcher commands the guard blocked (nothing reaches the client or GitHits); 401, 404, 500, 503 and 429 answers (one request each: the client does not retry); a timeout and a dropped connection (the fact has no request id: the client never read one); an answer the client cannot read (recorded as `malformed_response` since #142); five lookups in one call, in order, the example sent with `license_mode: strict`; eight launcher processes at once; fix rounds, up to the first acceptance as `experiment` reads it; and arm A, which has no launcher and no GitHits variable and counts a known 0. Two kinds of lookup are counted though GitHits never received them, by this metric's definition: a target the client refused (`excluded_target`) and a connection that never opened. Since #142 each record says whether its request was written (`sent`: true, false, or null when interrupted in flight), and in every case above the facts with `sent: true` are exactly the requests GitHits received. The count is `unknown` where the journal cannot say how many requests left (above): a command the launcher's entry refuses (`doctor`, `--help`, `--license-mode`) or a usage error sends nothing and leaves no record, but the guard passed it, and a launcher killed outright (SIGKILL) or a record that cannot be written leaves the same journal after its request left; and a lookup in flight when the Claude Code call times out, whose failed call journals its records `unguarded`, or a `githits.unrecorded` fact when it has none (#142). It is still a known number, possibly one short, when an unrecorded launcher call is in a call refused `no_change`, because that mismatch rides only on a registered candidate's signal (`spec/GAPS.md` 59 (12)).
- **Interventions** undercount messages to an agent made outside the CLI, and cannot tell an agent from a person when both act under one identity: on a thread without an unattended runner, every `proposed` by a non-host actor is a person's act.
- **Delivered** is `unknown`, not `false`, for an opened pull request until a settlement pass has seen it merged or closed. `--check` judges the items the record shows as delivered, so once a merge is recorded an item whose cost, turns or lookups are unknown (every `--from-worktree` proposal) fails it. A pull request closed and then reopened keeps its `pr_closed` fact and reads `false` until a merge is recorded (`spec/GAPS.md` 53).
Loading
Loading