Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/CLAUDE_CODE.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ syberlabs propose --claude-code --bare --githits # arm B: the same, plus t
exec '<host python>' -I '<host checkout>/syberlabs/githits_cli_entry.py' "$@"
```

`githits_cli_entry.py` puts the host checkout first on `sys.path`, refuses to run if `syberlabs` came from anywhere else, and runs the host's GitHits client with the arguments. It accepts only the five lookups (`search`, `example`, `read`, `list`, `grep`); `doctor` and anything else exit 2 with a usage message and write no record, and `example` takes no `--license-mode`, so it always sends `license_mode: strict` (`docs/GITHITS.md`, the launcher's entry). `doctor` stays the operator's: `syberlabs githits doctor`. `-I` keeps the worktree's directory and `PYTHONPATH` off the import path, so the candidate's own `syberlabs/` never runs. Claude Code's Bash tool runs bash (Git Bash on Windows), which runs the script. The launcher is named by its absolute path with forward slashes, the exact text the model types; a path holding a space or a quote is refused (`provider_unavailable`).
`githits_cli_entry.py` puts the host checkout first on `sys.path`, refuses to run if `syberlabs` came from anywhere else, and runs the host's GitHits client with the arguments. It accepts only the five lookups (`search`, `example`, `read`, `list`, `grep`); `doctor` and anything else exit 2 with a usage message and write no record, and `example` takes no `--license-mode`, so it always sends `license_mode: strict` (`docs/GITHITS.md`, the launcher's entry). `doctor` stays the operator's: `syberlabs githits doctor`. An `example` is held on the host until this host's last one is 40 s old, across processes and arms (`syberlabs/githits_pace.py`). `-I` keeps the worktree's directory and `PYTHONPATH` off the import path, so the candidate's own `syberlabs/` never runs. Claude Code's Bash tool runs bash (Git Bash on Windows), which runs the script. The launcher is named by its absolute path with forward slashes, the exact text the model types; a path holding a space or a quote is refused (`provider_unavailable`).

**What arm B adds.** Argv: `Bash` in `--tools`, `Bash(<launcher>:*)` in `--allowedTools`, and the skill after the harness note in `--append-system-prompt`. Environment, for the launcher's client:

Expand All @@ -158,7 +158,7 @@ exec '<host python>' -I '<host checkout>/syberlabs/githits_cli_entry.py' "$@"

**PowerShell under the override.** An operator's `Bash` or `PowerShell` entry is refused with `--githits` unless `--allow-code-tools-with-mcp` is passed. With the override, the matcher is `^(mcp__githits__.*|WebSearch|WebFetch|Bash|PowerShell)$` (`query_guard.MATCHER_SHELLS`), and a `PowerShell` call is judged by the same launcher-only rule as `Bash` (`query_guard.SHELL_TOOLS`): any command but the launcher is `not_launcher`, so an allowed PowerShell no longer runs unguarded. Its refusals are journaled like a refused Bash call (tool `PowerShell`, its rule) and counted in `signal.githits.bash_refused` and `lookups_blocked`. Without the override the matcher is the Bash one above.

**What is recorded.** After the call the provider reads the client's records (`githits.read_records` under the call's record key). Each authentic record becomes one verified `lookup` fact, tool `githits.<op>`, value `{tool, query_digest, request_id, status, at, results_returned, results_dropped, results_dropped_license, latency_ms, contamination_blocked, sent, ok, guard}`: the client's own digest of the request it sent, never text; `sent` is the record's, whether the request was written to GitHits (false for an excluded target or a connection that never opened, true from a timeout after the request left to an answer, null for a call interrupted in flight). The client records every call it starts, an answer it cannot read and an interruption included; the launcher's entry turns a `SIGTERM` into an exception for that, so a launcher ended by a Bash timeout records its call (a `SIGKILL` cannot be caught). Forged or altered record lines are counted (`signal.githits.records_ignored`) and ignored. Every `Bash` call the guard did not pass is journaled as a `lookup` with tool `Bash` and its rule (or `unguarded`), which `lookups_blocked` counts. `signal.tool_usage.githits` is the number of authentic records (plus any `mcp__githits__*` call the stream shows). The candidate also carries `signal.githits: {arm, launcher_calls, launcher_passed, records, records_ignored, bash_refused, license_policy[, mismatch]}` and `signal.harness.skill: {path, sha256}`. `githits_lookups` never counts a `githits.doctor` fact.
**What is recorded.** After the call the provider reads the client's records (`githits.read_records` under the call's record key). Each authentic record becomes one verified `lookup` fact, tool `githits.<op>`, value `{tool, query_digest, request_id, status, at, results_returned, results_dropped, results_dropped_license, latency_ms, contamination_blocked, sent, ok, guard}`, plus `paced_ms` when the launcher held an `example` to keep GitHits' per-organization rate (`docs/GITHITS.md`, example pacing): the client's own digest of the request it sent, never text; `sent` is the record's, whether the request was written to GitHits (false for an excluded target or a connection that never opened, true from a timeout after the request left to an answer, null for a call interrupted in flight). The client records every call it starts, an answer it cannot read and an interruption included; the launcher's entry turns a `SIGTERM` into an exception for that, so a launcher ended by a Bash timeout records its call (a `SIGKILL` cannot be caught). Forged or altered record lines are counted (`signal.githits.records_ignored`) and ignored. Every `Bash` call the guard did not pass is journaled as a `lookup` with tool `Bash` and its rule (or `unguarded`), which `lookups_blocked` counts. `signal.tool_usage.githits` is the number of authentic records (plus any `mcp__githits__*` call the stream shows). The candidate also carries `signal.githits: {arm, launcher_calls, launcher_passed, records, records_ignored, bash_refused, license_policy[, mismatch]}` and `signal.harness.skill: {path, sha256}`. `githits_lookups` never counts a `githits.doctor` fact.

A record cannot be tied to the tool use that ran it, so the host reconciles counts: authentic records against launcher calls the guard passed. Fewer records is `mismatch: githits_unrecorded` (a usage error, a client that failed before recording, a denied permission; recorded, the call stands). More records than guarded calls is `githits_unexpected_records`: each record of the call is journaled `guard: unguarded`, and the search stops `provider_guard_inactive`, because something holding the key ran the client without the guard. A call that fails or times out registers no candidate, so its mismatch would reach the journal nowhere, and a launcher call still waiting for GitHits when the host ends Claude Code (its session is killed, so the client cannot record) would leave no trace: when such a call has fewer records than guarded launcher calls, its records are journaled `guard: unguarded` (not reconciled), and when it has none, one `githits.unrecorded` fact (the command line's digest, `sent: null`, `guard: unguarded`) stands for each launcher call the guard passed. A `no_change` refusal with missing records journals a synthetic gap without assigning it to a concurrent command; its count remains unknown (`spec/GAPS.md` 59 (12)).

Expand Down
41 changes: 39 additions & 2 deletions docs/GITHITS.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,18 @@ unbalanced parenthesis, a dangling operator, a stray character, more than 64 tok
`results_dropped_license`, apart from the contamination count. Contamination is judged first. In contract
0.1.0 only an example carries licenses: the `(SPDX)` after each source link of its Markdown
(`[Source 1](https://github.com/o/r/blob/sha/path) (MIT)`), a `License:` line, or a structured `references`
list if one is ever returned. An example whose sources include a license that is not admitted, a source link
with no license, or no source link at all, is dropped whole. Exit 0, with a one-line notice.
list if one is ever returned. A source link is a link to a code host, or any link under a `References` or
`Sources` heading (up to the next heading or thematic break), which is how GitHits' documented responses cite
them (`### References` / `- [Source 1](https://example.org/source) (MIT)`); GitHits' own "Open in GitHits Web
App" link is never one. An example whose sources include a license that is not admitted, a source link
with no license, or code with no source link at all, is dropped whole. Exit 0, with a one-line notice. An
example with neither code (a fenced, indented or HTML block) nor a source carries nothing to license and
passes: GitHits' documented `degraded` ("Source search may be incomplete.") and `no_example` ("No suitable
example was found.") answers. All three documented example responses in the pinned OpenAPI document pass, and
the same shapes with a copyleft, `NOASSERTION`, unparsable or missing license are dropped
(`tests/test_githits.py`, `DocumentedExamples`). Before 2026-10-29 all three were withheld
(`results_dropped_license=1`): the `complete` example's source is not on a code host, and the other two have
no source.
- **It depends on license data being present.** No search, grep, list or read result in 0.1.0 has a license
field, so those results pass this check unjudged. The client would read `license_spdx_id`, `spdx_id`,
`license_id` or `license` on a result (or a search hit's locator) if GitHits added one.
Expand Down Expand Up @@ -130,10 +140,35 @@ the two is an error, so a run never believes it is recording when it is not.
- `read_records(directory, key)` returns the records whose MAC verifies, and counts every other line (bad MAC,
extra or missing field, not JSON) as ignored.

`paced_ms`, present only on a call the host's launcher paced (next section), is how long the launcher held the
call before sending it; the MAC covers it, and a record without it has exactly the fields above.

No query text, no result and no token is ever stored. As with the guard's key, the MAC shows that a process
holding the key wrote the line. The key is in the environment of whatever runs the client, so it does not show
which process that was.

## Example pacing (host side)

GitHits limits its example endpoint per organization to one or two requests a minute, and the pilot runs both arms
and every Claude Code call under one organization's token. The launcher's entry (`githits_cli_entry.py`) therefore
holds each `example` before it is sent until this host's last example is 40 s old (`githits_pace.EXAMPLE_INTERVAL`):
a cross-process lock on `example.lock` and the time of the last slot in `example.last`, in the host's pacing
directory, outside any worktree (`SYBERLABS_GITHITS_PACE_DIR`, an absolute path, else `syberlabs-githits-pace` in the
system temporary directory). Callers queue on the lock, so two examples from any two processes, either arm, are at
least 40 s apart. Search, read, list and grep are not paced, and `syberlabs githits example` (the operator's) is not
paced either.

This is pacing, not a retry: the client still sends each request once and never waits or retries on its own (a
missing retry and back-off is itself a pilot work item, so it is kept out of `githits.py`'s request code). The time
the call was held is recorded on its client record as `paced_ms` and on its `lookup` fact; `latency_ms` never
includes it. `syberlabs experiment report` leaves it out of an arm's time: `time_to_pass` subtracts the pacing between
the first search and the evaluation of the accepted candidate, the per-arm `agent_time` subtracts the thread's total,
and each arm reports its `paced_seconds`.

An example to a loopback base URL (a stand-in, such as the tests' fake GitHits) never reaches GitHits' limit and is
not paced unless `SYBERLABS_GITHITS_PACE_SECONDS` sets an interval for it; that variable is ignored for any other
base. A pacing directory that cannot be used fails the call before anything is sent. Limits: `spec/GAPS.md` 63.

## Token and transport

- The token is read only from `GITHITS_API_TOKEN`, and is sent only in `Authorization: Bearer`. It is not in any
Expand Down Expand Up @@ -300,6 +335,8 @@ misreports. Claim ceiling: implemented and tested with stand-ins; no pilot has r
| `SYBERLABS_GITHITS_EXCLUDE_REPLACE` | `1`: the list above replaces `SyberLabs` (tests only) |
| `SYBERLABS_GITHITS_LICENSES` | allowed SPDX ids, comma list, or `any` (default: the permissive list) |
| `SYBERLABS_GITHITS_RECORD_DIR`, `SYBERLABS_GITHITS_RECORD_KEY` | where records go and their MAC key (both or neither) |
| `SYBERLABS_GITHITS_PACE_DIR` | the host's example pacing directory, absolute (default: `syberlabs-githits-pace` in the temporary directory) |
| `SYBERLABS_GITHITS_PACE_SECONDS` | a pacing interval for a loopback base URL only (tests); ignored for GitHits itself, always 40 s |

Claim ceiling: **implemented and tested** against a loopback fake that follows the pinned schemas. It has not
been run against the real API, so the real server's behaviour is not observed: whether `strict` already excludes
Expand Down
7 changes: 5 additions & 2 deletions docs/GITHITS_OPERATOR_HANDOFF.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,13 @@ Required before spending:

- Approval naming provider, exact model/route, run plan, spending authority,
scheduling budget and treatment of search fees and overshoot.
- Python 3.11+; Git; Claude Code **2.1.311** for this frozen smoke recipe;
- Python 3.11+; Git; Claude Code **2.1.312** exactly for this frozen smoke recipe (the version the host
and the pilot VM run; `CLI_VERSION`, recorded in `plan.json` and `manifest.json`);
Git Bash on Windows. The Windows native CLI was downloaded from its official
npm package, checked against npm's SHA-512 and exercised with version/help
only. Those flags are compatible; the live hook canary remains unrun.
only. Those flags are compatible; the live hook canary remains unrun. (That
download check was of 2.1.311; the pin moved to 2.1.312 on 2026-10-29
because the host and the pilot VM run it, and 2.1.312 was not re-checked here.)
- A plain temporary path without spaces or shell punctuation for launchers.
- The exact approved candidate checked out cleanly. Do not reuse another
worker's checkout or branch, and do not commit the plan or run directory.
Expand Down
2 changes: 1 addition & 1 deletion examples/githits-engineering-smoke/plan.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"_purpose": "ENGINEERING SMOKE only: synthetic, visible checks, no formal items or held-out data; not an unbiased cost experiment",
"_formal_pack": "SyberLabs/MasterMind claude/githits-pilot-pack at 09864ac6633832cc5d7c082ec731aff7a5689cad; proposal and formal baseline unchanged",
"_configuration": {"provider": "Anthropic direct API through Claude Code", "model": "claude-sonnet-5-5", "cli": "2.1.311", "effort": "medium", "cache_ttl": "5m", "pricing_checked": "2026-10-26", "pricing_source": "https://platform.claude.com/docs/en/about-claude/pricing"},
"_configuration": {"provider": "Anthropic direct API through Claude Code", "model": "claude-sonnet-5-5", "cli": "2.1.312", "effort": "medium", "cache_ttl": "5m", "pricing_checked": "2026-10-26", "pricing_source": "https://platform.claude.com/docs/en/about-claude/pricing"},
"experiment": "githits-engineering-smoke-v1",
"items": [
{"id": "ENGSMOKE-CSV", "tags": ["synthetic", "routine"], "objective": "Implement render(rows) in src/smoke_csv.py. Return CSV text for a list of name/qty dictionaries, with header name,qty, CRLF record endings and standard CSV quoting. Preserve names containing commas, quotes, Unicode and newlines, serialize integer quantities including negatives, and emit the header for empty input. Use only Python's standard library. Change only this new module; the visible host checks must pass.", "paths": ["src/smoke_csv.py"]},
Expand Down
13 changes: 8 additions & 5 deletions scripts/githits_engineering_smoke.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
ROOT = Path(__file__).resolve().parents[1]
PACK = ROOT / "examples" / "githits-engineering-smoke"
MODEL = "claude-sonnet-5-5"
CLI_VERSION = "2.1.311"
CLI_VERSION = "2.1.312" # exact: the host and the pilot VM run it; recorded in plan.json and manifest.json


def prepare(work):
Expand All @@ -43,7 +43,7 @@ def prepare(work):
"factory": pilot.host_checkout(), "fixture_base": pilot.git(repo, "rev-parse", "HEAD"),
"plan_sha256": hashlib.sha256(path.read_bytes()).hexdigest(),
"check_sha256": hashlib.sha256((PACK / "check.py").read_bytes()).hexdigest(),
"skill_sha256": pilot.claude_code.githits_skill()[1]}
"skill_sha256": pilot.claude_code.githits_skill()[1], "cli_version": CLI_VERSION}
(work / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
return repo, path

Expand All @@ -53,7 +53,7 @@ def main(argv=None):
parser.add_argument("--work", type=Path, required=True, help="new private directory; never reused")
parser.add_argument("--factory-commit", required=True, help="exact approved factory SHA")
parser.add_argument("--run-approved", action="store_true", help="Mateo only, after explicit live/spend approval")
parser.add_argument("--claude", help="Claude Code 2.1.311 executable (else the existing configured command)")
parser.add_argument("--claude", help=f"Claude Code {CLI_VERSION} executable (else the existing configured command)")
args = parser.parse_args(argv)
factory = pilot.host_checkout()
if args.factory_commit != factory["commit"] or (args.run_approved and factory["dirty"]):
Expand All @@ -68,9 +68,12 @@ def main(argv=None):
os.environ[pilot.claude_code.BIN_ENV] = str(Path(args.claude).resolve())
command = pilot.claude_code.claude_executable()
version = subprocess.run([*command, "--version"], capture_output=True, timeout=30)
if version.returncode or not version.stdout.decode("utf-8", "replace").startswith(CLI_VERSION + " "):
print("REFUSED: the frozen smoke plan requires Claude Code", CLI_VERSION)
found = pilot.claude_code.version_tuple(version.stdout.decode("utf-8", "replace")) if not version.returncode else None
if found != tuple(int(part) for part in CLI_VERSION.split(".")):
print("REFUSED: the frozen smoke plan requires Claude Code", CLI_VERSION, "exactly; found",
".".join(map(str, found)) if found else "no readable version")
return 2
print("Claude Code", CLI_VERSION, "(the exact pin)")
if "--model" in command:
print("REFUSED: remove the command's model override; the smoke recipe pins ANTHROPIC_MODEL")
return 2
Expand Down
Loading
Loading