Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 20 additions & 7 deletions docs/CLAUDE_CODE.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/EXPERIMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,9 @@ The pilot itself runs a real model with a paid key: Class X (`AGENTS.md`). A nam
1. The item list, as the `--items` file: each work item's id and its tag, `needs_lookup` when the work depends on a library or API the agent is unlikely to know well (new, niche, or changed since the model's training), else `routine`; beside it, each item's objective and scope.
2. The primary metric and the group the decision rests on (for example: the delivery rule on `needs_lookup` items, with `--delivered review`).
3. The exclusion rules: the nine excluded checks above, and nothing else.
4. The frozen setup: the SyberWork commit, Claude Code version, model, effort, `--allowed-tools`, `--max-turns`, the query guard's thresholds, the credential lane, and arm B's GitHits route (`--githits`) with its license policy (`SYBERLABS_GITHITS_LICENSES`, recorded as `signal.githits.license_policy`). Use the `api` lane (`--bare`) in both arms: it is the lane whose cost is billed and known, and in bare mode the tree's `CLAUDE.md` is not read in either arm. The contract prices every model the runs report (`evolution.budget.prices`); a run that used a model the contract does not price has an unknown cost. Anything that learns from runs stays off for the week. AutoSkill is a design (`docs/AUTOSKILL.md`) and is in no proposer path; the `jev` and `empirical` branch policies belong to `--evolve`, which the pilot does not use.
4. The frozen setup: the SyberWork commit, Claude Code version, model, effort, `--allowed-tools`, `--max-turns`, the query guard's thresholds, the credential lane, and arm B's GitHits route (`--githits`) with its license policy (`SYBERLABS_GITHITS_LICENSES`, recorded as `signal.githits.license_policy`). Use the `api-hooked` lane (`--lane api-hooked --trusted-tree`) in both arms: the API key, so the cost is billed and known, without `--bare`, because Claude Code 2.1.312 runs no `--settings` hook under `--bare` and the provider refuses `--githits` there (`docs/CLAUDE_CODE.md`, the hooked API lane); each call gets an empty config directory and home. Not in bare mode, the tree's `CLAUDE.md` is read, equally in both arms: record it with the conditions. The contract prices every model the runs report (`evolution.budget.prices`); a run that used a model the contract does not price has an unknown cost. Anything that learns from runs stays off for the week. AutoSkill is a design (`docs/AUTOSKILL.md`) and is in no proposer path; the `jev` and `empirical` branch policies belong to `--evolve`, which the pilot does not use.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the pilot commands to use the hooked API lane

The following “Each item, each arm” commands still use --bare, and arm B combines it with --githits. This commit makes --githits with --bare fail with invalid_provider, so readers following the pilot instructions cannot run it; update both commands to use --lane api-hooked --trusted-tree.

Useful? React with 👍 / 👎.


**Before the week, the dry run** (Class X as well, a few dollars): `python scripts/pilot_dry_run.py --plan plan.json` runs this protocol on 3 to 5 items on the API route (`--githits`, `docs/GITHITS.md`, "The A/B report"), in a home of its own: a preflight that refuses to start on any failure, one canary call that shows the query guard fires under `--bare`, both arms per item from one `HEAD` in a seeded order, the blind review and this report under a hard budget cap (default $8). It passes when the nine metrics are known for every thread and the report compares every item; its docstring says how to run it, and its `receipt.json` and `report.md` are what to send.
**Before the week, the dry run** (Class X as well, a few dollars): `python scripts/pilot_dry_run.py --plan plan.json` runs this protocol on 3 to 5 items on the API route (`--githits`, `docs/GITHITS.md`, "The A/B report"), in a home of its own: a preflight that refuses to start on any failure, one canary call that shows the query guard fires on the `api-hooked` lane, both arms per item from one `HEAD` in a seeded order, the blind review and this report under a hard budget cap (default $8). It passes when the nine metrics are known for every thread and the report compares every item; its docstring says how to run it, and its `receipt.json` and `report.md` are what to send.

**Each item, each arm, from the same `HEAD`** (accepting moves only the thread's own branch, so `HEAD` stays put):

Expand Down
54 changes: 42 additions & 12 deletions docs/GITHITS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,24 @@ not be shown SyberLabs code as if it were outside evidence. For the pilot, conta
side: the work items are unpublished, both arms run locally on a pinned base, and this client is the backstop. An
exclusion at GitHits' index is optional, not something the pilot relies on.

The client works in three layers. The exclusion list is `SyberLabs` plus whatever `SYBERLABS_GITHITS_EXCLUDE` adds:
a comma list of owners or `owner/repo`, matched case-insensitively. The variable only adds; it never removes
`SyberLabs`, and an empty value keeps the default. `SYBERLABS_GITHITS_EXCLUDE_REPLACE=1` makes the variable's list
The client works in three layers. The exclusion list is the default deny-list plus whatever
`SYBERLABS_GITHITS_EXCLUDE` adds, matched case-insensitively. The default (`githits.DEFAULT_EXCLUDE`, from the
pilot's pre-registration) is:

- the GitHub owners `SyberLabs` and `sykosyber` (`github.com/sykosyber/*`, and each owner's own domains and
`*.github.io` pages, as for any owner);
- `pypi:syberwork`, the PyPI name of this repository's distribution (`pyproject.toml`; there is no npm package), as a
target and as `pypi.org/project/syberwork` in a result;
- `deepwiki.com` and `deepwiki.org` with their subdomains, the DeepWiki front-ends that re-serve public repositories,
SyberLabs' among them, as a target (`site:deepwiki.com/...`, a URL) and anywhere in a result's text;
- forks of SyberLabs repositories where the result's URL shows one: a repository named like a SyberLabs product
(`githits.FORK_NAMES`: `syberwork`, `sybersdk`, `syberfactory`, `syberlabs`, `syberstate`) under any owner on a
code host (`github.com/<anyone>/SyberWork`, `github:<anyone>/syberfactory`). This is by name only: a fork renamed
to something else is not detected.

An entry of the variable is an owner, `owner/repo`, a package `registry:name` (`npm:left-pad`) or a host name with a
dot and no slash (`example.org`, which also covers its subdomains). The variable only adds; it never removes the
default, and an empty value keeps it. `SYBERLABS_GITHITS_EXCLUDE_REPLACE=1` makes the variable's list
replace the default; it exists for tests only, and the Claude Code provider never passes it to the launcher's client.
Passing another list to `Client(exclude=...)` in code replaces the default too.

Expand Down Expand Up @@ -127,21 +142,29 @@ the two is an error, so a run never believes it is recording when it is not.
{"op": "search", "query_digest": "<sha256>", "targets_count": 1, "status": "ok", "http_status": 200,
"request_id": "<X-Request-ID>", "results_returned": 3, "results_dropped": 1, "results_dropped_license": 0,
"license_policy": "<sha256>", "contamination_blocked": false, "latency_ms": 412, "rate_limit_remaining": 97,
"at": "2026-10-26T17:01:16.123Z", "mac": "<hmac-sha256>"}
"at": "2026-10-26T17:01:16.123Z", "sent": true, "mac": "<hmac-sha256>"}
```

- `status` is `ok`, `excluded_target`, `contamination_blocked` or `error`.
- `status` is `ok`, `excluded_target`, `contamination_blocked`, `error`, or `unguarded` (the launcher's entry
refused the call before sending: no query-guard decision for it; below).
- `sent` says whether the request was written to GitHits: `false` when it never left (an excluded target, an
`unguarded` refusal, no token, a connection, TLS or proxy failure), `true` once it was (an answer, an HTTP error,
a read timeout, a connection closed before the answer), `null` when the call was interrupted in flight.
- `query_digest` is the SHA-256 of the canonical JSON request: the body of a POST, the query parameters of a
read.
- `rate_limit_remaining` is `X-RateLimit-Remaining` when the answer has it. The spec documents it for every
operation here except `example`.
operation here except `example`. Observed live (pilot canary, 2026-10-30): `search` and `grep` answers carry it
(599, then 598), `example` and `doctor` answers do not (`null`); `read` and `list` were not called.
- `mac` is HMAC-SHA256 under the record key over the canonical JSON of every other field. It is
`query_guard.sign`, the same canonical form and MAC the query guard's decisions use.
- `read_records(directory, key)` returns the records whose MAC verifies, and counts every other line (bad MAC,
extra or missing field, not JSON) as ignored.

`paced_ms`, present only on a call the host's launcher paced (next section), is how long the launcher held the
call before sending it; the MAC covers it, and a record without it has exactly the fields above.
call before sending it; the MAC covers it, and a record without it has exactly the fields above. Every launcher
`example` to GitHits goes through the pacing lock, so `paced_ms` is present on it even when nothing was waited for,
with a small value: the lock and the time file's own cost (the canary's first example recorded `paced_ms: 4`; its
second, held for the slot, `26312`). Read a small value as "not held", not as a wait.

No query text, no result and no token is ever stored. As with the guard's key, the MAC shows that a process
holding the key wrote the line. The key is in the environment of whatever runs the client, so it does not show
Expand Down Expand Up @@ -221,6 +244,13 @@ five lookups, `search`, `example`, `read`, `list` and `grep` (`githits.LOOKUPS`,
`--license-mode`: through the launcher an example always asks for `license_mode: strict`. `doctor`, `--help`, an
unknown operation and a usage error exit 2 with a usage message before a client exists, so nothing is sent and
nothing is recorded. The operator keeps `syberlabs githits doctor` and `--license-mode`.

A lookup is sent only when the query guard's hook passed that launcher call: the entry claims the hook's
MAC-verified decision whose `arguments_digest` equals its own argv's (`query_guard.claim_launch`; one decision, one
launch) before it paces or sends. Without one (no hook ran, as under `claude --bare` on 2.1.312; no guard key in
the environment; a forged, mismatched or used decision) it records the call `status: unguarded`, `sent: false`,
prints `refused: unguarded: ...` and exits 2, with no HTTP request (`docs/CLAUDE_CODE.md`, the pre-send refusal).
`syberlabs githits` (the operator's) has no such gate.
It prints as `syberlabs` does, unencodable characters replaced, and before the record: a failed print records `error`.

## The pinned contract
Expand Down Expand Up @@ -262,13 +292,13 @@ before its first candidate and keep everything else equal, `--allowed-tools` inc
```bash
syberlabs start "Parse ISO dates with the UTC offset" --paths src tests --unattended
syberlabs experiment label --name githits-pilot --item LIB-1 --arm off --tag needs_lookup
syberlabs propose --claude-code --bare --allowed-tools Read,Edit,Write,Glob,Grep,WebSearch,WebFetch
syberlabs propose --claude-code --lane api-hooked --trusted-tree --allowed-tools Read,Edit,Write,Glob,Grep,WebSearch,WebFetch
syberlabs run --once

syberlabs start "Parse ISO dates with the UTC offset" --paths src tests --unattended
syberlabs experiment label --name githits-pilot --item LIB-1 --arm on --tag needs_lookup
syberlabs propose --claude-code --bare --allowed-tools Read,Edit,Write,Glob,Grep,WebSearch,WebFetch \
--githits --githits-arm on # the only difference: the launcher and the skill
syberlabs propose --claude-code --lane api-hooked --trusted-tree --allowed-tools Read,Edit,Write,Glob,Grep,WebSearch,WebFetch \
--githits --githits-arm on # the only difference: the launcher and the skill (never with --bare)
syberlabs run --once

syberlabs experiment report --name githits-pilot --items items.txt --delivered review
Expand Down Expand Up @@ -331,8 +361,8 @@ misreports. Claim ceiling: implemented and tested with stand-ins; no pilot has r
| `GITHITS_API_TOKEN` | the bearer token (required for a call; a refused target needs none) |
| `GITHITS_API_URL` | base URL, default `https://api.githits.dev/v1` |
| `GITHITS_SESSION_ID`, `GITHITS_AGENT` | optional `X-GitHits-Session-ID` and `X-GitHits-Agent` |
| `SYBERLABS_GITHITS_EXCLUDE` | owners or `owner/repo` to exclude besides `SyberLabs`, comma list (it only adds) |
| `SYBERLABS_GITHITS_EXCLUDE_REPLACE` | `1`: the list above replaces `SyberLabs` (tests only) |
| `SYBERLABS_GITHITS_EXCLUDE` | owners, `owner/repo`, `registry:name` packages or hosts to exclude besides the default deny-list (`SyberLabs`, `sykosyber`, `pypi:syberwork`, `deepwiki.com`, `deepwiki.org`, and forks by name), comma list (it only adds) |
| `SYBERLABS_GITHITS_EXCLUDE_REPLACE` | `1`: the list above replaces the default deny-list, the fork rule included (tests only) |
| `SYBERLABS_GITHITS_LICENSES` | allowed SPDX ids, comma list, or `any` (default: the permissive list) |
| `SYBERLABS_GITHITS_RECORD_DIR`, `SYBERLABS_GITHITS_RECORD_KEY` | where records go and their MAC key (both or neither) |
| `SYBERLABS_GITHITS_PACE_DIR` | the host's example pacing directory, absolute (default: `syberlabs-githits-pace` in the temporary directory) |
Expand Down
27 changes: 16 additions & 11 deletions examples/githits_ab.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
``syberlabs experiment report`` over it. Two routes for arm B's GitHits:

- ``--route api`` (the default; the pilot's route since #134): both arms run the real Claude Code proposer,
``ClaudeCodeProvider`` on the api lane, arm B with ``githits=True``. For each call the host writes the GitHits
``ClaudeCodeProvider`` on the api-hooked lane (the API key without ``--bare``, so the guard's hook runs), arm B
with ``githits=True``. For each call the host writes the GitHits
launcher, its query guard judges every lookup and command, the host's own GitHits client makes each lookup and signs
its record, and the host journals one ``lookup`` per record and its ledger of the call on the candidate
(``signal.githits``), which is what the report reads the arm from. Only the ``claude`` executable is a stand-in: a
Expand Down Expand Up @@ -290,7 +291,7 @@ def run(name, tool_input):
Path(path).write_bytes(text.encode("utf-8")) # the exact text: no newline translation on Windows
model, usage = "stand-in-model", {"input_tokens": tokens, "output_tokens": 350}
say({"type": "system", "subtype": "init", "claude_code_version": "2.1.309", "model": model, "effort": "high",
"apiKeySource": "ANTHROPIC_API_KEY" if "--bare" in sys.argv else "none", "mcp_servers": [],
"apiKeySource": "ANTHROPIC_API_KEY" if os.environ.get("ANTHROPIC_API_KEY") else "none", "mcp_servers": [],
"session_id": "rehearsal", "tools": ["Read", "Edit", "Write"] + (["Bash"] if launcher else [])})
say({"type": "assistant", "message": {"id": "msg_1", "model": model, "usage": usage,
"content": [{"type": "text", "text": "Working on " + item}]}})
Expand Down Expand Up @@ -367,14 +368,16 @@ def close(self) -> None:

@contextlib.contextmanager
def api_environment(url: str):
"""The API route's environment, put back afterwards: GitHits at the loopback stand-in under a placeholder token, no
GitHits setting from the shell, and no gateway or provider switch (``claude_code.THIRD_PARTY_SWITCHES`` would refuse
the api lane; the stand-in sends nothing anywhere)."""
"""The API route's environment, put back afterwards: GitHits at the loopback stand-in under a placeholder token, a
placeholder Anthropic key (the api-hooked lane requires one; the stand-in sends nothing anywhere), no GitHits
setting from the shell, and no gateway or provider switch (``claude_code.THIRD_PARTY_SWITCHES`` would refuse the
lane)."""
saved = dict(os.environ)
for name in (*THIRD_PARTY_SWITCHES, githits.EXCLUDE_ENV, githits.EXCLUDE_REPLACE_ENV, githits.LICENSES_ENV,
githits.SESSION_ENV, githits.AGENT_ENV, githits.RECORD_DIR_ENV, githits.RECORD_KEY_ENV):
os.environ.pop(name, None)
os.environ.update({githits.TOKEN_ENV: "rehearsal-placeholder-not-a-token", githits.URL_ENV: url})
os.environ.update({githits.TOKEN_ENV: "rehearsal-placeholder-not-a-token", githits.URL_ENV: url,
"ANTHROPIC_API_KEY": "rehearsal-placeholder-not-a-key"})
try:
yield
finally:
Expand All @@ -389,11 +392,13 @@ def git(root: Path, *args: str) -> None:


def claude_code(kit: Kit, thread, script: Path, item: str, arm: str) -> ClaudeCodeProvider:
"""The API route's proposer, as the pilot builds it: ``syberlabs propose --claude-code --bare`` in arm A, the same
with ``--githits --githits-arm on`` in arm B, except on the misconfigured item, whose treatment forgot
``--githits``. The executable is the stand-in; everything else is the real provider."""
return ClaudeCodeProvider(kit.repo, bare=True, executable=[sys.executable, str(script)], max_calls=3,
githits=arm == "on" and item != MISCONFIGURED, arm=arm, thread_label=thread.id)
"""The API route's proposer, as the pilot builds it: ``syberlabs propose --claude-code --lane api-hooked
--trusted-tree`` in arm A, the same with ``--githits --githits-arm on`` in arm B (never ``--bare``: Claude Code
2.1.312 runs no ``--settings`` hook there), except on the misconfigured item, whose treatment forgot ``--githits``.
The executable is the stand-in; everything else is the real provider."""
return ClaudeCodeProvider(kit.repo, lane="api-hooked", trusted_tree=True, executable=[sys.executable, str(script)],
max_calls=3, githits=arm == "on" and item != MISCONFIGURED, arm=arm,
thread_label=thread.id)


@contextlib.contextmanager
Expand Down
Loading
Loading