diff --git a/docs/CLAUDE_CODE.md b/docs/CLAUDE_CODE.md index 5521e63a..089af68e 100644 --- a/docs/CLAUDE_CODE.md +++ b/docs/CLAUDE_CODE.md @@ -136,7 +136,7 @@ syberlabs propose --claude-code --bare --githits # arm B: the same, plus t exec '' -I '/syberlabs/githits_cli_entry.py' "$@" ``` -`githits_cli_entry.py` puts the host checkout first on `sys.path`, refuses to run if `syberlabs` came from anywhere else, and runs the host's GitHits client with the arguments. It accepts only the five lookups (`search`, `example`, `read`, `list`, `grep`); `doctor` and anything else exit 2 with a usage message and write no record, and `example` takes no `--license-mode`, so it always sends `license_mode: strict` (`docs/GITHITS.md`, the launcher's entry). `doctor` stays the operator's: `syberlabs githits doctor`. `-I` keeps the worktree's directory and `PYTHONPATH` off the import path, so the candidate's own `syberlabs/` never runs. Claude Code's Bash tool runs bash (Git Bash on Windows), which runs the script. The launcher is named by its absolute path with forward slashes, the exact text the model types; a path holding a space or a quote is refused (`provider_unavailable`). +`githits_cli_entry.py` puts the host checkout first on `sys.path`, refuses to run if `syberlabs` came from anywhere else, and runs the host's GitHits client with the arguments. It accepts only the five lookups (`search`, `example`, `read`, `list`, `grep`); `doctor` and anything else exit 2 with a usage message and write no record, and `example` takes no `--license-mode`, so it always sends `license_mode: strict` (`docs/GITHITS.md`, the launcher's entry). `doctor` stays the operator's: `syberlabs githits doctor`. An `example` is held on the host until this host's last one is 40 s old, across processes and arms (`syberlabs/githits_pace.py`). `-I` keeps the worktree's directory and `PYTHONPATH` off the import path, so the candidate's own `syberlabs/` never runs. Claude Code's Bash tool runs bash (Git Bash on Windows), which runs the script. The launcher is named by its absolute path with forward slashes, the exact text the model types; a path holding a space or a quote is refused (`provider_unavailable`). **What arm B adds.** Argv: `Bash` in `--tools`, `Bash(:*)` in `--allowedTools`, and the skill after the harness note in `--append-system-prompt`. Environment, for the launcher's client: @@ -158,7 +158,7 @@ exec '' -I '/syberlabs/githits_cli_entry.py' "$@" **PowerShell under the override.** An operator's `Bash` or `PowerShell` entry is refused with `--githits` unless `--allow-code-tools-with-mcp` is passed. With the override, the matcher is `^(mcp__githits__.*|WebSearch|WebFetch|Bash|PowerShell)$` (`query_guard.MATCHER_SHELLS`), and a `PowerShell` call is judged by the same launcher-only rule as `Bash` (`query_guard.SHELL_TOOLS`): any command but the launcher is `not_launcher`, so an allowed PowerShell no longer runs unguarded. Its refusals are journaled like a refused Bash call (tool `PowerShell`, its rule) and counted in `signal.githits.bash_refused` and `lookups_blocked`. Without the override the matcher is the Bash one above. -**What is recorded.** After the call the provider reads the client's records (`githits.read_records` under the call's record key). Each authentic record becomes one verified `lookup` fact, tool `githits.`, value `{tool, query_digest, request_id, status, at, results_returned, results_dropped, results_dropped_license, latency_ms, contamination_blocked, sent, ok, guard}`: the client's own digest of the request it sent, never text; `sent` is the record's, whether the request was written to GitHits (false for an excluded target or a connection that never opened, true from a timeout after the request left to an answer, null for a call interrupted in flight). The client records every call it starts, an answer it cannot read and an interruption included; the launcher's entry turns a `SIGTERM` into an exception for that, so a launcher ended by a Bash timeout records its call (a `SIGKILL` cannot be caught). Forged or altered record lines are counted (`signal.githits.records_ignored`) and ignored. Every `Bash` call the guard did not pass is journaled as a `lookup` with tool `Bash` and its rule (or `unguarded`), which `lookups_blocked` counts. `signal.tool_usage.githits` is the number of authentic records (plus any `mcp__githits__*` call the stream shows). The candidate also carries `signal.githits: {arm, launcher_calls, launcher_passed, records, records_ignored, bash_refused, license_policy[, mismatch]}` and `signal.harness.skill: {path, sha256}`. `githits_lookups` never counts a `githits.doctor` fact. +**What is recorded.** After the call the provider reads the client's records (`githits.read_records` under the call's record key). Each authentic record becomes one verified `lookup` fact, tool `githits.`, value `{tool, query_digest, request_id, status, at, results_returned, results_dropped, results_dropped_license, latency_ms, contamination_blocked, sent, ok, guard}`, plus `paced_ms` when the launcher held an `example` to keep GitHits' per-organization rate (`docs/GITHITS.md`, example pacing): the client's own digest of the request it sent, never text; `sent` is the record's, whether the request was written to GitHits (false for an excluded target or a connection that never opened, true from a timeout after the request left to an answer, null for a call interrupted in flight). The client records every call it starts, an answer it cannot read and an interruption included; the launcher's entry turns a `SIGTERM` into an exception for that, so a launcher ended by a Bash timeout records its call (a `SIGKILL` cannot be caught). Forged or altered record lines are counted (`signal.githits.records_ignored`) and ignored. Every `Bash` call the guard did not pass is journaled as a `lookup` with tool `Bash` and its rule (or `unguarded`), which `lookups_blocked` counts. `signal.tool_usage.githits` is the number of authentic records (plus any `mcp__githits__*` call the stream shows). The candidate also carries `signal.githits: {arm, launcher_calls, launcher_passed, records, records_ignored, bash_refused, license_policy[, mismatch]}` and `signal.harness.skill: {path, sha256}`. `githits_lookups` never counts a `githits.doctor` fact. A record cannot be tied to the tool use that ran it, so the host reconciles counts: authentic records against launcher calls the guard passed. Fewer records is `mismatch: githits_unrecorded` (a usage error, a client that failed before recording, a denied permission; recorded, the call stands). More records than guarded calls is `githits_unexpected_records`: each record of the call is journaled `guard: unguarded`, and the search stops `provider_guard_inactive`, because something holding the key ran the client without the guard. A call that fails or times out registers no candidate, so its mismatch would reach the journal nowhere, and a launcher call still waiting for GitHits when the host ends Claude Code (its session is killed, so the client cannot record) would leave no trace: when such a call has fewer records than guarded launcher calls, its records are journaled `guard: unguarded` (not reconciled), and when it has none, one `githits.unrecorded` fact (the command line's digest, `sent: null`, `guard: unguarded`) stands for each launcher call the guard passed. A `no_change` refusal with missing records journals a synthetic gap without assigning it to a concurrent command; its count remains unknown (`spec/GAPS.md` 59 (12)). diff --git a/docs/GITHITS.md b/docs/GITHITS.md index 2a9df027..c6c25c8a 100644 --- a/docs/GITHITS.md +++ b/docs/GITHITS.md @@ -94,8 +94,18 @@ unbalanced parenthesis, a dangling operator, a stray character, more than 64 tok `results_dropped_license`, apart from the contamination count. Contamination is judged first. In contract 0.1.0 only an example carries licenses: the `(SPDX)` after each source link of its Markdown (`[Source 1](https://github.com/o/r/blob/sha/path) (MIT)`), a `License:` line, or a structured `references` - list if one is ever returned. An example whose sources include a license that is not admitted, a source link - with no license, or no source link at all, is dropped whole. Exit 0, with a one-line notice. + list if one is ever returned. A source link is a link to a code host, or any link under a `References` or + `Sources` heading (up to the next heading or thematic break), which is how GitHits' documented responses cite + them (`### References` / `- [Source 1](https://example.org/source) (MIT)`); GitHits' own "Open in GitHits Web + App" link is never one. An example whose sources include a license that is not admitted, a source link + with no license, or code with no source link at all, is dropped whole. Exit 0, with a one-line notice. An + example with neither code (a fenced, indented or HTML block) nor a source carries nothing to license and + passes: GitHits' documented `degraded` ("Source search may be incomplete.") and `no_example` ("No suitable + example was found.") answers. All three documented example responses in the pinned OpenAPI document pass, and + the same shapes with a copyleft, `NOASSERTION`, unparsable or missing license are dropped + (`tests/test_githits.py`, `DocumentedExamples`). Before 2026-10-29 all three were withheld + (`results_dropped_license=1`): the `complete` example's source is not on a code host, and the other two have + no source. - **It depends on license data being present.** No search, grep, list or read result in 0.1.0 has a license field, so those results pass this check unjudged. The client would read `license_spdx_id`, `spdx_id`, `license_id` or `license` on a result (or a search hit's locator) if GitHits added one. @@ -130,10 +140,35 @@ the two is an error, so a run never believes it is recording when it is not. - `read_records(directory, key)` returns the records whose MAC verifies, and counts every other line (bad MAC, extra or missing field, not JSON) as ignored. +`paced_ms`, present only on a call the host's launcher paced (next section), is how long the launcher held the +call before sending it; the MAC covers it, and a record without it has exactly the fields above. + No query text, no result and no token is ever stored. As with the guard's key, the MAC shows that a process holding the key wrote the line. The key is in the environment of whatever runs the client, so it does not show which process that was. +## Example pacing (host side) + +GitHits limits its example endpoint per organization to one or two requests a minute, and the pilot runs both arms +and every Claude Code call under one organization's token. The launcher's entry (`githits_cli_entry.py`) therefore +holds each `example` before it is sent until this host's last example is 40 s old (`githits_pace.EXAMPLE_INTERVAL`): +a cross-process lock on `example.lock` and the time of the last slot in `example.last`, in the host's pacing +directory, outside any worktree (`SYBERLABS_GITHITS_PACE_DIR`, an absolute path, else `syberlabs-githits-pace` in the +system temporary directory). Callers queue on the lock, so two examples from any two processes, either arm, are at +least 40 s apart. Search, read, list and grep are not paced, and `syberlabs githits example` (the operator's) is not +paced either. + +This is pacing, not a retry: the client still sends each request once and never waits or retries on its own (a +missing retry and back-off is itself a pilot work item, so it is kept out of `githits.py`'s request code). The time +the call was held is recorded on its client record as `paced_ms` and on its `lookup` fact; `latency_ms` never +includes it. `syberlabs experiment report` leaves it out of an arm's time: `time_to_pass` subtracts the pacing between +the first search and the evaluation of the accepted candidate, the per-arm `agent_time` subtracts the thread's total, +and each arm reports its `paced_seconds`. + +An example to a loopback base URL (a stand-in, such as the tests' fake GitHits) never reaches GitHits' limit and is +not paced unless `SYBERLABS_GITHITS_PACE_SECONDS` sets an interval for it; that variable is ignored for any other +base. A pacing directory that cannot be used fails the call before anything is sent. Limits: `spec/GAPS.md` 63. + ## Token and transport - The token is read only from `GITHITS_API_TOKEN`, and is sent only in `Authorization: Bearer`. It is not in any @@ -300,6 +335,8 @@ misreports. Claim ceiling: implemented and tested with stand-ins; no pilot has r | `SYBERLABS_GITHITS_EXCLUDE_REPLACE` | `1`: the list above replaces `SyberLabs` (tests only) | | `SYBERLABS_GITHITS_LICENSES` | allowed SPDX ids, comma list, or `any` (default: the permissive list) | | `SYBERLABS_GITHITS_RECORD_DIR`, `SYBERLABS_GITHITS_RECORD_KEY` | where records go and their MAC key (both or neither) | +| `SYBERLABS_GITHITS_PACE_DIR` | the host's example pacing directory, absolute (default: `syberlabs-githits-pace` in the temporary directory) | +| `SYBERLABS_GITHITS_PACE_SECONDS` | a pacing interval for a loopback base URL only (tests); ignored for GitHits itself, always 40 s | Claim ceiling: **implemented and tested** against a loopback fake that follows the pinned schemas. It has not been run against the real API, so the real server's behaviour is not observed: whether `strict` already excludes diff --git a/docs/GITHITS_OPERATOR_HANDOFF.md b/docs/GITHITS_OPERATOR_HANDOFF.md index 6a3517a4..30607b57 100644 --- a/docs/GITHITS_OPERATOR_HANDOFF.md +++ b/docs/GITHITS_OPERATOR_HANDOFF.md @@ -32,10 +32,13 @@ Required before spending: - Approval naming provider, exact model/route, run plan, spending authority, scheduling budget and treatment of search fees and overshoot. -- Python 3.11+; Git; Claude Code **2.1.311** for this frozen smoke recipe; +- Python 3.11+; Git; Claude Code **2.1.312** exactly for this frozen smoke recipe (the version the host + and the pilot VM run; `CLI_VERSION`, recorded in `plan.json` and `manifest.json`); Git Bash on Windows. The Windows native CLI was downloaded from its official npm package, checked against npm's SHA-512 and exercised with version/help - only. Those flags are compatible; the live hook canary remains unrun. + only. Those flags are compatible; the live hook canary remains unrun. (That + download check was of 2.1.311; the pin moved to 2.1.312 on 2026-10-29 + because the host and the pilot VM run it, and 2.1.312 was not re-checked here.) - A plain temporary path without spaces or shell punctuation for launchers. - The exact approved candidate checked out cleanly. Do not reuse another worker's checkout or branch, and do not commit the plan or run directory. diff --git a/examples/githits-engineering-smoke/plan.json b/examples/githits-engineering-smoke/plan.json index 64f6e486..7d02d131 100644 --- a/examples/githits-engineering-smoke/plan.json +++ b/examples/githits-engineering-smoke/plan.json @@ -1,7 +1,7 @@ { "_purpose": "ENGINEERING SMOKE only: synthetic, visible checks, no formal items or held-out data; not an unbiased cost experiment", "_formal_pack": "SyberLabs/MasterMind claude/githits-pilot-pack at 09864ac6633832cc5d7c082ec731aff7a5689cad; proposal and formal baseline unchanged", - "_configuration": {"provider": "Anthropic direct API through Claude Code", "model": "claude-sonnet-5-5", "cli": "2.1.311", "effort": "medium", "cache_ttl": "5m", "pricing_checked": "2026-10-26", "pricing_source": "https://platform.claude.com/docs/en/about-claude/pricing"}, + "_configuration": {"provider": "Anthropic direct API through Claude Code", "model": "claude-sonnet-5-5", "cli": "2.1.312", "effort": "medium", "cache_ttl": "5m", "pricing_checked": "2026-10-26", "pricing_source": "https://platform.claude.com/docs/en/about-claude/pricing"}, "experiment": "githits-engineering-smoke-v1", "items": [ {"id": "ENGSMOKE-CSV", "tags": ["synthetic", "routine"], "objective": "Implement render(rows) in src/smoke_csv.py. Return CSV text for a list of name/qty dictionaries, with header name,qty, CRLF record endings and standard CSV quoting. Preserve names containing commas, quotes, Unicode and newlines, serialize integer quantities including negatives, and emit the header for empty input. Use only Python's standard library. Change only this new module; the visible host checks must pass.", "paths": ["src/smoke_csv.py"]}, diff --git a/scripts/githits_engineering_smoke.py b/scripts/githits_engineering_smoke.py index 98f1065b..6800516d 100644 --- a/scripts/githits_engineering_smoke.py +++ b/scripts/githits_engineering_smoke.py @@ -20,7 +20,7 @@ ROOT = Path(__file__).resolve().parents[1] PACK = ROOT / "examples" / "githits-engineering-smoke" MODEL = "claude-sonnet-5-5" -CLI_VERSION = "2.1.311" +CLI_VERSION = "2.1.312" # exact: the host and the pilot VM run it; recorded in plan.json and manifest.json def prepare(work): @@ -43,7 +43,7 @@ def prepare(work): "factory": pilot.host_checkout(), "fixture_base": pilot.git(repo, "rev-parse", "HEAD"), "plan_sha256": hashlib.sha256(path.read_bytes()).hexdigest(), "check_sha256": hashlib.sha256((PACK / "check.py").read_bytes()).hexdigest(), - "skill_sha256": pilot.claude_code.githits_skill()[1]} + "skill_sha256": pilot.claude_code.githits_skill()[1], "cli_version": CLI_VERSION} (work / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") return repo, path @@ -53,7 +53,7 @@ def main(argv=None): parser.add_argument("--work", type=Path, required=True, help="new private directory; never reused") parser.add_argument("--factory-commit", required=True, help="exact approved factory SHA") parser.add_argument("--run-approved", action="store_true", help="Mateo only, after explicit live/spend approval") - parser.add_argument("--claude", help="Claude Code 2.1.311 executable (else the existing configured command)") + parser.add_argument("--claude", help=f"Claude Code {CLI_VERSION} executable (else the existing configured command)") args = parser.parse_args(argv) factory = pilot.host_checkout() if args.factory_commit != factory["commit"] or (args.run_approved and factory["dirty"]): @@ -68,9 +68,12 @@ def main(argv=None): os.environ[pilot.claude_code.BIN_ENV] = str(Path(args.claude).resolve()) command = pilot.claude_code.claude_executable() version = subprocess.run([*command, "--version"], capture_output=True, timeout=30) - if version.returncode or not version.stdout.decode("utf-8", "replace").startswith(CLI_VERSION + " "): - print("REFUSED: the frozen smoke plan requires Claude Code", CLI_VERSION) + found = pilot.claude_code.version_tuple(version.stdout.decode("utf-8", "replace")) if not version.returncode else None + if found != tuple(int(part) for part in CLI_VERSION.split(".")): + print("REFUSED: the frozen smoke plan requires Claude Code", CLI_VERSION, "exactly; found", + ".".join(map(str, found)) if found else "no readable version") return 2 + print("Claude Code", CLI_VERSION, "(the exact pin)") if "--model" in command: print("REFUSED: remove the command's model override; the smoke recipe pins ANTHROPIC_MODEL") return 2 diff --git a/spec/GAPS.md b/spec/GAPS.md index f3f781cc..f310976f 100644 --- a/spec/GAPS.md +++ b/spec/GAPS.md @@ -123,3 +123,5 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e 61. **TypeSafe's Jev decides inside one search from ids and numbers; its decisions are not journaled by the host; its spend is outside the host's dollars; it cannot be registered as a SyberEval arm.** `TypeSafeJevPolicy` (`syberlabs/jev_api.py`, `docs/EVOLUTION.md`, 2026-10-28) asks `jev-1.13.0` one `choice` per decision over the actions `JevPolicy` offers, validates the answer and falls back to `JevPolicy`. What is open: (1) the decision record (`source`, `model`, `answers_digest`, `request_digest`, `latency_ms`, `fallback`, `local_action`) lives in the provider's in-memory trace, as every policy's does; `GitSearchSpace.observe` accepts only `PROVIDER_FACTS = ("lookup",)` (`syberlabs/build.py`), outside this packet, so a provider cannot journal it, and `benchmarks/model_vs_patch.py` writes the entries beside each exported journal instead. The smallest change that closes it is a `policy_decision` key in `PROVIDER_FACTS`, bounded like `lookup`. (2) Jev's dollars (input tokens at TypeSafe's published $0.042 per million) are counted by the policy and the arm's row, not by the host's `max_dollars`, which sees `providers.run_json` usage only; at about $0.00003 a decision this does not move a bound, but it is not the host's observation. (3) The answer is a provider's signal: its probabilities are not calibrated against admission and were not repeatable on 2026-10-28 (one request, three answers, `mutate` at 0.58, 0.56, 0.53). (4) Jev sees no text, so it decides from the same numbers the local heuristic reads; whether that adds anything is the pre-registered measurement in `docs/EVOLUTION.md`, unrun. (5) `syberlabs/eval/spec.py` closes `ARM_KINDS` without `evolve_jevapi`, so that measurement is a plan in the docs and not an `evals/` spec. (6) The pin is `jev-1.13.0` while `jev-latest` is `jev-1.14.0`; TypeSafe's page does not state a deprecation policy, so a retired pin would show up as 400 `Unknown model`, fall back to local on every decision and switch after three. (7) Jev cannot be named by the contract, like every policy (entry 41). 62. **SyberEval scores, gates and reports its runs; some gate rules cannot be computed for some arms, the agent-facing safety tasks do not exist yet, and candidate code still runs beside the hidden checks.** `eval-score` (2026-10-26, landed on the `eval-core` verification fixes of 2026-10-28: the runner's lock, bundle verification, the per-run preflight gate, strict JSON, refused run-id collisions and finite numbers) adds `syberlabs/eval/gate.py`, `score.py` and `report.py` and `syberlabs eval score|report` (`docs/EVAL.md`); the required spec field `hidden_feedback` (`tails`, `status`, `none`), enforced through `RepairProvider`'s optional `feedback_filter`, which every other caller leaves unset; the hidden copy's hashing (`tasks.tree_digests`) at four points against the copy as staged (`tasks.hidden_changes`, which names a symbolic link by its target and tolerates pytest's own `.pytest_cache`), with the run outcome `not_acceptable`, reason `hidden_tampered`; a host root where the hidden copy and the kit home get random names outside the run's work directory; the append-only spend ledger the eval's cap counts, read strictly, which also records every run that raised (it replaces the `runs/failed.jsonl` of the verification fixes); and `syberlabs.eval` in the wheel. It closes entry 60's (1) and (8) and narrows its (3) and (4). The contract schema, the event kinds, the golden traces and the kernel's behaviour for every existing caller are unchanged. Open: (1) the agent-facing safety tasks (`evals/tasks/safety-*`, packet `eval-safety-fixtures`) and the gate's validation on them: no planted canary, forbidden action or unsafe path has yet been offered to an agent, and every gate rule is tested on synthetic journals and with stand-in adapters only; (2) `canary.provider_requests` is `not_computable` for `claude-code` arms, whose own model requests hold whatever Claude Code's tools read, and for `evolve` and `evolve_jev` arms, whose mutate requests' feedback is not recorded; such a run's gate is `incomplete`, never `pass`; (3) the policy is not in the journal, so `approval_or_prerequisite_skipped` checks that at least one approval came between a `needs_approval` decision and its effect, not the policy's count; (4) `credential_observed` matches credential shapes (`syberlabs._patterns.SECRET`, at a word boundary) and the values of the host's credential variables present when the run ended, not every possible secret, and full check outputs and provider stderr are not retained, so only their recorded tails are scanned; (5) the hidden copy is hashed when staged, after the proposal (one hash after every check run inside it, the repair loop's evaluations included), after `Thread.check` (which runs the checks again only when the chosen candidate's result is missing or stale, so it usually follows a check that ran nothing) and before acceptance; the gate's `hidden_tampered` rule is `not_computable` for a bundle missing a hash the runner always writes; so a rewrite undone between two hashes, or inside one proposal's evaluations, is not detected, and candidate code inside a check can still read the copy and print it into its output, which `tails` sends to the next request; only a sandbox per check closes this (research register N4); (6) `hidden_feedback` filters the `repair` and `claude-code` feedback path only: an evolve arm builds its mutate requests in `syberlabs/evolve.py`, which no filter reaches, so a spec may use evolve arms under `tails` only; (7) under `none` the repair loop still stops on the first candidate that passes the hidden checks, so they still choose among up to K attempts on the host's side, and a call after a failure starts from the failed candidate's tree; (8) the eval's recorded spend can exceed `total_dollars` by one call's cost, and a call in flight when the runner is killed outright is in no journal and no ledger line; (9) the model API takes no sampling seed, so a run's seed indexes a repeat and is not a reproducible draw; (10) E5's registration is deleted, because adding `hidden_feedback` moved its digest and `register` returns an existing registration of the same digest unchanged; it is registered again on a clean checkout of `main` before it runs; (11) the package reads `benchmarks/`, the adapters and `evals/` from a checkout, so the wheel's `syberlabs eval` needs one to run anything; (12) the design gaps the verification pinned as `expectedFailure` in `tests/test_eval_adversarial.py` stay open: a candidate that exits 0 at import passes every hidden check; under `tails` a candidate can print its hidden check into the next request (under `status` and `none` it cannot, and that test passes); a `claude-code` arm's model is a declaration the runner neither sets nor compares with the model that served; a `repo:` task's run repository holds the commits after its base; and a runner killed outright charges nothing (tested on POSIX only, where `SIGKILL` exists). + +63. **The GitHits pilot's example pacing and license reading rest on GitHits' documentation, not on observed traffic.** (1) The host paces `example` calls to one per 40 s per host (`syberlabs/githits_pace.py`, called by the launcher entry); GitHits' limit is per organization, so two hosts, or an operator's own `syberlabs githits example` (not paced), share the organization's budget without seeing each other's slots. (2) A loopback base URL is not paced unless `SYBERLABS_GITHITS_PACE_SECONDS` asks: a loopback proxy that forwards to GitHits would go unpaced. (3) Calls queued behind several others wait one interval each, so a Bash tool timeout shorter than the queue ends a call before it is sent, and the host then flags it `githits_unrecorded`. (4) The time file is plain text a process of the same user can rewrite; it paces cooperating processes and is no security boundary. (5) The license reading admits an example's `References`/`Sources` links by their `(SPDX)` and passes an example with neither code (a fenced, indented or HTML block) nor a source; inline code in prose is not read as code, and the three documented shapes (`complete`, `degraded`, `no_example` in the pinned OpenAPI document) are the only evidence of what real responses look like until the first live example is recorded. (6) The engineering smoke pins Claude Code 2.1.312 exactly because the host and the pilot VM run it; that version's flags were not re-checked here. diff --git a/syberlabs/claude_code.py b/syberlabs/claude_code.py index ad3f0a44..95d0c9f8 100644 --- a/syberlabs/claude_code.py +++ b/syberlabs/claude_code.py @@ -324,7 +324,9 @@ def is_launcher_call(use: dict, launcher: str | None) -> bool: def record_lookup(record: dict, guard: str) -> dict: """A ``lookup`` fact from one authentic client record: ``githits.`` and the record's counts, never text. - ``sent`` is the record's: whether the request was written to GitHits (null: not known).""" + ``sent`` is the record's: whether the request was written to GitHits (null: not known). ``paced_ms``, only when + the record carries it, is how long the host's launcher held the call to keep GitHits' example rate + (``githits_pace``); the experiment report leaves it out of the arm's agent time.""" op = record.get("op") if isinstance(record.get("op"), str) and re.fullmatch(r"[a-z]{1,16}", record["op"]) else "unknown" found = {"tool": CLIENT_TOOL + op} for field in ("query_digest", "request_id", "status", "at"): @@ -333,6 +335,8 @@ def record_lookup(record: dict, guard: str) -> dict: found[field] = _count(record.get(field)) found["contamination_blocked"] = record.get("contamination_blocked") is True found["sent"] = record.get("sent") if isinstance(record.get("sent"), bool) else None + if "paced_ms" in record: + found["paced_ms"] = _count(record.get("paced_ms")) found["ok"] = found["status"] == githits_client.OK and guard == query_guard.PASS found["guard"] = guard return found diff --git a/syberlabs/experiment.py b/syberlabs/experiment.py index f3445dc3..a944aa04 100644 --- a/syberlabs/experiment.py +++ b/syberlabs/experiment.py @@ -17,7 +17,10 @@ What it adds to the per-item metrics: - ``time_to_pass``: hours from the thread's first search to the host's evaluation of the candidate that was accepted. - The time the change then waited for a person, or for ``syberlabs run``, is left out on both arms. + The time the change then waited for a person, or for ``syberlabs run``, is left out on both arms, and so is the time + the host's GitHits launcher held example calls to keep GitHits' per-organization rate (``paced_ms`` on the + ``githits.`` lookup facts, ``githits_pace``): that wait is the host's, not the agent's, and is reported per arm + as ``paced_seconds``. ``agent_time`` (per arm) leaves it out the same way. - ``accepted``: the contract's promotion succeeded (``effect_succeeded``, or a verified reconciliation of it). Both arms of an item can be accepted; only one arm's pull request can merge, so ``agent_time`` (to merge) is reported per arm. - ``reviewed``: accepted, and a ``review`` fact on the accepted commit passed (unknown until one is recorded). @@ -198,6 +201,7 @@ class Run: work: metrics.WorkItemMetrics upto: metrics.WorkItemMetrics | None = None # the same fold over the events up to acceptance observed: int = 0 # lookup facts showing a lookup on the server, in the whole thread (``_observed``) + paced: float = 0.0 # seconds the host's launcher held GitHits calls, in the whole thread (``_paced``) routes: tuple[str, ...] = () # how its calls had GitHits: ``launcher``, ``mcp`` or both (``_routes``) blocked: int = 0 # GitHits MCP attempts blocked by the host, included in declared attempt counts @@ -284,6 +288,22 @@ def _routes(registrations: list[dict], server: str | None) -> tuple[str, ...]: return tuple(sorted(found)) +def _paced(events: list[dict], after: int | None = None, upto: int | None = None) -> float: + """Seconds the host's GitHits launcher held calls before sending them (``paced_ms`` on the verified + ``githits.`` lookup facts, ``githits_pace``), over the events with ``after < seq <= upto`` (either bound + open when None). Host time spent keeping GitHits' example rate, not the agent's.""" + total = 0 + for event in events: + if (after is not None and event["seq"] <= after) or (upto is not None and event["seq"] > upto): + continue + value = metrics.fact(event, LOOKUP) or {} + paced = value.get("paced_ms") + if (isinstance(value.get("tool"), str) and value["tool"].startswith(CLIENT_TOOL) and type(paced) is int + and paced > 0): + total += paced + return total / 1000 + + def _observed(events: list[dict], server: str | None) -> int: """The thread's verified ``lookup`` facts that show a lookup on the server: every ``mcp____*`` call the guard passed, and for GitHits each client record a guarded launcher call stands behind (``githits.``, guard @@ -374,9 +394,11 @@ def _run(case_id: str, events: list[dict], name: str, work: metrics.WorkItemMetr passed = next((e for e in reversed(events) if accept is not None and e["kind"] == "candidate_evaluated" and e["seq"] < accept["seq"] and e["body"].get("candidate") == candidate), None) if start is not None and passed is not None: - hours = round((as_seconds(passed["at"]) - as_seconds(start["at"])) / 3600, 4) + paced = _paced(events, start["seq"], passed["seq"]) + hours = round(max(0.0, as_seconds(passed["at"]) - as_seconds(start["at"]) - paced) / 3600, 4) time_to_pass = metrics.Metric(hours, (start["hash"], passed["hash"]), - "first search to the evaluation of the accepted candidate") + "first search to the evaluation of the accepted candidate" + + (f", less {paced:g} s the host held GitHits calls" if paced else "")) else: time_to_pass = metrics.Metric(None, (accept["hash"],) if accept else (), "not accepted" if accept is None else "no search or evaluation of the accepted " @@ -389,6 +411,7 @@ def _run(case_id: str, events: list[dict], name: str, work: metrics.WorkItemMetr base=_base(events), objective=_objective(events), server=_servers(registrations, server), models=_models(registrations), accepted=accepted, time_to_pass=time_to_pass, reviewed=_reviewed(events, accept, commit), work=work, upto=upto, observed=_observed(events, server), + paced=_paced(events), routes=_routes(registrations, server), blocked=sum( 1 for f in (metrics.fact(e, LOOKUP) or {} for e in events) if str(f.get("tool", "")).startswith(f"mcp__{server}__") @@ -600,9 +623,10 @@ def delivery_rule(rows: list[dict], baseline: str, treatment: str, delivered_by: def _per_arm(item_runs: Iterable[Run]) -> dict: - """Unpaired: what each arm's threads show for the metrics only one arm of a pair can reach.""" + """Unpaired: what each arm's threads show for the metrics only one arm of a pair can reach. ``agent_time`` + leaves out the time the host's launcher held GitHits calls (``paced_seconds``, the arm's total).""" found = list(item_runs) - hours = [float(r.work.agent_time.value) for r in found if r.work.agent_time.known] + hours = [max(0.0, float(r.work.agent_time.value) - r.paced / 3600) for r in found if r.work.agent_time.known] blocked = [] for run in found: try: @@ -615,7 +639,8 @@ def _per_arm(item_runs: Iterable[Run]) -> dict: "opened": sum(1 for r in found if r.work.opened.value is True), "delivered": sum(1 for r in found if r.work.delivered.value is True), "agent_time_known": len(hours), "agent_time_mean": round(statistics.fmean(hours), 6) if hours else None, - "lookups_blocked": sum(blocked) if blocked else None} + "lookups_blocked": sum(blocked) if blocked else None, + "paced_seconds": round(sum(r.paced for r in found), 3)} def report(events: Iterable[dict], name: str, contracts: Mapping | None = None, *, baseline: str = BASELINE, @@ -807,7 +832,9 @@ def table(found: dict) -> str: p = found["per_arm"][arm] lines.append(f"{arm:>5}: {p['threads']} threads, {p['accepted']} accepted, {p['opened']} pull requests opened, " f"{p['delivered']} merged, agent time to merge known for {p['agent_time_known']}" - + ("" if p["lookups_blocked"] is None else f", {p['lookups_blocked']} lookups blocked by the guard")) + + ("" if p["lookups_blocked"] is None else f", {p['lookups_blocked']} lookups blocked by the guard") + + (f", {p['paced_seconds']:g} s of GitHits pacing left out of its time" if p.get("paced_seconds") + else "")) for flag, names in found["checks"].items(): effect = "excluded" if flag in EXCLUDE else "kept" lines.append(f"check {flag} ({effect}): {', '.join(names)}: {CHECKS[flag]}") diff --git a/syberlabs/githits.py b/syberlabs/githits.py index 0c21f352..65f95aa9 100644 --- a/syberlabs/githits.py +++ b/syberlabs/githits.py @@ -45,11 +45,13 @@ expression (``spdx_allows``: ``AND``, ``OR``, ``WITH`` and parentheses, ``AND`` binding tighter than ``OR``), is on the allow list: copyleft (GPL, LGPL, AGPL, MPL, EPL, EUPL, CDDL, SSPL), ``NOASSERTION``, unlisted and missing licenses are dropped. In the 0.1.0 contract only an example carries licenses, as ``(SPDX)`` after each source link -of its Markdown (a link to a code host in any Markdown or HTML form: inline, titled, angle-bracketed, a reference -definition, an autolink, an anchor, or a bare URL directly followed by its license), plus any -``SPDX-License-Identifier:`` tag in its code; an example with a source link that has no license, or with no source -link, is dropped whole. Search, grep, list and read results carry no license field, so they pass this check -unjudged until GitHits adds one. ``SYBERLABS_GITHITS_LICENSES`` replaces +of its Markdown (a link to a code host, or any link under a ``References`` or ``Sources`` heading as GitHits' documented +responses write them, in any Markdown or HTML form: inline, titled, angle-bracketed, a reference definition, an +autolink, an anchor, or a bare URL directly followed by its license; GitHits' own app link is never a source), plus +any ``SPDX-License-Identifier:`` tag in its code; an example with a source link that has no license, or with code +and no source link, is dropped whole. An example with no code and no source (GitHits' documented ``degraded`` and +``no_example`` answers) carries nothing to license and passes. Search, grep, list and read results carry no license +field, so they pass this check unjudged until GitHits adds one. ``SYBERLABS_GITHITS_LICENSES`` replaces the allow list (a comma list of SPDX ids) or turns the client-side check off (``any``); every record carries the digest of the policy in force (``license_policy``). @@ -70,7 +72,10 @@ target, no token, a connection, TLS or proxy failure), true once it was (an answer, an HTTP error, a read timeout, a connection closed before the answer), null when the call was interrupted in flight. So a timeout after GitHits received the request and a refused connection no longer leave the same record. ``read_records`` returns the -records whose MAC verifies and counts the rest. +records whose MAC verifies and counts the rest. A record may also carry ``paced_ms`` (``RECORD_OPTIONAL``): the +milliseconds the host's launcher held the call before sending it, to keep GitHits' per-organization example rate +(``githits_pace``, ``run(pace=...)``). This client never waits or retries on its own; the field is present only when +the caller paced the call, and the MAC then covers it. """ from __future__ import annotations @@ -117,6 +122,7 @@ RECORD_FIELDS = ("op", "query_digest", "targets_count", "status", "http_status", "request_id", "results_returned", "results_dropped", "results_dropped_license", "license_policy", "contamination_blocked", "latency_ms", "rate_limit_remaining", "at", "sent") +RECORD_OPTIONAL = ("paced_ms",) # present only when the host's launcher paced the call (``run(pace=...)``) OK, EXCLUDED, BLOCKED, ERROR = "ok", "excluded_target", "contamination_blocked", "error" STATUSES = (OK, EXCLUDED, BLOCKED, ERROR) @@ -493,22 +499,45 @@ def satisfied(node) -> bool: re.IGNORECASE | re.MULTILINE) _CODE_HOST = re.compile(r"^https?://(?:[a-z0-9-]+\.)*(?:github\.com|gitlab\.com|codeberg\.org|bitbucket\.org" r"|githubusercontent\.com)(?::\d+)?/", re.IGNORECASE) +# GitHits' own pages (the "Open in GitHits Web App" link every documented example ends with) are never a source. +_GITHITS_HOST = re.compile(r"^https?://(?:[a-z0-9-]+\.)*githits\.(?:com|dev)(?::\d+)?(?:[/?#]|$)", re.IGNORECASE) +# A "References" or "Sources" heading (ATX, bold or a bare line, as the documented examples write "### References"): +# every link under it, up to the next heading or thematic break, is a source, whatever its host. +_SOURCES_HEADING = re.compile(r"^[ \t]{0,3}(?:#{1,6}[ \t]+)?\**(?:references|sources)\**[ \t]*:?[ \t]*#*[ \t]*$", + re.IGNORECASE | re.MULTILINE) +_SECTION_END = re.compile(r"^[ \t]{0,3}(?:#{1,6}(?:[ \t]|$)|(?:[-*_][ \t]*){3,}$)", re.MULTILINE) +# Code an agent could take: a fenced block, an indented block, or HTML code. An example without any, and without a +# source, carries nothing to license (GitHits' documented "degraded" and "no_example" answers). +_CODE = re.compile(r"^[ \t]{0,3}(?:`{3,}|~{3,})|^(?: {4}|\t)[ \t]*\S|<(?:pre|code)\b", re.IGNORECASE | re.MULTILINE) _LICENSE_LINE = re.compile(r"^[ \t]*(?:[-*][ \t]*)?\**licen[cs]es?\**[ \t]*:[ \t]*(.+?)[ \t]*$", re.IGNORECASE | re.MULTILINE) _SPDX_TAG = re.compile(r"SPDX-License-Identifier:[ \t]*(.*?)[ \t]*(?:\*/|-->)?[ \t]*$", re.IGNORECASE | re.MULTILINE) +def _source_sections(text: str) -> list: + """The ``(start, end)`` offsets of each ``References`` or ``Sources`` section of an example's Markdown.""" + spans = [] + for heading in _SOURCES_HEADING.finditer(text): + end = _SECTION_END.search(text, heading.end()) + spans.append((heading.end(), end.start() if end else len(text))) + return spans + + def example_licenses(data: dict) -> list: - """The licenses an example names for its sources: the ``(SPDX)`` after each code-host link of its Markdown, - whatever form the link takes (``_SOURCE_LINK``), ``License:`` lines, ``SPDX-License-Identifier:`` tags (in its - code too), and structured ``references`` when present. ``None`` stands for a source with no license, and an - example with no source at all is ``[None]``.""" + """The licenses an example names for its sources: the ``(SPDX)`` after each source link of its Markdown (a + code-host link, or any link in a ``References`` or ``Sources`` section, never GitHits' own pages), whatever form + the link takes (``_SOURCE_LINK``), ``License:`` lines, ``SPDX-License-Identifier:`` tags (in its code too), and + structured ``references`` when present. ``None`` stands for a source with no license. An example with code and no + source at all is ``[None]``; one with neither code nor source (``_CODE``) is ``[]``: nothing to license.""" found = [] text = data.get("result") if isinstance(data.get("result"), str) else "" + sections = _source_sections(text) for match in _SOURCE_LINK.finditer(text): groups = match.groups() url, license_text = next((groups[i], groups[i + 1]) for i in range(0, len(groups), 2) if groups[i]) - if _CODE_HOST.match(url): + if _GITHITS_HOST.match(url): + continue + if _CODE_HOST.match(url) or any(start <= match.start() < end for start, end in sections): found.append((license_text or "").strip() or None) found += [line.strip("* ") or None for line in _LICENSE_LINE.findall(text)] found += [tag.strip() or None for tag in _SPDX_TAG.findall(text)] @@ -517,7 +546,7 @@ def example_licenses(data: dict) -> list: if isinstance(reference, dict): value = next((reference[key] for key in LICENSE_KEYS if key in reference), None) found.append(value if isinstance(value, str) and value.strip() else None) - return found or [None] + return found or ([None] if _CODE.search(text) else []) # ------------------------------------------------------------------------------------------------------------ requests @@ -703,13 +732,18 @@ def _hit_excluded(hit, exclude: Exclusions) -> bool: # ------------------------------------------------------------------------------------------------------------ records +def _record_fields(record: dict) -> tuple: + """``RECORD_FIELDS``, plus each ``RECORD_OPTIONAL`` field the record holds a value for.""" + return RECORD_FIELDS + tuple(field for field in RECORD_OPTIONAL if record.get(field) is not None) + + def record_mac(record: dict, key: str) -> str: - return sign({field: record.get(field) for field in RECORD_FIELDS}, key) + return sign({field: record.get(field) for field in _record_fields(record)}, key) def write_record(directory, key: str, record: dict) -> None: """Append one record with its MAC as a single line. Raises OSError when the line cannot be written.""" - entry = {field: record.get(field) for field in RECORD_FIELDS} + entry = {field: record.get(field) for field in _record_fields(record)} entry["mac"] = record_mac(entry, key) folder = Path(directory) folder.mkdir(parents=True, exist_ok=True) @@ -747,11 +781,13 @@ def read_records(directory, key: str) -> RecordRead: except ValueError: ignored += 1 continue - if (not isinstance(entry, dict) or set(entry) != set(RECORD_FIELDS) | {"mac"} - or not verify({field: entry[field] for field in RECORD_FIELDS}, entry.get("mac"), key)): + fields = _record_fields(entry) if isinstance(entry, dict) else () + if (not isinstance(entry, dict) or set(entry) != set(fields) | {"mac"} + or ("paced_ms" in entry and not (type(entry["paced_ms"]) is int and entry["paced_ms"] >= 0)) + or not verify({field: entry[field] for field in fields}, entry.get("mac"), key)): ignored += 1 continue - found.append({field: entry[field] for field in RECORD_FIELDS}) + found.append({field: entry[field] for field in fields}) return RecordRead(found, ignored) @@ -868,18 +904,23 @@ def list(self, target: str, **options) -> Result: def grep(self, pattern: str, targets, **options) -> Result: return self.call(grep_request(pattern, targets, **options)) - def call(self, request: Request, deliver=None) -> Result: + def call(self, request: Request, deliver=None, paced_ms: int | None = None) -> Result: """Send ``request`` (unless a target is excluded), filter the answer and record the call. ``deliver``, when given, is called with the result before the record is written (the CLI prints there): if it raises any exception (including an interruption), the call is recorded ``error`` and ``GitHitsError("output_failed")`` is raised, so a record never says ``ok`` for a result its - caller did not receive.""" + caller did not receive. ``paced_ms``, when given, is how long the caller held the call before this (the + host launcher's pacing); it is recorded, never waited here, and ``latency_ms`` never includes it.""" started = time.monotonic() record = {"op": request.op, "query_digest": query_digest(request), "targets_count": len(request.targets), "status": ERROR, "http_status": None, "request_id": None, "results_returned": 0, "results_dropped": 0, "results_dropped_license": 0, "license_policy": self.licenses.digest, "contamination_blocked": False, "latency_ms": 0, "rate_limit_remaining": None, "sent": False} + if paced_ms is not None: + if type(paced_ms) is not int or paced_ms < 0: + raise GitHitsError("bad_pacing", "paced_ms is a non-negative integer") + record["paced_ms"] = paced_ms if (any(self.exclude.names(target) for target in request.targets) or any(self.exclude.mentioned(path) for path in request_paths(request))): result = Result(request.op, EXCLUDED, None, None, None, 0, 0, False, None, @@ -1305,10 +1346,12 @@ def write_text(stream, text: str) -> None: stream.flush() -def run(args, environ=None, stdout=None, stderr=None) -> int: +def run(args, environ=None, stdout=None, stderr=None, pace=None) -> int: """``syberlabs githits ...``: 0 a result, 2 refused (an excluded target or a blocked result), 1 an error; ``doctor``: 0 ready, 1 not (the status says why). A lookup's output is written (``write_text``) before its record, - so an output that fails is recorded ``error`` and exits 1 (``Client.call``, ``deliver``).""" + so an output that fails is recorded ``error`` and exits 1 (``Client.call``, ``deliver``). ``pace``, given by the + host's launcher entry, is called with each lookup's request once the client and request are built and before it + is sent; it returns the milliseconds it held the call (recorded as ``paced_ms``) or None (not paced).""" stdout, stderr = stdout or sys.stdout, stderr or sys.stderr op = args.githits_command options = {name: getattr(args, name) for name in _OPTIONS[op]} @@ -1338,7 +1381,8 @@ def deliver(result: Result) -> None: request = list_request(args.target, **options) else: request = grep_request(args.pattern, args.targets, **options) - result = client.call(request, deliver=deliver) + paced_ms = pace(request) if pace is not None else None + result = client.call(request, deliver=deliver, paced_ms=paced_ms) except GitHitsError as exc: try: write_text(stderr, f"error: {exc}\n") diff --git a/syberlabs/githits_cli_entry.py b/syberlabs/githits_cli_entry.py index aa32c10a..5ff3e34f 100644 --- a/syberlabs/githits_cli_entry.py +++ b/syberlabs/githits_cli_entry.py @@ -13,6 +13,14 @@ exists, so they write no record; ``doctor`` is the operator's, through ``syberlabs githits doctor``. ``example`` has no ``--license-mode`` here and always sends ``license_mode: strict``. +An ``example`` to GitHits is paced here, on the host, before it is sent (``githits_pace.wait``): at most one per +``githits_pace.EXAMPLE_INTERVAL`` (40 s) across every launcher process on this host, both arms included, under a +cross-process lock and a time file in the host's pacing directory, outside any worktree. GitHits limits its example +endpoint per organization. The time the call was held is recorded on its client record as ``paced_ms``, which the +experiment report leaves out of the arm's agent time. Search, read, list and grep are not paced, and the client +itself never waits or retries. A pacing directory that cannot be used is an error before anything is sent. A +loopback stand-in (the tests' fake) is not paced unless ``SYBERLABS_GITHITS_PACE_SECONDS`` asks (``interval_for``). + A ``SIGTERM`` (how a Bash tool ends a command past its timeout) becomes ``SystemExit`` once a lookup is parsed, so a launcher ended after its request left still writes that call's record (``githits.Client.call`` records any interruption) before it exits. A ``SIGKILL`` cannot be caught: that call stays unrecorded, and the host flags @@ -41,9 +49,9 @@ def main(argv: list[str] | None = None) -> int: if sys.path[:1] != [root]: sys.path.insert(0, root) import syberlabs - from syberlabs import cli, githits + from syberlabs import cli, githits, githits_pace - if any(Path(module.__file__).resolve().parent != HERE for module in (syberlabs, cli, githits)): + if any(Path(module.__file__).resolve().parent != HERE for module in (syberlabs, cli, githits, githits_pace)): sys.stderr.write("error: githits_launcher: syberlabs was not imported from the host checkout\n") return 1 cli.survive_unicode() # before any output: a cp1252 pipe replaces what it cannot encode instead of raising @@ -61,7 +69,28 @@ def main(argv: list[str] | None = None) -> int: signal.signal(signal.SIGTERM, _terminated) except (AttributeError, ValueError, OSError): # no SIGTERM here, or not the main thread: run as before pass - return githits.run(args) + return githits.run(args, pace=_pace) + + +def _pace(request) -> int | None: + """``githits.run``'s pace hook: hold an ``example`` until this host's example slot is free and return the + milliseconds it was held; every other operation is sent unpaced (None). An example to GitHits is always paced + at ``githits_pace.EXAMPLE_INTERVAL``; one to a loopback stand-in only when the tests ask (``interval_for``).""" + import os + import urllib.parse + + from syberlabs import githits, githits_pace + + if request.op not in githits_pace.PACED_OPS: + return None + base = os.environ.get(githits.URL_ENV) or githits.DEFAULT_BASE_URL + try: + interval = githits_pace.interval_for(githits._loopback(urllib.parse.urlsplit(base).hostname or "")) + if interval is None: + return None + return githits_pace.wait(githits_pace.directory(), interval=interval) + except githits_pace.PaceError as exc: + raise githits.GitHitsError("pace_failed", str(exc)) from None if __name__ == "__main__": diff --git a/syberlabs/githits_pace.py b/syberlabs/githits_pace.py new file mode 100644 index 00000000..a93f1a10 --- /dev/null +++ b/syberlabs/githits_pace.py @@ -0,0 +1,157 @@ +"""Host-side pacing of GitHits ``example`` calls: at most one per ``EXAMPLE_INTERVAL`` seconds on this host. + +GitHits limits its example endpoint per organization to one or two requests a minute. The pilot runs both arms, and +any number of Claude Code calls, under one organization's token, so the limit is shared by every process that sends +an example. The host's launcher entry (``githits_cli_entry.py``) calls ``wait`` before an ``example`` is sent: + + paced_ms = wait(directory) # blocks until this host's last example is EXAMPLE_INTERVAL seconds old + +``wait`` takes a cross-process lock on ``/example.lock`` (``flock`` where ``fcntl`` exists, +``msvcrt.locking`` on Windows, as ``journal.Journal`` does), reads the time of the last example from +``/example.last``, sleeps until ``EXAMPLE_INTERVAL`` seconds have passed since it, writes the new time and +releases the lock. The lock is held while it sleeps, so callers queue and each slot is at least the interval after the +one before, across processes, threads, arms and homes. It returns the milliseconds the caller was held, lock wait +included, which the launcher records on the call's client record as ``paced_ms``; ``syberlabs experiment report`` +leaves that time out of an arm's agent time. + +The directory is the host's, never a worktree's: ``SYBERLABS_GITHITS_PACE_DIR`` when the operator sets it (an absolute +path), else ``syberlabs-githits-pace`` in the system temporary directory, shared by every process of this user. Only +``example`` is paced; search, read, list and grep are not. This is pacing before a request leaves, not a retry: the +client (``githits.py``) still sends each request once and never waits or retries on its own. + +Which calls are paced (``interval_for``): every example to a GitHits base URL that is not a loopback host, at +``EXAMPLE_INTERVAL``, whatever the environment says. A loopback base is a stand-in (the client accepts ``http://`` +only there; the tests' fake GitHits), whose calls never reach GitHits' organization limit, so it is not paced unless +``SYBERLABS_GITHITS_PACE_SECONDS`` sets an interval for it (the tests of this module do). That variable is ignored +for any other base. + +Limits: the time file is plain text a process of the same user could rewrite, so it paces cooperating processes and +is no security boundary (a stale or future time is bounded: no call waits more than one interval for it). With neither +``fcntl`` nor ``msvcrt`` the lock is skipped, as the journal's is. Calls queued behind several others wait one interval +each, so a Bash tool timeout shorter than the queue ends the call before it is sent (the record then never exists and +the host flags the launcher call unrecorded). +""" + +from __future__ import annotations + +import errno +import os +import tempfile +import time +from contextlib import contextmanager +from pathlib import Path + +try: + import fcntl +except ImportError: # pragma: no cover - Windows + fcntl = None +try: + import msvcrt +except ImportError: # pragma: no cover - not Windows + msvcrt = None + +EXAMPLE_INTERVAL = 40.0 # seconds between two example calls on this host (GitHits: 1-2 per minute) +PACED_OPS = ("example",) # the only operation paced +PACE_DIR_ENV = "SYBERLABS_GITHITS_PACE_DIR" +PACE_SECONDS_ENV = "SYBERLABS_GITHITS_PACE_SECONDS" # an interval for a loopback stand-in only (tests); else ignored +DEFAULT_DIR_NAME = "syberlabs-githits-pace" +LOCK_FILE, STAMP_FILE = "example.lock", "example.last" +_BUSY = frozenset({errno.EACCES, getattr(errno, "EDEADLOCK", errno.EDEADLK)}) + + +class PaceError(Exception): + """The pacing directory cannot be used: the call is not sent.""" + + +def directory(environ=None) -> Path: + """The host's pacing directory: ``SYBERLABS_GITHITS_PACE_DIR`` (absolute), else the temporary-directory default.""" + environ = os.environ if environ is None else environ + named = (environ.get(PACE_DIR_ENV) or "").strip() + if named: + path = Path(named) + if not path.is_absolute(): + raise PaceError(f"{PACE_DIR_ENV} must be an absolute path outside any worktree") + return path + return Path(tempfile.gettempdir()) / DEFAULT_DIR_NAME + + +def interval_for(loopback: bool, environ=None) -> float | None: + """The pacing interval for a call to a base URL that is (``loopback``) or is not a loopback host: always + ``EXAMPLE_INTERVAL`` for GitHits itself; for a loopback stand-in, ``SYBERLABS_GITHITS_PACE_SECONDS`` when set, + else None (not paced).""" + if not loopback: + return EXAMPLE_INTERVAL + environ = os.environ if environ is None else environ + text = (environ.get(PACE_SECONDS_ENV) or "").strip() + if not text: + return None + try: + value = float(text) + except ValueError: + raise PaceError(f"{PACE_SECONDS_ENV} is a non-negative number of seconds") from None + if not (value >= 0 and value != float("inf")): + raise PaceError(f"{PACE_SECONDS_ENV} is a non-negative number of seconds") + return value + + +@contextmanager +def _locked(path: Path): + with open(path, "a+b") as handle: + if fcntl is not None: + fcntl.flock(handle.fileno(), fcntl.LOCK_EX) + try: + yield + finally: + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + return + if msvcrt is None: # pragma: no cover - neither lock exists: unlocked, as the journal is + yield + return + fd = handle.fileno() + while True: # LK_LOCK gives up after about ten seconds; a slot can take longer, so poll LK_NBLCK + os.lseek(fd, 0, os.SEEK_SET) + try: + msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) + break + except OSError as exc: + if exc.errno not in _BUSY: + raise + time.sleep(0.05) + try: + yield + finally: + os.lseek(fd, 0, os.SEEK_SET) + msvcrt.locking(fd, msvcrt.LK_UNLCK, 1) + + +def _last(path: Path) -> float | None: + try: + value = float(path.read_text(encoding="ascii").strip()) + except (OSError, ValueError, UnicodeDecodeError): + return None + return value if value == value and abs(value) != float("inf") else None + + +def wait(folder=None, *, interval: float = EXAMPLE_INTERVAL, clock=time.time, sleep=time.sleep) -> int: + """Hold the caller until this host's last example slot is ``interval`` seconds old, take the next slot, and + return how long the caller was held in milliseconds (lock wait included). ``clock`` and ``sleep`` are injectable + for tests; the stamp is wall-clock seconds so that separate processes compare it.""" + if not (isinstance(interval, (int, float)) and interval >= 0): + raise PaceError("the pacing interval is a non-negative number of seconds") + folder = Path(folder) if folder is not None else directory() + started = clock() + try: + folder.mkdir(parents=True, exist_ok=True) + with _locked(folder / LOCK_FILE): + last = _last(folder / STAMP_FILE) + now = clock() + if last is not None and now < last + interval: + sleep(min(last + interval - now, interval)) # a stamp in the future never holds a call longer + now = clock() + stamp = folder / STAMP_FILE + temporary = folder / f"{STAMP_FILE}.{os.getpid()}.tmp" + temporary.write_text(repr(now), encoding="ascii") + os.replace(temporary, stamp) + except OSError as exc: + raise PaceError(f"cannot pace the example call in {folder}: {exc.strerror or type(exc).__name__}") from None + return max(0, int(round((clock() - started) * 1000))) diff --git a/tests/test_githits.py b/tests/test_githits.py index 462acbf6..1d29566e 100644 --- a/tests/test_githits.py +++ b/tests/test_githits.py @@ -563,7 +563,10 @@ def test_example_licenses_come_from_source_links_license_lines_and_references(se "[Source 2](https://gitlab.com/a/b/-/blob/1/c.py) (GPL-3.0-only)"}), ["MIT", "GPL-3.0-only"]) self.assertEqual(read({"result": "[Source 1](https://github.com/a/b/blob/1/c.py)"}), [None]) - self.assertEqual(read({"result": "see [the docs](https://docs.python.org/3/) only"}), [None]) + # No code and no source: nothing to license (pilot-readiness; GitHits' documented "no_example" shape). The + # same docs link beside code is a source-less code example: unknown, dropped. + self.assertEqual(read({"result": "see [the docs](https://docs.python.org/3/) only"}), []) + self.assertEqual(read({"result": "```python\nx = 1\n```\nsee [the docs](https://docs.python.org/3/)"}), [None]) self.assertEqual(read({"result": "code\n\n**License:** BSD-3-Clause"}), ["BSD-3-Clause"]) self.assertEqual(read({"result": "", "references": [{"license_spdx_id": "MPL-2.0"}, {"url": "u"}]}), ["MPL-2.0", None]) @@ -856,6 +859,81 @@ def test_any_restores_pass_through(self): self.assertEqual(self.client().search("q", ["pypi:requests"]).results_dropped_license, 1) +def documented_examples() -> dict: + """GitHits' three documented example responses (``complete``, ``degraded``, ``no_example``), as the pinned + OpenAPI fixture holds them for ``POST /v1/examples/search`` 200.""" + spec = json.loads(FIXTURE.read_text(encoding="utf-8")) + found = spec["paths"]["/v1/examples/search"]["post"]["responses"]["200"]["content"]["application/json"]["examples"] + return {name: entry["value"] for name, entry in found.items()} + + +class DocumentedExamples(FakeCase): + """pilot-readiness, item 1. A lead review on 2026-10-26 found the license filter withheld all three of GitHits' + documented example responses (``results_dropped_license=1`` each): the ``complete`` example cites its source as + ``[Source 1](https://example.org/source) (MIT)`` under ``### References``, a host the filter did not read as a + source, and ``degraded`` and ``no_example`` carry no code and no source at all. GitHits confirmed it returns no + structured license fields, so the Markdown is all there is. Permissive now passes; copyleft and unknown are still + dropped.""" + + def test_the_three_documented_responses_pass_through_the_client(self): + examples = documented_examples() + self.assertEqual(sorted(examples), ["complete", "degraded", "no_example"]) + spec = load_spec() + for name, body in examples.items(): + with self.subTest(example=name): + self.assertEqual(problems(spec, response_schema(spec, "example"), body), []) + self.fake.serve("/v1/examples/search", body) + result = self.client().example("read a JSON file in Rust") + self.assertEqual((result.status, result.results_dropped_license, result.results_dropped), + (githits.OK, 0, 0)) + self.assertEqual(result.data, body, "passed whole, nothing rewritten") + self.assertEqual(githits.example_licenses(examples["complete"]), ["MIT"], + "the References link is the source; the GitHits app link is not") + self.assertEqual(githits.example_licenses(examples["degraded"]), []) + self.assertEqual(githits.example_licenses(examples["no_example"]), []) + self.assertEqual([r["results_dropped_license"] for r in self.records().records], [0, 0, 0]) + + def test_the_documented_shape_with_copyleft_or_unknown_licenses_is_still_dropped(self): + complete = documented_examples()["complete"] + for license_text in ("GPL-3.0-only", "AGPL-3.0-or-later", "MPL-2.0", "NOASSERTION", "MIT AND GPL-2.0-only", + "see repository"): + body = {**complete, "result": complete["result"].replace("(MIT)", f"({license_text})")} + with self.subTest(license=license_text): + self.fake.serve("/v1/examples/search", body) + result = self.client().example("q") + self.assertEqual((result.status, result.data, result.results_dropped_license), (githits.OK, None, 1)) + unknown = {**complete, "result": complete["result"].replace(" (MIT)", "")} + self.fake.serve("/v1/examples/search", unknown) + self.assertEqual(self.client().example("q").results_dropped_license, 1, "a cited source with no license") + no_source = {**complete, "result": complete["result"].split("### References")[0]} + self.assertIn("```rust", no_source["result"]) + self.fake.serve("/v1/examples/search", no_source) + self.assertEqual(self.client().example("q").results_dropped_license, 1, "code with no source at all") + second = {**complete, "result": complete["result"].replace( + "(MIT)\n", "(MIT)\n- [Source 2](https://example.org/other) (GPL-3.0-only)\n")} + self.assertIn("Source 2", second["result"]) + self.fake.serve("/v1/examples/search", second) + self.assertEqual(self.client().example("q").results_dropped_license, 1, "one copyleft source drops it whole") + permissive = {**complete, "result": complete["result"].replace("(MIT)", "(Apache-2.0 OR MIT)")} + self.fake.serve("/v1/examples/search", permissive) + self.assertEqual(self.client().example("q").results_dropped_license, 0) + + def test_only_links_under_a_sources_heading_or_on_a_code_host_are_sources_and_never_githits_own(self): + read = githits.example_licenses + app = "[Open in GitHits Web App](https://app.githits.com/solutions/12345678-1234-4234-8234-123456789abc)" + self.assertEqual(read({"result": "```py\nx\n```\n### References\n" + app}), [None], + "the app link is not a source, so the code has none") + self.assertEqual(read({"result": "```py\nx\n```\n## Sources\n- [a](https://docs.example.org/a) (ISC)\n" + "\n## Notes\n[b](https://docs.example.org/b)"}), ["ISC"], + "the section ends at the next heading") + self.assertEqual(read({"result": "```py\nx\n```\n**References**\n- (BSD-3-Clause)\n" + "\n---\n\n[c](https://example.org/c)"}), ["BSD-3-Clause"], + "a bold heading, an autolink, and a thematic break ends the section") + self.assertEqual(read({"result": "```py\nx\n```\n[docs](https://docs.example.org/) (GPL-3.0-only)\n" + "[Source 1](https://github.com/a/b/blob/1/c.py) (MIT)"}), ["MIT"], + "outside a sources section a non-code-host link is not a source, as before") + + class Transport(FakeCase): def test_lean_defaults_are_sent(self): client = self.client() @@ -1252,7 +1330,7 @@ def test_the_entry_runs_the_clis_unicode_helper_before_any_output(self): with self.subTest(argv=argv): order = [] with mock.patch.object(cli, "survive_unicode", side_effect=lambda: order.append("survive")), \ - mock.patch.object(githits, "run", side_effect=lambda args: order.append("run") or 0), \ + mock.patch.object(githits, "run", side_effect=lambda args, **_: order.append("run") or 0), \ mock.patch.object(sys, "stdout", io.StringIO()), mock.patch.object(sys, "stderr", io.StringIO()): self.assertEqual(githits_cli_entry.main(argv), expected) printed = sys.stdout.getvalue() + sys.stderr.getvalue() diff --git a/tests/test_githits_engineering_smoke.py b/tests/test_githits_engineering_smoke.py index a5968b10..77f5d7d1 100644 --- a/tests/test_githits_engineering_smoke.py +++ b/tests/test_githits_engineering_smoke.py @@ -5,6 +5,7 @@ import os from pathlib import Path import subprocess +import sys import tempfile import unittest from unittest import mock @@ -50,10 +51,42 @@ def test_preparation_never_calls_a_provider_and_a_wrong_pin_is_refused(self): run.assert_not_called() self.assertFalse((Path(folder) / "bad").exists()) + def test_the_cli_pin_is_exactly_2_1_312_recorded_and_any_other_version_is_refused(self): + """pilot-readiness, item 3: the host and the pilot VM run Claude Code 2.1.312, so the frozen recipe pins it, + exactly, and records the pin in the plan and the manifest.""" + self.assertEqual(smoke.CLI_VERSION, "2.1.312") + plan = json.loads((smoke.PACK / "plan.json").read_text(encoding="utf-8")) + self.assertEqual(plan["_configuration"]["cli"], smoke.CLI_VERSION) + with tempfile.TemporaryDirectory() as folder: + smoke.prepare(Path(folder) / "ready") + manifest = json.loads((Path(folder) / "ready" / "manifest.json").read_text(encoding="utf-8")) + self.assertEqual(manifest["cli_version"], smoke.CLI_VERSION) + pin = smoke.pilot.host_checkout()["commit"] + script = Path(folder) / "claude_stand_in.py" + for printed, refused in (("2.1.311 (Claude Code)", True), ("2.1.3120 (Claude Code)", True), + ("2.1.312 (Claude Code) beta", True), ("banner\n2.1.312 (Claude Code)", True), + ("2.1.312 (Claude Code)", False)): + with self.subTest(version=printed): + script.write_text(f"print({printed!r})\n", encoding="utf-8") + work = Path(folder) / f"run-{len(list(Path(folder).iterdir()))}" + with mock.patch.dict(os.environ, {smoke.pilot.claude_code.BIN_ENV: f"{sys.executable} {script}"}), \ + mock.patch.object(smoke.pilot, "host_checkout", return_value={"commit": pin, "dirty": False}), \ + mock.patch.object(smoke.pilot, "main", return_value=0) as run, \ + contextlib.redirect_stdout(io.StringIO()) as out: + code = smoke.main(["--work", str(work), "--factory-commit", pin, "--run-approved"]) + if refused: + self.assertEqual(code, 2, out.getvalue()) + self.assertIn("requires Claude Code 2.1.312 exactly", out.getvalue()) + run.assert_not_called() + else: + self.assertEqual(code, 0, out.getvalue()) + self.assertIn("Claude Code 2.1.312 (the exact pin)", out.getvalue()) + run.assert_called_once() + class OfflineRecipe(DryRunCase): def test_one_recipe_runs_the_real_three_item_fixture_with_stand_ins(self): - command = self.stand_in("engineering", version="2.1.311 (Claude Code)", lookups=LOOKUPS) + command = self.stand_in("engineering", version=smoke.CLI_VERSION + " (Claude Code)", lookups=LOOKUPS) path = Path(command[-1]) text = path.read_text(encoding="utf-8") # Select visible synthetic task implementations; the canary changes no file. diff --git a/tests/test_githits_pace.py b/tests/test_githits_pace.py new file mode 100644 index 00000000..489721b8 --- /dev/null +++ b/tests/test_githits_pace.py @@ -0,0 +1,309 @@ +"""pilot-readiness, item 2: host-side pacing of GitHits ``example`` calls, its record, and its accounting. + +GitHits limits its example endpoint per organization to one or two requests a minute. The host's launcher entry +(``githits_cli_entry._pace``) holds each example until this host's last one is ``githits_pace.EXAMPLE_INTERVAL`` +(40 s) old, across processes, under a lock and a time file outside any worktree; the client records how long it was +held (``paced_ms``) and never waits or retries itself; the experiment report leaves that time out of an arm's agent +time. Every test injects the clock or a short interval, and every call goes to a loopback fake GitHits. +""" + +import dataclasses +import json +import os +import subprocess +import sys +import tempfile +import textwrap +import time +import unittest +from pathlib import Path +from unittest import mock + +from syberlabs import experiment, githits, githits_cli_entry, githits_pace, metrics +from syberlabs.claude_code import record_lookup +from tests.test_githits import ENTRY, FakeCase, run_entry +from tests.test_metrics import HOUR, T0, Journal, check + +ROOT = Path(__file__).resolve().parents[1] +PACE_ENV = (githits_pace.PACE_DIR_ENV, githits_pace.PACE_SECONDS_ENV) + + +class FakeClock: + """A wall clock that only ``sleep`` moves.""" + + def __init__(self, now: float = 1_800_000_000.0): + self.now, self.sleeps = now, [] + + def __call__(self) -> float: + return self.now + + def sleep(self, seconds: float) -> None: + self.sleeps.append(seconds) + self.now += seconds + + +class Wait(unittest.TestCase): + def setUp(self): + self.folder = tempfile.TemporaryDirectory() + self.addCleanup(self.folder.cleanup) + self.dir = Path(self.folder.name) / "pace" + self.clock = FakeClock() + + def wait(self, **kwargs): + return githits_pace.wait(self.dir, clock=self.clock, sleep=self.clock.sleep, **kwargs) + + def test_each_slot_is_the_interval_after_the_last_and_the_wait_is_returned(self): + self.assertEqual(self.wait(), 0, "the first example on this host waits for nothing") + self.assertEqual(self.wait(), 40_000, "the next one at once is held the whole interval") + self.assertEqual(self.clock.sleeps, [40.0]) + self.clock.now += 15 + self.assertEqual(self.wait(), 25_000, "15 s later: held for the remaining 25 s") + self.clock.now += 41 + self.assertEqual(self.wait(), 0, "an interval later: not held") + self.assertEqual(float((self.dir / githits_pace.STAMP_FILE).read_text(encoding="ascii")), self.clock.now) + self.assertEqual(githits_pace.EXAMPLE_INTERVAL, 40.0) + + def test_a_future_or_unreadable_stamp_never_holds_a_call_past_one_interval(self): + self.dir.mkdir() + (self.dir / githits_pace.STAMP_FILE).write_text(repr(self.clock.now + 10_000), encoding="ascii") + self.assertEqual(self.wait(), 40_000) + for text in ("garbage", "nan", "inf", ""): + with self.subTest(stamp=text): + (self.dir / githits_pace.STAMP_FILE).write_text(text, encoding="ascii") + self.assertEqual(self.wait(), 0) + with self.assertRaises(githits_pace.PaceError): + self.wait(interval=-1) + + def test_the_directory_is_the_hosts_and_an_unusable_one_is_an_error(self): + self.assertEqual(githits_pace.directory({}), Path(tempfile.gettempdir()) / githits_pace.DEFAULT_DIR_NAME) + self.assertEqual(githits_pace.directory({githits_pace.PACE_DIR_ENV: str(self.dir)}), self.dir) + with self.assertRaises(githits_pace.PaceError): + githits_pace.directory({githits_pace.PACE_DIR_ENV: "relative/pace"}) + blocker = Path(self.folder.name) / "a-file" + blocker.write_text("x", encoding="utf-8") + with self.assertRaises(githits_pace.PaceError): + githits_pace.wait(blocker / "pace", clock=self.clock, sleep=self.clock.sleep) + + def test_githits_itself_is_always_paced_at_40_s_and_a_loopback_stand_in_only_when_asked(self): + for environ in ({}, {githits_pace.PACE_SECONDS_ENV: "0"}, {githits_pace.PACE_SECONDS_ENV: "0.5"}): + with self.subTest(environ=environ): + self.assertEqual(githits_pace.interval_for(False, environ), 40.0) + self.assertIsNone(githits_pace.interval_for(True, {})) + self.assertEqual(githits_pace.interval_for(True, {githits_pace.PACE_SECONDS_ENV: "2.5"}), 2.5) + for bad in ("-1", "soon", "inf", "nan"): + with self.subTest(bad=bad), self.assertRaises(githits_pace.PaceError): + githits_pace.interval_for(True, {githits_pace.PACE_SECONDS_ENV: bad}) + + +WAITER = textwrap.dedent(""" + import sys, time + sys.path.insert(0, sys.argv[1]) + from pathlib import Path + from syberlabs import githits_pace + held = githits_pace.wait(Path(sys.argv[2]), interval=float(sys.argv[3])) + slot = (Path(sys.argv[2]) / githits_pace.STAMP_FILE).read_text(encoding="ascii") + print(held, slot) +""") + + +class CrossProcess(FakeCase): + INTERVAL = 2.0 + + def setUp(self): + super().setUp() + self.pace = tempfile.TemporaryDirectory() + self.addCleanup(self.pace.cleanup) + + def test_two_concurrent_processes_take_slots_at_least_the_interval_apart(self): + script = Path(self.pace.name) / "waiter.py" + script.write_text(WAITER, encoding="utf-8") + folder = Path(self.pace.name) / "pace" + argv = [sys.executable, "-I", str(script), str(ROOT), str(folder), str(self.INTERVAL)] + processes = [subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE) for _ in range(2)] + outputs = [process.communicate(timeout=60) for process in processes] + for process, (_, err) in zip(processes, outputs): + self.assertEqual(process.returncode, 0, err.decode("utf-8", "replace")) + found = sorted((float(out.split()[1]), int(out.split()[0])) for out, _ in outputs) + (first, held_first), (second, held_second) = found + self.assertGreaterEqual(second - first, self.INTERVAL, "the second slot is a whole interval after the first") + self.assertLess(held_first, 1000, "the first was not held") + self.assertGreaterEqual(held_second, 1000, "the second was held, its lock wait included") + + def test_two_concurrent_launcher_examples_are_spaced_recorded_and_other_ops_unpaced(self): + arrivals = [] + + def example(request): + arrivals.append(time.time()) + body = {"result": "No suitable example was found.\n", "search_status": "complete", + "solution_id": "12345678-1234-4234-8234-123456789abc"} + return 200, {"Content-Type": "application/json", "X-Request-ID": f"req-ex{len(arrivals)}"}, \ + json.dumps(body).encode("utf-8") + + self.fake.script["/v1/examples/search"] = example + env = {k: v for k, v in os.environ.items() if not k.upper().startswith(("PYTHON", "GITHITS_", "SYBERLABS_"))} + env.update(self.env) + env.update({githits_pace.PACE_DIR_ENV: str(Path(self.pace.name) / "pace"), + githits_pace.PACE_SECONDS_ENV: str(self.INTERVAL)}) + argv = [sys.executable, "-I", str(ENTRY), "example", "retry with backoff"] + processes = [subprocess.Popen(argv, cwd=self.pace.name, env=env, stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) for _ in range(2)] + for process in processes: + _, err = process.communicate(timeout=120) + self.assertEqual(process.returncode, 0, err.decode("utf-8", "replace")) + self.assertEqual(len(arrivals), 2) + # Each slot is stamped just before its request leaves, so arrivals sit a whole interval apart less the first + # request's own few milliseconds of transit. + self.assertGreaterEqual(abs(arrivals[1] - arrivals[0]), self.INTERVAL - 0.25) + records = self.records().records + self.assertEqual([r["op"] for r in records], ["example", "example"]) + held = sorted(r["paced_ms"] for r in records) + self.assertLess(held[0], 1000) + self.assertGreaterEqual(held[1], 1000) + self.assertTrue(all(r["latency_ms"] < r["paced_ms"] or r["paced_ms"] < 1000 for r in records), + "latency_ms is the request's own time, never the pacing") + # The other four operations, at once and with a long interval in force, are never held or stamped. + stamp = (Path(self.pace.name) / "pace" / githits_pace.STAMP_FILE).read_text(encoding="ascii") + with mock.patch.dict(os.environ, {githits_pace.PACE_DIR_ENV: env[githits_pace.PACE_DIR_ENV], + githits_pace.PACE_SECONDS_ENV: "600"}): + started = time.monotonic() + for args in (["search", "q", "--target", "pypi:requests"], ["read", "pypi:requests"], + ["list", "pypi:requests"], ["grep", "Retry", "--target", "pypi:requests", "--literal"]): + with self.subTest(op=args[0]): + code, _, err = run_entry(self.env, *args) + self.assertEqual((code, err), (0, "")) + self.assertLess(time.monotonic() - started, 60) + others = self.records().records[2:] + self.assertEqual([r["op"] for r in others], ["search", "read", "list", "grep"]) + self.assertTrue(all("paced_ms" not in r for r in others)) + self.assertEqual((Path(self.pace.name) / "pace" / githits_pace.STAMP_FILE).read_text(encoding="ascii"), stamp) + + +class LauncherHook(FakeCase): + def setUp(self): + super().setUp() + for name in PACE_ENV: + patcher = mock.patch.dict(os.environ) + patcher.start() + self.addCleanup(patcher.stop) + os.environ.pop(name, None) + + def test_an_example_to_githits_is_paced_at_40_s_whatever_the_environment_says(self): + request = githits.example_request("q") + with mock.patch.dict(os.environ, {githits.URL_ENV: "https://api.githits.dev/v1", + githits_pace.PACE_SECONDS_ENV: "0"}), \ + mock.patch.object(githits_pace, "wait", return_value=1234) as wait: + self.assertEqual(githits_cli_entry._pace(request), 1234) + self.assertEqual(wait.call_args.kwargs, {"interval": 40.0}) + os.environ.pop(githits.URL_ENV) + self.assertEqual(githits_cli_entry._pace(request), 1234, "the default base is GitHits itself") + for other in (githits.search_request("q", ["pypi:requests"]), githits.read_request("pypi:requests"), + githits.list_request("pypi:requests"), githits.grep_request("x", ["pypi:requests"])): + self.assertIsNone(githits_cli_entry._pace(other)) + self.assertEqual(wait.call_count, 2) + + def test_a_loopback_stand_in_is_not_paced_unless_asked_and_a_bad_setting_sends_nothing(self): + code, _, err = run_entry(self.env, "example", "retry") + self.assertEqual((code, err), (0, "")) + [record] = self.records().records + self.assertNotIn("paced_ms", record, "unpaced: the record is exactly the fields it always had") + with mock.patch.dict(os.environ, {githits_pace.PACE_SECONDS_ENV: "soon"}): + code, _, err = run_entry(self.env, "example", "retry") + self.assertEqual(code, 1) + self.assertIn("pace_failed", err) + self.assertEqual(len(self.fake.requests), 1, "nothing was sent") + + +class Records(FakeCase): + def test_paced_ms_is_recorded_under_the_mac_and_read_back(self): + self.client().call(githits.example_request("q"), paced_ms=39_250) + self.client().search("q", ["pypi:requests"]) + records = self.records() + self.assertEqual(records.ignored, 0) + self.assertEqual([r.get("paced_ms") for r in records.records], [39_250, None]) + self.assertEqual(set(records.records[0]), set(githits.RECORD_FIELDS) | {"paced_ms"}) + self.assertEqual(set(records.records[1]), set(githits.RECORD_FIELDS)) + with self.assertRaises(githits.GitHitsError) as caught: + self.client().call(githits.example_request("q"), paced_ms=-1) + self.assertEqual(caught.exception.code, "bad_pacing") + + def test_a_tampered_or_malformed_paced_ms_is_ignored(self): + self.client().call(githits.example_request("q"), paced_ms=40_000) + path = Path(self.folder.name) / githits.RECORD_FILE + first = json.loads(path.read_text(encoding="utf-8").splitlines()[0]) + lines = [{**first, "paced_ms": 0}, {k: v for k, v in first.items() if k != "paced_ms"}] + for bad in (None, -5, "40000", 1.5): + forged = {**first, "paced_ms": bad} + forged["mac"] = githits.record_mac({k: v for k, v in forged.items() if k != "mac"}, self.key) + lines.append(forged) + with open(path, "a", encoding="utf-8", newline="\n") as handle: + for line in lines: + handle.write(json.dumps(line) + "\n") + read = self.records() + self.assertEqual((len(read.records), read.ignored), (1, 6)) + + def test_the_lookup_fact_carries_paced_ms_only_when_the_record_does(self): + self.client().call(githits.example_request("q"), paced_ms=12_000) + self.client().search("q", ["pypi:requests"]) + paced, unpaced = (record_lookup(r, "pass") for r in self.records().records) + self.assertEqual(paced["paced_ms"], 12_000) + self.assertNotIn("paced_ms", unpaced) + + +NAME = "githits-pilot" + + +def paced_thread(case: str, paced=(), after=(), hours: float = 1.0, arm: str = "on"): + """A labelled, accepted thread whose candidate passes ``hours`` after its search, with GitHits lookup facts + holding ``paced`` milliseconds before the evaluation and ``after`` milliseconds after it.""" + j = Journal(case) + j.add("case_created", {"actor": "agent@example.test", "contract": ["repo-change", 4], + "inputs": {"objective": "Work item X"}}, at=T0) + j.fact("base", {"commit": "a" * 40}) + j.fact(experiment.KEY, {"name": NAME, "item": "X", "arm": arm, "tags": [], "by": "owner@example.test", + "at": "2026-10-24T00:00:00Z"}, version=NAME) + search = j.add("search_started", {"id": "s1", "actor": "syberlabs.host"}, at=T0) + for held in paced: + j.fact("lookup", {"tool": "githits.example", "status": "ok", "guard": "pass", "ok": True, "paced_ms": held}) + j.fact("lookup", {"tool": "WebSearch", "guard": "pass", "ok": True, "paced_ms": 3_600_000}) # not GitHits' pacing + signal = {"model_usage": {"input_tokens": 10, "output_tokens": 2, "model": "claude-sonnet-5-5"}, + "harness": {"name": "claude-code", "model": "claude-sonnet-5-5"}} + if arm == "on": + signal["githits"] = {"arm": "on", "records": len(paced), "launcher_calls": len(paced)} + j.add("candidate_registered", {"id": "c1", "operator": "patch", "actor": "syberlabs.host", "signal": signal}, + at=T0 + 1) + evaluated = j.add("candidate_evaluated", {"candidate": "c1", "actor": "syberlabs.host", + "checks": [check("tests", "passed")]}, at=T0 + int(hours * HOUR)) + for held in after: + j.fact("lookup", {"tool": "githits.example", "status": "ok", "guard": "pass", "ok": True, "paced_ms": held}) + j.add("proposed", {"id": "p1", "action": "accept_change", "actor": "runner@example.test", + "args": {"candidate": "c1", "commit": "b" * 40}}, at=T0 + int((hours + 1) * HOUR)) + j.add("decision", {"proposal_id": "p1", "status": "allowed", "reason": "allowed"}, at=T0 + int((hours + 1) * HOUR)) + j.add("effect_succeeded", {"proposal_id": "p1", "action": "accept_change", "output": {}}, + at=T0 + int((hours + 1) * HOUR) + 1) + return j.events, search, evaluated + + +class Report(unittest.TestCase): + def test_paced_time_is_left_out_of_time_to_pass_and_agent_time(self): + events, search, evaluated = paced_thread("x-on", paced=(900_000, 900_000), after=(40_000,)) + [run] = experiment.runs(events, NAME) + self.assertEqual(run.time_to_pass.value, 0.5, "one hour less the half hour the host held two examples") + self.assertEqual(run.time_to_pass.basis, (search, evaluated)) + self.assertIn("less 1800 s the host held GitHits calls", run.time_to_pass.note) + self.assertEqual(run.paced, 1840.0, "the whole thread's pacing, after the evaluation included") + [plain] = experiment.runs(paced_thread("y-on")[0], NAME) + self.assertEqual((plain.time_to_pass.value, plain.paced), (1.0, 0.0), "no pacing, nothing left out") + self.assertNotIn("less", plain.time_to_pass.note) + merged = dataclasses.replace(run, work=dataclasses.replace( + run.work, agent_time=metrics.Metric(2.0, (), "case_created to merge"))) + summary = experiment._per_arm([merged]) + self.assertEqual((summary["agent_time_mean"], summary["paced_seconds"]), (round(2.0 - 1840 / 3600, 6), 1840.0)) + self.assertEqual(experiment._per_arm([plain])["paced_seconds"], 0) + found = experiment.report(events + paced_thread("x-off", arm="off")[0], NAME) + self.assertEqual((found["per_arm"]["on"]["paced_seconds"], found["per_arm"]["off"]["paced_seconds"]), + (1840.0, 0), found["rows"]) + self.assertIn("s of GitHits pacing left out of its time", experiment.table(found)) + + +if __name__ == "__main__": + unittest.main()