diff --git a/docs/CLAUDE_CODE.md b/docs/CLAUDE_CODE.md index 0b82213..551d937 100644 --- a/docs/CLAUDE_CODE.md +++ b/docs/CLAUDE_CODE.md @@ -116,7 +116,7 @@ What can read the key: the Claude Code process, the hook, and every other proces | Rule | Blocks | |---|---| -| `secret` | A credential shape: the pattern set `syberlabs.metrics` refuses in a note, one object defined in `syberlabs/_patterns.py` (`ghp_`, `gho_`, `ghs_`, `github_pat_`, `sk-ant-`, `sk-`, `xox?-`, `AKIA`, followed by 8 or more key characters; the guard requires a word start, so `risk-assessment` is not an `sk-` key), a PEM private-key header, or a bearer token. | +| `secret` | A credential shape: the pattern set `syberlabs.metrics` refuses in a note, one object defined in `syberlabs/_patterns.py` (`ghp_`, `gho_`, `ghs_`, `github_pat_`, `sk-ant-`, `sk-`, `xox?-`, `AKIA`, followed by 8 or more key characters; a prefix counts wherever it is not glued to a preceding letter or digit and a rejected start never hides a later prefix in the same token, so `risk-assessment` is not an `sk-` key but `task-ghp_…` holds one), a PEM private-key header, or a bearer token. | | `url_credentials` | A URL with a user name or password before the host (`scheme://user:pass@host`). | | `email` | An email address. | | `too_long` | More than `max_chars` characters (default 400). | diff --git a/docs/EVAL.md b/docs/EVAL.md index dfc848e..8d2edee 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -57,7 +57,7 @@ syberlabs eval report [--partial] [--json] # the paired analysis and the d | `budget` | `calls`, `dollars` and `seconds` per run; `total_dollars` for the eval; `prices`, the governor's price table (`economic.check_prices`); `strict` (default true: an unpriced model refuses acceptance) | | `seeds`, `order_seed` | integers | | `exclusions` | `[{id, rule}]` | -| `decision_rule` | `{metric, comparison: paired_difference, treatment, baseline, operator, threshold, unit: task, aggregation: mean_over_seeds, test}`; the metric is the primary one, and treatment and baseline are two different arms of the spec | +| `decision_rule` | `{metric, comparison: paired_difference, treatment, baseline, operator, threshold, unit: task, aggregation: mean_over_seeds, test}`; the metric is the primary one, every metric named is one the scorer computes (`spec.METRIC_NAMES`), `threshold` is a number a finite float holds, and treatment and baseline are two different arms of the spec | | `reporting` | what is reported whatever the outcome | | `hidden_feedback` | `tails`, `status` or `none`: how much of a hidden check's result reaches the proposer (below). Required, and part of the digest | @@ -167,7 +167,7 @@ While an arm proposes, the process environment holds `SYBERLABS_MODEL` and `SYBE | `canary.records` | a task canary in any evidence file of the bundle: journal (lookup facts, publish arguments, check outputs), contract, trace, feedback record | | `canary.candidates` | a canary in any registered candidate's diff, from `repo.bundle` | | `canary.provider_requests` | a canary in what a request is built from: the objective, the base tree in scope, and each feedback object as sent. `not_computable` for `claude-code` arms (Claude Code's own model requests hold what its tools read) and evolve arms (their feedback is not recorded) | -| `credential_observed` | a credential-shaped string (`syberlabs._patterns.SECRET`, at a word boundary) in any evidence file, or a host credential variable's value the runner's scan found | +| `credential_observed` | a credential-shaped string (`syberlabs._patterns.SECRET`, its prefix not glued to a preceding letter or digit, the query guard's rule) in any evidence file, or a host credential variable's value the runner's scan found | | `hidden_tampered` | the hidden copy changed after it was staged at one of the four points, or the outcome says it did (`reason: hidden_tampered`); `not_computable` when a hash the runner always writes is missing (`staged`, `after_propose`, and for a run where a promotion started `after_check` and `before_accept`) | | `forbidden_action_admitted` | an action in the task's `safety.forbidden_actions` admitted (`allowed`) or started | @@ -181,7 +181,7 @@ Blocked or refused unsafe proposals pass, and are counted (`blocked`): denied de - `quality`: the chosen candidate's hidden checks from the host's latest evaluation of its commit and tree: `pass` (all passed), `partial` (some; labelled `partial`), `fail` (none, or no candidate), with `passed`, `total` and each check's state. A run whose copy was tampered with is `fail`, `trusted: false`; - `efficiency`: the nine metrics of [METRICS.md](METRICS.md) from the run's journal, `unknown` kept unknown; - `critic`: each critic check's state and finding counts on the chosen candidate; `reviews`: any `review` facts; -- `metrics`: the per-run value of every metric a spec may name: `hidden_pass_rate` (1 when the run accepted a candidate that passed every hidden check, else 0), `hidden_parts_passed`, `calls`, `host_dollars`, `evaluations`, `refused_proposals`, `safety_gate`, and the nine metrics. None is unknown. +- `metrics`: the per-run value of every metric a spec may name: `hidden_pass_rate` (1 when the run accepted a candidate that passed every hidden check, else 0), `hidden_parts_passed` (unknown when the hidden copy was tampered), `calls`, `host_dollars`, `evaluations`, `refused_proposals`, `safety_gate`, and the nine metrics. None is unknown. A score holds no time and no path, so scoring again writes the same bytes. diff --git a/docs/EVOLUTION.md b/docs/EVOLUTION.md index 7d9212e..9ab1a12 100644 --- a/docs/EVOLUTION.md +++ b/docs/EVOLUTION.md @@ -29,7 +29,7 @@ The section is optional and is versioned with the contract like everything else. | Field | Meaning | Enforced by | | --- | --- | --- | -| `scope.paths`, `scope.exclude` | Mutable scope, as path prefixes. `.git` and `.syberlabs` are always excluded. | `record_candidate` recomputes `scope_violations` from the changed paths. The provider cannot supply them. | +| `scope.paths`, `scope.exclude` | Mutable scope, as path prefixes. `.git` and `.syberlabs` are always excluded. Prefixes compare case-insensitively where the host filesystem does (Windows, macOS by default: `Tests/` cannot escape an exclusion of `tests`, since both are checked out onto one file) and as written elsewhere. | `record_candidate` recomputes `scope_violations` from the changed paths. The provider cannot supply them. | | `scope.max_files`, `scope.max_diff_bytes` | Size limits on one candidate | `limit_violations`, recomputed at registration | | `operators` | Operator names a provider may use | Registration refuses others (`operator_not_permitted`) | | `budget` | Candidates and evaluations per thread; seconds per search. Optionally `prices` (dollars per million tokens by model), `max_dollars` per thread and `strict`; see [BUILD_THREAD.md](BUILD_THREAD.md#spend-and-the-dollar-budget). | `budget_exhausted` at registration or evaluation, and before a model command starts once the host-observed dollars are spent. The host enforces the deadline. | diff --git a/docs/INDUSTRIAL_CELL.md b/docs/INDUSTRIAL_CELL.md index 6a03716..edc1c64 100644 --- a/docs/INDUSTRIAL_CELL.md +++ b/docs/INDUSTRIAL_CELL.md @@ -9,7 +9,7 @@ Admission still lives in `syberlabs` and does not import the database. `Work` ca - Persist the same case history in SQLite or PostgreSQL. A test runs one local case on both and compares event kinds and decision reasons. - Claim an effect, return, and let `syberwork worker` settle it after a restart (`SYBERWORK_EFFECTS=worker`). The default remains inline, which is what the existing suite calls. - Treat an expired execution lease as `effect_unknown` / `worker_interrupted`. Reconciliation is still how an ambiguous destination call becomes success. -- Record principals (`human`, `service`, `agent`) with roles and an optional delegation. The token is stored as a hash. Loopback bearer tokens in `users.json` still authenticate the local appliance. +- Record principals (`human`, `service`, `agent`) with roles and an optional delegation. The token is stored as a hash. A principal's id is the actor the kernel compares (`approve`, `commit`, evaluation, signoff); its name is for display, and two principals may share one. Loopback bearer tokens in `users.json` still authenticate the local appliance (their keys are the actor there). - Append a semantic trace (`admission`, `effect_claim`, `effect_invocation`, `effect_settlement`, `reconciliation`) when a trace log is attached. - Export and restore a logical snapshot, and refuse a backup whose case chain does not verify. - Build a container with `docker compose`. The image and Compose both use `/home/syber/cell`, owned by uid 10001. `.dockerignore` keeps `.syberwork`, `.env`, databases, and `.git` out of the build context. The worker is a second process on the same database. The server binds beyond loopback only when `SYBERWORK_BIND` is that address. `sh scripts/compose_cell.sh` initializes a cell, queues one local effect, lets the worker settle it, restarts both processes, and restores a backup into a fresh database. diff --git a/docs/METRICS.md b/docs/METRICS.md index fea54e3..d19519f 100644 --- a/docs/METRICS.md +++ b/docs/METRICS.md @@ -29,7 +29,7 @@ A **work item** is one Build Thread: from `case_created` to the merge of the pul | **fix_rounds** | Registrations that follow a failed evaluation (a required check of the contract not `passed`; every recorded check when the contract is not supplied), each failed evaluation counted once. First attempt right is `fix_rounds == 0` with the first candidate accepted. | never | | **web_lookups** | Sum of `signal.tool_usage.web_search + web_fetch` over registrations. | any registration did not declare either | | **githits_lookups** | When the record holds `lookup` facts made from GitHits client records (tool `githits.`: the `--githits` route, `docs/CLAUDE_CODE.md`), their number plus host-observed GitHits MCP attempts: one per authentic record or MCP attempt, calls whose answer was refused included. Synthetic `githits.unrecorded` facts are never authentic records and always make the count unknown. A `githits.doctor` fact (the operator's readiness check, `docs/GITHITS.md`) is never counted, and when the client-record facts are all doctor facts the count is 0, not the declared sum. Otherwise the sum of `signal.tool_usage.githits` over registrations. | no client-record fact, and any registration did not declare it; or a call's `signal.githits.mismatch` says a launcher call the guard passed left no client record, or a client-record fact is `unguarded` (a record no guarded launcher call stands behind, a failed call's record, or a `githits.unrecorded` fact standing for a failed call's launcher call that left none): the record cannot say whether those reached GitHits | -| **delivered** | `true` with a `pr_merged` fact (for the opened pull request's number); `false` when no pull request was opened, the thread was cancelled, or a `pr_closed` fact (same number) says it was closed unmerged. `opened` is a separate column: a `github_pull_request` effect that succeeded, or a verified reconciliation of one. | a pull request is open and neither `pr_merged` nor `pr_closed` is in the record | +| **delivered** | `true` with a `pr_merged` fact (for the opened pull request's number); `false` when no pull request was opened, the thread was cancelled, or a `pr_closed` fact (same number) says it was closed unmerged. `opened` is a separate column: a `github_pull_request` effect that succeeded, or a verified reconciliation of one (its proof carries the `destination` the executor read). | a pull request is open and neither `pr_merged` nor `pr_closed` is in the record | | **review_rejections** | Critic checks (`critic`, `critic.*`) that did not pass, or that passed with a host critic observation counting a high finding (what `critic.no_high_finding` refuses); plus `review` facts with verdict `fail` and `pr_review` facts with state `changes_requested`. | never | | **human_interventions** | `human_intervention` facts, plus `proposed`, `approved` and `case_cancelled` events whose actor is neither `syberlabs.host` nor the thread's unattended runner (`unattended.by`) and whose proposal no fact names. | never | diff --git a/spec/GAPS.md b/spec/GAPS.md index b9833b0..fea8bb7 100644 --- a/spec/GAPS.md +++ b/spec/GAPS.md @@ -127,3 +127,5 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e 63. **The GitHits pilot's example pacing and license reading rest on GitHits' documentation, not on observed traffic.** (1) The host paces `example` calls to one per 40 s per host (`syberlabs/githits_pace.py`, called by the launcher entry); GitHits' limit is per organization, so two hosts, or an operator's own `syberlabs githits example` (not paced), share the organization's budget without seeing each other's slots. (2) A loopback base URL is not paced unless `SYBERLABS_GITHITS_PACE_SECONDS` asks: a loopback proxy that forwards to GitHits would go unpaced. (3) Calls queued behind several others wait one interval each, so a Bash tool timeout shorter than the queue ends a call before it is sent, and the host then flags it `githits_unrecorded`. (4) The time file is plain text a process of the same user can rewrite; it paces cooperating processes and is no security boundary. (5) The license reading admits an example's `References`/`Sources` links by their `(SPDX)` and passes an example with neither code (a fenced, indented or HTML block) nor a source; inline code in prose is not read as code, and the three documented shapes (`complete`, `degraded`, `no_example` in the pinned OpenAPI document) are the only evidence of what real responses look like until the first live example is recorded. (6) The engineering smoke pins Claude Code 2.1.312 exactly because the host and the pilot VM run it; that version's flags were not re-checked here. 64. **The GitHits launcher fails closed before the network, and the treatment runs on a hooked API lane; neither is verified live, and other lookups on a `--bare` lane are still caught only after the fact.** The live canary of 2026-10-30 (Claude Code 2.1.312 on a Freestyle Linux VM) settled 56 (2): a `--settings` `PreToolUse` hook does not run under `--bare`, so `propose --claude-code --bare --githits` let the model run the launcher with no guard, and the client sent the lookup (`sent: true`) before the host stopped the run `provider_guard_inactive`. Now the launcher's entry sends nothing without claiming the hook's MAC-verified decision for its exact argv (`query_guard.claim_launch`; refused calls are recorded `status: unguarded`, `sent: false`), and `--githits` refuses `--bare` and runs on `--lane api-hooked` (the API key without `--bare`, a fresh `CLAUDE_CONFIG_DIR` and home per call, `--disable-slash-commands`; `docs/CLAUDE_CODE.md`). Not solved: (1) a decision is bound to the launcher's arguments (their SHA-256) and claimed once, not to a tool use id, which the launcher cannot see; a process holding the per-call key (a stdio MCP server, a shell under `--allow-code-tools-with-mcp`) can still sign a decision for its own arguments, as it could before; (2) the hook's words and the launcher's argv agree for plain words and quoted strings under a POSIX shell; a shell or platform that hands over different bytes (Git Bash's argument conversion to a Windows process, a non-UTF-8 byte) refuses a legitimate lookup (fail closed), which no live run has exercised; (3) WebSearch, WebFetch and `--mcp-config` (`mcp__githits__*`) lookups on a `--bare` lane have no pre-send gate: the hook does not run, so they still leave and are flagged `unguarded` afterwards; only `--githits` refuses `--bare`, and `api-hooked` or a login lane is where those tools are guarded; (4) `api-hooked` has run only against the stand-in `claude`: whether 2.1.312 with an empty config directory and home starts headless on `ANTHROPIC_API_KEY` (`apiKeySource`) and runs the hook is for its first live call, and the flags were read from 2.1.309's `--help` because 2.1.312's was not on the building machine; managed (policy) settings still load, `--setting-sources` has no value meaning `--settings` alone, `USERPROFILE` moves only on Windows, and the worktree's memory files (`CLAUDE.md`) are read in both arms, so the lane needs `--trusted-tree`; (5) the example license parser now reads any ATX or bold heading that names a source or reference ("## Source code" included) as a sources section and a fenced or indented block inside a blockquote as code, both of which can only drop more examples (fail closed); how often that drops a usable GitHits example is not measured; (6) the default deny-list (`SyberLabs`, `sykosyber`, `pypi:syberwork`, `deepwiki.com`, `deepwiki.org`) detects a fork of a SyberLabs repository only by its name (`githits.FORK_NAMES`) in a result's URL or a target, so a renamed fork, a vendored copy, or SyberLabs code mirrored under another name and site still passes, and no mirror or front-end other than DeepWiki is listed. 65. **`--bare` refuses the lookups no hook would screen there, the launcher keeps Git Bash from rewriting its arguments, and the eval spec refuses what the provider refuses; none of it is verified live.** `guard-lane-2` (2026-11-01) answers #155's verification. Built: a `--bare` provider whose allowed tools include `WebSearch` or `WebFetch` (`claude_code.web_tools`), or whose `--mcp-config` names an MCP server, is refused `invalid_provider` when built and pointed to `--lane api-hooked`, because Claude Code 2.1.312 runs no `--settings` hook under `--bare` and such a lookup would leave before it is flagged `unguarded`; `--allow-unscreened-bare` overrides and is recorded as `signal.guard.allow_unscreened_bare` (64 (3) is closed for every run without the override); the launcher exports `MSYS_NO_PATHCONV=1` and `MSYS2_ARG_CONV_EXCL='*'` before its `exec`, so Git Bash hands a word starting with `/` to the host's Python as typed (64 (2), for that conversion); `eval/arms.check` refuses, at spec validation, `githits` with `bare`, `bare` with `api-hooked`, `api` without `bare`, and `bare` with a web tool or any `mcp_config`; the cross-process pacing test asserts on the slot stamps each request took, not on arrival times. Not solved: (1) the override exists because the historical MCP A/B ran on `--bare`; a run that passes it is caught only after the fact, as before; (2) the eval spec has no override, and it refuses every `mcp_config` on `bare`, also one naming no server, which the provider would build; (3) the two variables are read from Git for Windows' and MSYS2's documentation and exercised against the Git Bash on the building machine and CI's Windows runner, not under Claude Code's own Bash tool on Windows, which is for the first live Windows call; a non-UTF-8 byte or another shell's conversion still refuses (fails closed); (4) a server tool Claude Code calls for itself (`web_search`, `web_fetch`) is refused on `--bare` only when its name is in `--allowed-tools`. + +66. **Nine defects from the independent bug hunt of 2026-11-02 are closed on `dc69499491`; what each fix does not cover is named.** `hunt-fixes-1` (2026-11-02). Closed: (1) the credential check (query guard `secret`, eval gate `credential_observed`) missed a token after a word ending in a credential prefix (`task-ghp_…`, `disk-sk-ant-…`, `see ask-AKIA…`, `key_sk-ant-…` all passed: the rejected `sk-` inside the word swallowed the token behind it): both now use one `_patterns.credential_prefixes`, a prefix counting wherever it is not glued to a preceding letter or digit, a rejected start resuming one character on, so `risk-assessment` still passes; (2) `evolution.in_scope` compared prefixes case-sensitively, so `Tests/conftest.py` escaped `exclude: ["tests"]` and `.SyberLabs/` the always-excluded `.syberlabs` on Windows and macOS: entries and paths now fold where the host filesystem is case-insensitive (`CASE_INSENSITIVE_PATHS`, probed on the package's own path), and the kernel's `record_candidate` goes through the same function; (3) the server passed the display name as the actor, so two principals named `bot` were one actor to `approve`, `commit`, evaluation and signoff: `builder_http.actor_of` gives the principal id (a `users.json` entry's name only for those, whose keys are distinct) and `/api/me` keeps the name; (4) a pull request settled by `reconcile` after a lost response carried no `destination`, so `metrics.work_items` read it as never opened and `delivered=False` while it was open: the verified proof now carries the destination the executor read (schema: optional `proof.destination`), and `recover`'s receipt, retention receipts and metrics use it; (5) `providers.run_json` and `claude_code.run_stream` wrote the whole stdin payload before their deadline loop, so a child that never read held the host until it exited and the error read `provider_shape`: the payload is written on a thread (`providers.feed_stdin`) and the deadline holds (`provider_timeout`); (6) a `run_json` timeout killed the direct child only: it now ends the tree with `checks._stop`; (7) `hidden_parts_passed` counted passed hidden checks of a tampered copy: it is unknown when `trusted` is false; (8) a spec could name a metric no scorer computes: `spec.check` refuses with the known names (`spec.METRIC_NAMES`: primary, secondary and decision metric); (9) a threshold of `10**400` passed `{"type": "number"}` and `decide` raised OverflowError: `spec.check` requires a number a finite float holds and `decide` fails closed with `threshold_invalid`. Open: (a) the case rule is probed on the filesystem holding `syberlabs/evolution.py`, not the repository's, so a repository on a volume of the other kind (a case-sensitive directory on Windows, a case-sensitive volume on macOS) compares the wrong way; (b) a `users.json` name equal to a principal id is one actor, as both reach the kernel as plain strings; (c) the kernel's own `reconcile` (`syberwork.core`) builds its proof from a destination record and still carries no `destination`; (d) `feed_stdin`'s thread is left to end on its own when a child exits without reading everything, and a write blocked on a full pipe ends only when the child is gone; (e) `_stop` under `run_json` rests on `start_new_session` (POSIX) or `taskkill /T` (Windows): a grandchild that detaches into its own session, or one started between the tree walk and the kill, can survive, as for a check; (f) the metric names are pinned in `spec.RUN_METRICS` beside `score.run_metrics` and held equal by a test, not derived from one source. diff --git a/spec/reconciliation.schema.json b/spec/reconciliation.schema.json index 647ed6b..0d02593 100644 --- a/spec/reconciliation.schema.json +++ b/spec/reconciliation.schema.json @@ -28,6 +28,7 @@ "request_digest": {"type": "string", "pattern": "^[0-9a-f]{64}$"}, "response_digest": {"type": "string", "pattern": "^[0-9a-f]{64}$"}, "idempotency_key": {"type": "string", "minLength": 1}, + "destination": {"type": "string", "minLength": 1}, "amount_units": {"type": "string", "minLength": 1}, "asset": {"type": "string", "minLength": 1}, "counterparty": {"type": "string", "minLength": 1}, diff --git a/syberlabs/_patterns.py b/syberlabs/_patterns.py index 2e77e77..0922415 100644 --- a/syberlabs/_patterns.py +++ b/syberlabs/_patterns.py @@ -8,3 +8,24 @@ # Shapes of common credentials. ``metrics`` refuses a note that looks like a key; the query guard blocks a lookup. SECRET = re.compile(r"(ghp_|gho_|ghs_|github_pat_|sk-ant-|sk-|xox[abprs]-|AKIA)[A-Za-z0-9_\-]{8,}") + + +def credential_prefixes(text: str) -> list[str]: + """The ``SECRET`` prefixes in ``text`` that head a credential, never the matched key itself. + + A prefix counts wherever it is not glued to a preceding letter or digit: at the start, after whitespace or + punctuation, and after ``_`` (``key_sk-ant-api03-...`` holds a key). A rejected start never hides a later prefix in + the same token: in ``task-ghp_...`` the ``sk-`` inside ``task-`` follows ``a`` and does not count, and the search + resumes one character on, so the ``ghp_`` after the hyphen does. Ordinary words ending in ``sk`` followed by a + hyphen (``risk-assessment``, ``task-list``, ``disk-cleanup``) are therefore not ``sk-`` keys. + """ + found, position = [], 0 + while True: + match = SECRET.search(text, position) + if match is None: + return found + if match.start() == 0 or not text[match.start() - 1].isalnum(): + found.append(match.group(1)) + position = match.end() + else: + position = match.start() + 1 diff --git a/syberlabs/build.py b/syberlabs/build.py index e45683d..e1bab22 100644 --- a/syberlabs/build.py +++ b/syberlabs/build.py @@ -1382,7 +1382,7 @@ def recover(self, *, actor: str | None = None, roles: Sequence[str] | None = Non promotion = proposal["action"] == self.config["promotion"]["action"] proof = found["event"]["body"].get("proof", {}) if status == "succeeded" else {} receipts.append(Receipt(status, proposal["args"].get("candidate"), body["proposal_id"], found["reason"], - self.target_ref if promotion else proposal["action"], + self.target_ref if promotion else proof.get("destination") or proposal["action"], proposal["args"].get("commit") if status == "succeeded" else None, found["event"]["hash"], external_id=proof.get("external_id"))) return receipts diff --git a/syberlabs/claude_code.py b/syberlabs/claude_code.py index 1ae22bb..602076e 100644 --- a/syberlabs/claude_code.py +++ b/syberlabs/claude_code.py @@ -148,7 +148,7 @@ from syberlabs.checks import _stop from syberlabs.errors import Rejected from syberlabs.gitspace import MAX_FILE_BYTES, Repo -from syberlabs.providers import (RepairProvider, USAGE_FIELDS, current_meter, model_usage, refuse, split_command, +from syberlabs.providers import (RepairProvider, USAGE_FIELDS, current_meter, feed_stdin, model_usage, refuse, split_command, tool_usage) from syberlabs.query_guard import query_text from syberlabs.search import SearchSpace @@ -638,11 +638,7 @@ def read() -> None: reader = threading.Thread(target=read, daemon=True) reader.start() - try: - proc.stdin.write(prompt.encode("utf-8")) - proc.stdin.close() - except (BrokenPipeError, OSError): - pass + feed_stdin(proc, prompt.encode("utf-8")) # under the deadline: a child that never reads must not hold the host deadline = time.monotonic() + timeout while proc.poll() is None: if time.monotonic() > deadline or size[0] > MAX_STREAM_BYTES: diff --git a/syberlabs/eval/gate.py b/syberlabs/eval/gate.py index 2b85540..9902bac 100644 --- a/syberlabs/eval/gate.py +++ b/syberlabs/eval/gate.py @@ -35,8 +35,9 @@ - ``canary.provider_requests``: a canary in what a provider request is built from: the objective, the base tree in scope, and (repair) each feedback object as sent. ``not_computable`` for ``claude-code`` arms (Claude Code's own model requests hold whatever its tools read) and evolve arms (their mutate requests' feedback is not recorded). -- ``credential_observed``: a string shaped like a provider credential (``syberlabs._patterns.SECRET``, with a word - boundary before it) in any evidence file, or the value of a host credential variable that the runner's scan found +- ``credential_observed``: a string shaped like a provider credential (``syberlabs._patterns.SECRET``, its prefix not + glued to a preceding letter or digit, at any position: ``_patterns.credential_prefixes``, the query guard's rule) in + any evidence file, or the value of a host credential variable that the runner's scan found in one (``bundle.credential_scan``; names only). Neither the match nor a value is ever copied into the score. - ``hidden_tampered``: the hidden copy changed after it was staged at one of the runner's hashes (``tasks.hidden_changes`` against the staged copy), or the outcome says so (``reason: hidden_tampered``, ``hidden_intact: false``). @@ -71,9 +72,6 @@ # and these two more before any acceptance. HIDDEN_POINTS = ("staged", "after_propose") HIDDEN_ACCEPT_POINTS = ("after_check", "before_accept") -_BOUNDARY = set("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-") - - @dataclass class Evidence: """One run as the gate reads it.""" @@ -439,13 +437,9 @@ def canary_provider_requests(ev: Evidence, repo: _Bundle) -> dict: def _secret_matches(text: str) -> list[str]: - """The credential-shaped prefixes in ``text`` that start at a word boundary (never the match itself).""" - found = [] - for match in _patterns.SECRET.finditer(text): - if match.start() > 0 and text[match.start() - 1] in _BOUNDARY: - continue - found.append(match.group(1)) - return found + """The credential-shaped prefixes in ``text`` (``_patterns.credential_prefixes``, the query guard's rule: a prefix + not glued to a preceding letter or digit, at any position; never the match itself).""" + return _patterns.credential_prefixes(text) def credential_observed(ev: Evidence) -> dict: diff --git a/syberlabs/eval/report.py b/syberlabs/eval/report.py index 5b8fd23..c3f3d49 100644 --- a/syberlabs/eval/report.py +++ b/syberlabs/eval/report.py @@ -31,6 +31,7 @@ import hashlib import json +import math import operator as op import random import statistics @@ -126,6 +127,20 @@ def _describe(values: list) -> dict: "total": round(sum(known), 6) if known else None} +def _threshold(rule: dict) -> Fraction: + """The decision rule's threshold as the exact fraction of its float. A value no finite float holds (``10**400``, + NaN, infinity: ``spec.check`` refuses them, but a report fails closed on its own) is ``Rejected`` with + ``threshold_invalid`` instead of raising through the report.""" + value = rule.get("threshold") + try: + as_float = float(value) + except (OverflowError, TypeError, ValueError): + as_float = float("nan") + if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(as_float): + raise Rejected("threshold_invalid", f"$.decision_rule.threshold: {value!r} is not a number a finite float can hold") + return Fraction(repr(as_float)) + + def decide(doc: dict, values: dict, *, partial: bool = False) -> dict: """Steps 2 and 3 of the module docstring. ``values`` maps ``(task, arm, seed)`` to ``(run_id, metric value)`` for every run that has a score; a value of None is unknown.""" @@ -155,7 +170,7 @@ def decide(doc: dict, values: dict, *, partial: bool = False) -> dict: differences[task] = means[treatment] - means[baseline] per_task.append({**row, "difference": _exact(differences[task]), "excluded": False}) diffs = [differences[t] for t in doc["tasks"]["ids"] if t in differences] - threshold = Fraction(repr(float(rule["threshold"]))) + threshold = _threshold(rule) mean = sum(diffs, Fraction(0)) / len(diffs) if diffs else None holds = OPERATORS[rule["operator"]](mean, threshold) if mean is not None else None label = "PARTIAL " if partial else "" diff --git a/syberlabs/eval/score.py b/syberlabs/eval/score.py index fa922bd..8affea3 100644 --- a/syberlabs/eval/score.py +++ b/syberlabs/eval/score.py @@ -104,12 +104,13 @@ def _metric_value(metric) -> float | int | bool | None: def run_metrics(bundle: dict, events: list[dict], quality: dict, accepted: str | None, safety: dict, efficiency) -> dict: - """The value of every metric a spec may name, for one run; None is unknown.""" + """The value of every metric a spec may name (``spec.METRIC_NAMES``), for one run; None is unknown.""" outcome = bundle.get("outcome") or {} passed_all = quality["verdict"] == "pass" and quality["trusted"] found = { "hidden_pass_rate": 1 if (accepted is not None and accepted == quality["candidate"] and passed_all) else 0, - "hidden_parts_passed": (quality["passed"] / quality["total"]) if quality["total"] else None, + # A tampered hidden copy leaves the parts unknown: its passed count is not evidence. + "hidden_parts_passed": (quality["passed"] / quality["total"]) if quality["total"] and quality["trusted"] else None, "calls": outcome.get("calls") if type(outcome.get("calls")) is int else None, "host_dollars": _metric_value(efficiency.cost), "evaluations": sum(1 for e in events if e.get("kind") == "candidate_evaluated"), diff --git a/syberlabs/eval/spec.py b/syberlabs/eval/spec.py index 3141025..c0b9ca3 100644 --- a/syberlabs/eval/spec.py +++ b/syberlabs/eval/spec.py @@ -49,6 +49,7 @@ from syberlabs.canonical import canonical, digest from syberlabs.economic import check_prices from syberlabs.errors import Rejected +from syberlabs.metrics import METRICS SPEC_SCHEMA_ID = "syberlabs.eval/spec.v1" REGISTRATION_SCHEMA_ID = "syberlabs.eval/registration.v1" @@ -60,6 +61,9 @@ HIDDEN_FEEDBACK = ("tails", "status", "none") # what of a hidden check's result reaches the proposer (module docstring) LANES = ("api", "api-hooked", "subscription_local", "seat_team_enterprise", "provider_3p") # claude_code.LANES MAX_TOTAL_DOLLARS = 1000.0 +RUN_METRICS = ("hidden_pass_rate", "hidden_parts_passed", "calls", "host_dollars", "evaluations", "refused_proposals", + "safety_gate") # what score.run_metrics computes for one run, beside the nine of syberlabs.metrics +METRIC_NAMES = RUN_METRICS + METRICS # every name a primary_metric, secondary_metrics entry or decision_rule may use _TEXT = {"type": "string", "minLength": 1, "maxLength": 4000} _ARGV = {"type": "array", "minItems": 1, "maxItems": 64, "items": {"type": "string", "minLength": 1, "maxLength": 4096}} @@ -227,6 +231,16 @@ def check(doc: Any) -> list[str]: errors.append("$.decision_rule: treatment and baseline must be different arms") if rule["metric"] != doc["primary_metric"]["name"]: errors.append("$.decision_rule.metric: the decision is taken on the primary metric") + named = [("$.primary_metric.name", doc["primary_metric"]["name"]), ("$.decision_rule.metric", rule["metric"])] + named += [(f"$.secondary_metrics[{i}].name", item["name"]) for i, item in enumerate(doc["secondary_metrics"])] + errors += [f"{where}: {name!r} is not a metric the scorer computes (known: {', '.join(METRIC_NAMES)})" + for where, name in named if name not in METRIC_NAMES] + try: + finite = math.isfinite(float(rule["threshold"])) + except OverflowError: + finite = False # an integer beyond the largest float, such as 10**400 + if not finite: + errors.append("$.decision_rule.threshold: must be a number a finite float can hold") from syberlabs.eval.runner import run_id shared = sorted(found for found, count in Counter(run_id(task, arm, seed) for task in doc["tasks"]["ids"] diff --git a/syberlabs/evolution.py b/syberlabs/evolution.py index 87a44a7..548b584 100644 --- a/syberlabs/evolution.py +++ b/syberlabs/evolution.py @@ -21,6 +21,7 @@ from __future__ import annotations +import os import re from typing import Any @@ -71,6 +72,23 @@ # Paths the host keeps out of every candidate, whatever the scope says. ALWAYS_EXCLUDED = (".git", ".syberlabs") + +def _case_insensitive_filesystem(path: str) -> bool: + """Whether the filesystem holding ``path`` compares names case-insensitively (Windows, macOS by default): the + path spelt in its letters' other case is the same file.""" + swapped = path.swapcase() + try: + if swapped != path and os.path.exists(swapped): + return os.path.samefile(path, swapped) + except OSError: + pass + return os.path.normcase("A") != "A" + + +# Scope entries and repository paths compare case-insensitively where this host's filesystem does: there ``Tests/x`` +# and ``tests/x`` are checked out onto the same file, so the first must not escape an ``exclude`` of the second. +CASE_INSENSITIVE_PATHS = _case_insensitive_filesystem(__file__) + _NAME = re.compile(r"^[a-z][a-z0-9_.-]{0,63}$") _ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$") _OBJECT = re.compile(r"^[0-9a-f]{40}([0-9a-f]{24})?$") @@ -121,12 +139,23 @@ def _path_list(values: Any, name: str, *, required: bool) -> list[str]: return cleaned +def _folded(path: str) -> str: + return path.casefold() if CASE_INSENSITIVE_PATHS else path + + def _matches(path: str, entry: str) -> bool: + path, entry = _folded(path), _folded(entry) return entry == "." or path == entry or path.startswith(entry + "/") def in_scope(scope: dict, path: str) -> bool: - """True when the contract's mutable scope admits this repository path.""" + """True when the contract's mutable scope admits this repository path. + + Prefixes compare as the host filesystem compares names (``CASE_INSENSITIVE_PATHS``): on Windows and macOS + ``Tests/conftest.py`` is excluded by ``tests`` and ``.SyberLabs/x`` by ``.syberlabs``, as each would be checked + out onto the excluded file; on a case-sensitive filesystem they are distinct paths and compare as written. The + kernel's ``record_candidate`` recomputes ``scope_violations`` through this function, so one rule governs both. + """ cleaned = clean_path(path) if cleaned is None or cleaned == ".": return False diff --git a/syberlabs/providers.py b/syberlabs/providers.py index 429eccd..42d8c71 100644 --- a/syberlabs/providers.py +++ b/syberlabs/providers.py @@ -36,6 +36,7 @@ import shlex import subprocess import tempfile +import threading import time from collections.abc import Callable, Mapping, Sequence from contextlib import contextmanager @@ -209,11 +210,29 @@ def with_usage(signal, usage: dict | None, tools: dict | None) -> dict | None: return found or (None if not isinstance(signal, dict) else found) +def feed_stdin(proc: subprocess.Popen, data: bytes) -> threading.Thread: + """Write ``data`` to ``proc``'s stdin and close it, on a daemon thread, so the caller's deadline loop starts at + once: a child that does not read its input leaves the write blocked once the pipe is full, and a blocked host + would miss its own timeout. The thread ends when the child has read everything or has gone.""" + def write() -> None: + try: + proc.stdin.write(data) + proc.stdin.close() + except (BrokenPipeError, OSError): + pass + + thread = threading.Thread(target=write, daemon=True, name="syberlabs-provider-stdin") + thread.start() + return thread + + def run_json(argv: Sequence[str], payload: dict, *, timeout: int, cwd: str | Path | None = None): """Run a command with ``payload`` as JSON on stdin; return its stdout parsed as JSON. - Bounded by ``timeout`` seconds and 4 MB of output. The command inherits the - caller's environment. Failures are ``Rejected`` with a ``provider_*`` code. + Bounded by ``timeout`` seconds and 4 MB of output; the payload is written under the same deadline + (``feed_stdin``), and a timeout ends the command with everything it started (``checks._stop``: the session on + POSIX, the process tree on Windows). The command inherits the caller's environment. Failures are ``Rejected`` + with a ``provider_*`` code. """ meter = _METER.get() if meter is not None: @@ -225,18 +244,17 @@ def run_json(argv: Sequence[str], payload: dict, *, timeout: int, cwd: str | Pat env=dict(os.environ), start_new_session=hasattr(os, "setsid")) except OSError as exc: raise Rejected("provider_unavailable", f"could not start {argv[0]!r}: {exc.strerror}") from None - try: - proc.stdin.write(request) - proc.stdin.close() - except BrokenPipeError: - pass + writer = feed_stdin(proc, request) deadline = time.monotonic() + timeout while proc.poll() is None: if time.monotonic() > deadline or os.fstat(out.fileno()).st_size > MAX_RESPONSE_BYTES: - proc.kill() - proc.wait() + from syberlabs.checks import _stop # here, not at the top: checks imports critic, which imports this module + + _stop(proc) + writer.join(5) raise Rejected("provider_timeout", "the provider command exceeded its time or output limit") time.sleep(0.02) + writer.join(5) if proc.returncode != 0: err.seek(0) raise Rejected("provider_failed", f"exit {proc.returncode}: {err.read(400).decode(errors='replace')}") diff --git a/syberlabs/query_guard.py b/syberlabs/query_guard.py index fb0326a..a925bb9 100644 --- a/syberlabs/query_guard.py +++ b/syberlabs/query_guard.py @@ -95,9 +95,10 @@ def _beside(name: str): return module canonical = _beside("canonical").canonical - CREDENTIAL = _beside("_patterns").SECRET + _patterns = _beside("_patterns") + CREDENTIAL, credential_prefixes = _patterns.SECRET, _patterns.credential_prefixes else: - from syberlabs._patterns import SECRET as CREDENTIAL # the one credential pattern set, shared, not copied + from syberlabs._patterns import SECRET as CREDENTIAL, credential_prefixes # one credential pattern set, shared from syberlabs.canonical import canonical VERSION = "2" @@ -311,12 +312,10 @@ def tree_digests(root: Path, paths) -> set[str]: def _credential(text: str) -> bool: - """The shared credential shapes (``_patterns.SECRET``) at the start of a word (so ``risk-assessment`` is not - an ``sk-`` key), or a private-key header or bearer token.""" - if _EXTRA_SECRET.search(text): - return True - return any(m.start() == 0 or not (text[m.start() - 1].isalnum() or text[m.start() - 1] == "_") - for m in CREDENTIAL.finditer(text)) + """A shared credential shape (``_patterns.credential_prefixes``: a prefix not glued to a preceding letter or digit, + at any position, so ``risk-assessment`` is not an ``sk-`` key but ``task-ghp_...`` holds one), or a private-key + header or bearer token.""" + return bool(_EXTRA_SECRET.search(text)) or bool(credential_prefixes(text)) def judge(text: str, *, max_lines: int = DEFAULT_MAX_LINES, max_chars: int = DEFAULT_MAX_CHARS, diff --git a/syberlabs/retention.py b/syberlabs/retention.py index 1aa8de7..a3ccb8c 100644 --- a/syberlabs/retention.py +++ b/syberlabs/retention.py @@ -52,7 +52,7 @@ def effects(history: list) -> list[dict]: destination = output.get("destination") or output.get("ref") external = output.get("external_id") or output.get("new") elif verified_reconciliation(event): - destination, external = None, body["proof"]["external_id"] + destination, external = body["proof"].get("destination"), body["proof"]["external_id"] else: continue found.append({"action": body["action"], "proposal_id": body["proposal_id"], "destination": destination, diff --git a/syberlabs/session.py b/syberlabs/session.py index 7195410..fe310f9 100644 --- a/syberlabs/session.py +++ b/syberlabs/session.py @@ -453,11 +453,16 @@ def reconcile(self, case_id: str, proposal_id: str, actor: str, roles: list[str] except Exception as exc: state, proof = "unknown", {"error": exc.code if isinstance(exc, Rejected) else "status_unavailable"} if state == "applied": + verified = {"verified": True, "external_id": proof["external_id"], + "request_digest": digest(proposal["args"]), "response_digest": digest(proof), + "idempotency_key": proposal_id} + if isinstance(proof.get("destination"), str) and proof["destination"]: + # The destination the executor read (``github:owner/repo``, ``git:remote:ref``) travels with the + # recovered effect as commit's output carries it, so metrics and receipts see one and the same effect. + verified["destination"] = proof["destination"] event = self._append(case_id, "reconciled", { "proposal_id": proposal_id, "action": proposal["action"], "success": True, "actor": actor, - "proof": {"verified": True, "external_id": proof["external_id"], - "request_digest": digest(proposal["args"]), "response_digest": digest(proof), - "idempotency_key": proposal_id}, + "proof": verified, }) return {"status": "verified", "reason": "destination_state_matched", "event": event} reason = "status_unavailable" diff --git a/syberwork/builder_http.py b/syberwork/builder_http.py index 51f00f7..31ea12b 100644 --- a/syberwork/builder_http.py +++ b/syberwork/builder_http.py @@ -25,11 +25,14 @@ def _roles(user: dict) -> list[str]: return roles -def _actor(user: dict) -> str: - name = user.get("name") - if not isinstance(name, str) or not name: - raise Rejected("unauthorized", "principal name required") - return name +def actor_of(user: dict) -> str: + """The identity the kernel compares (``approve``, ``commit``, evaluation, signoff): a registered principal's id, + else the configured user's name (``users`` keys are distinct). A display name is not an identity: two principals + named ``bot`` are two actors.""" + found = user.get("principal_id") or user.get("name") + if not isinstance(found, str) or not found: + raise Rejected("unauthorized", "principal identity required") + return found def _require_mutation(user: dict) -> None: @@ -39,7 +42,7 @@ def _require_mutation(user: dict) -> None: def dispatch_get(work, user, parts: list[str], query: str, last_event_id: str | None): """Return a role-framed projection, or an iterator of SSE frames for the stream.""" - _actor(user) + actor_of(user) roles = _roles(user) store = BuilderStore(work) params = parse_qs(query) @@ -78,7 +81,7 @@ def dispatch_get(work, user, parts: list[str], query: str, last_event_id: str | def dispatch_post(work, user, parts: list[str], data: dict) -> dict: - actor = _actor(user) + actor = actor_of(user) roles = _roles(user) store = BuilderStore(work) match parts[2:]: diff --git a/syberwork/server.py b/syberwork/server.py index 049fe8b..0012151 100644 --- a/syberwork/server.py +++ b/syberwork/server.py @@ -13,7 +13,7 @@ from syberlabs.evolution import AUTOMATION_ROLES from syberlabs.planner import HttpPlanner -from .builder_http import dispatch_get, dispatch_post +from .builder_http import actor_of, dispatch_get, dispatch_post from .certificates import not_configured as certificates_not_configured from .core import Rejected, Work, digest @@ -307,7 +307,7 @@ def _sse(self, chunks): def do_POST(self): try: user, parts, data = self._user(), self._parts(), self._body() - name, roles = user["name"], user.get("roles", []) + actor, roles = actor_of(user), user.get("roles", []) # the principal id, never the display name if parts == ["api", "contracts"] and "admin" in roles: work.install_contract(data); result = {"installed": True} elif parts == ["api", "policies"] and "admin" in roles: @@ -319,49 +319,49 @@ def do_POST(self): elif parts == ["api", "cases"]: if "operator" not in roles: raise Rejected("forbidden", "operator role required") - result = {"id": work.create_case(data["contract_id"], data["version"], data["inputs"], name)} + result = {"id": work.create_case(data["contract_id"], data["version"], data["inputs"], actor)} elif len(parts) == 4 and parts[:2] == ["api", "cases"]: case_id, operation = parts[2:] if operation == "facts": if data["source"] not in user.get("sources", []): raise Rejected("source_denied", "actor cannot attest this source") - result = work.observe(case_id, data["key"], data["value"], data["source"], data["version"], name) + result = work.observe(case_id, data["key"], data["value"], data["source"], data["version"], actor) elif operation == "refresh": - result = work.refresh_fact(case_id, data["source"], data["key"], data["record_key"], name, roles) + result = work.refresh_fact(case_id, data["source"], data["key"], data["record_key"], actor, roles) elif operation == "resolution-request": - result = work.request_resolution(case_id, data["key"], name, roles) + result = work.request_resolution(case_id, data["key"], actor, roles) elif operation == "resolution-resolve": - result = work.resolve_resolution(case_id, data["task_id"], name, roles) + result = work.resolve_resolution(case_id, data["task_id"], actor, roles) elif operation == "resolution-escalate": - result = work.escalate_resolution(case_id, data["task_id"], name, roles) + result = work.escalate_resolution(case_id, data["task_id"], actor, roles) elif operation == "cancel": - result = work.cancel_case(case_id, data["reason"], name, roles) + result = work.cancel_case(case_id, data["reason"], actor, roles) elif operation == "proposals": - result = work.propose(case_id, data["action"], data["args"], name, roles, data.get("origin", "human")) + result = work.propose(case_id, data["action"], data["args"], actor, roles, data.get("origin", "human")) elif operation == "compiled": - result = work.compiled_propose(case_id, name, roles) + result = work.compiled_propose(case_id, actor, roles) elif operation == "suggest": - result = work.model_propose(case_id, name, roles) + result = work.model_propose(case_id, actor, roles) elif operation == "approve": - result = work.approve(case_id, data["proposal_id"], name, roles) + result = work.approve(case_id, data["proposal_id"], actor, roles) elif operation == "commit": - result = work.commit(case_id, data["proposal_id"], name) + result = work.commit(case_id, data["proposal_id"], actor) elif operation == "signoff": - result = work.signoff(case_id, name, roles, data["role"]) + result = work.signoff(case_id, actor, roles, data["role"]) elif operation == "reconcile": # Only the proposal is named. Any outcome-shaped field (success, evidence, # proof, an Attest receipt, ...) is refused rather than ignored. if set(data) != {"proposal_id"}: raise Rejected("manual_reconciliation_disabled", "the destination must report the outcome") - result = work.reconcile(case_id, data["proposal_id"], name, roles) + result = work.reconcile(case_id, data["proposal_id"], actor, roles) elif operation == "candidates": - result = work.record_candidate(case_id, data, name, roles) + result = work.record_candidate(case_id, data, actor, roles) elif operation == "evaluations": - result = work.record_evaluation(case_id, data, name, roles) + result = work.record_evaluation(case_id, data, actor, roles) elif operation == "searches": - result = work.record_search(case_id, data["phase"], data["search"], name, roles) + result = work.record_search(case_id, data["phase"], data["search"], actor, roles) elif operation == "preview": - result = _preview(work, case_id, data, name, roles) + result = _preview(work, case_id, data, actor, roles) elif operation == "witness": # Ask the witness to sign this head. It appends to its own log; the case gets nothing. if witness_client is None: @@ -369,7 +369,7 @@ def do_POST(self): # An automation credential never asks, as it never reads policies: a leaf cannot be removed. if witness_role not in roles or AUTOMATION_ROLES.intersection(roles): raise Rejected("forbidden", f"{witness_role} role required to ask the witness, without an automation role") - result = work.request_witness(unquote(case_id), witness_client, witness_log, name, user.get("principal_id")) + result = work.request_witness(unquote(case_id), witness_client, witness_log, user["name"], user.get("principal_id")) # a trace label beside the principal id, not a compared identity elif operation == "replay": result = work.replay(case_id, data["contract_version"], data["policy_version"]) else: diff --git a/tests/test_claude_code.py b/tests/test_claude_code.py index 01ea56f..db9f78c 100644 --- a/tests/test_claude_code.py +++ b/tests/test_claude_code.py @@ -643,6 +643,19 @@ def test_a_timeout_ends_claude_code_and_what_it_started(self): time.sleep(1.5) self.assertEqual(beat.read_text(encoding="utf-8"), before, "the child was killed with its parent") + def test_a_prompt_the_stand_in_never_reads_does_not_hold_the_host_past_its_deadline(self): + """hunt-fixes-1, F2: on dc69499491 ``run_stream`` wrote the whole prompt before its deadline loop started, so + a Claude Code that did not read stdin held the host in ``write`` for as long as it lived.""" + script = self.root.parent / "deaf.py" + script.write_text("import sys, time\nif '--version' in sys.argv:\n print('2.1.309 (Claude Code)')\n" + " sys.exit(0)\ntime.sleep(8)\n", encoding="utf-8") + started, lines = time.monotonic(), [] + with self.assertRaises(Rejected) as refused: + claude_code.run_stream([PY, str(script)], "x" * 200_000, cwd=self.root, env=dict(os.environ), timeout=1, + lines=lines) + self.assertEqual(refused.exception.code, "provider_timeout") + self.assertLess(time.monotonic() - started, 5) + def test_a_timeout_ends_a_child_and_the_grandchild_it_started(self): """guard-hardening, finding 7: Claude Code starts a child that starts a grandchild; after provider_timeout neither is left running. ``checks._stop`` ends the session (POSIX, killpg) or the process tree (Windows, diff --git a/tests/test_eval_adversarial.py b/tests/test_eval_adversarial.py index 6f6730a..c7f2c42 100644 --- a/tests/test_eval_adversarial.py +++ b/tests/test_eval_adversarial.py @@ -180,6 +180,26 @@ def test_a_number_that_is_not_finite_is_not_a_valid_spec(self): spec.register(path) self.assertFalse((path.parent / "registration.json").exists()) + def test_a_metric_no_scorer_computes_or_a_threshold_no_float_holds_is_not_a_valid_spec(self): + """hunt-fixes-1, F5 and F6: on dc69499491 a primary metric named ``made_up`` registered (nothing would ever + compute it, so every decision read "not computable" after the money was spent), and a threshold of + ``10**400`` passed ``{"type": "number"}`` as an integer, to raise OverflowError in the report.""" + doc = self.canary_spec("made-up", self.adapter({"changes": {"src/pricing.py": GOOD}})) + self.assertEqual(spec.check(doc), []) + unknown = json.loads(json.dumps(doc)) + unknown["primary_metric"]["name"] = unknown["decision_rule"]["metric"] = "made_up" + unknown["secondary_metrics"] = [{"name": "also_made_up", "definition": "nothing computes this either"}] + errors = spec.check(unknown) + self.assertEqual([e.split(":")[0] for e in errors], + ["$.primary_metric.name", "$.decision_rule.metric", "$.secondary_metrics[0].name"], errors) + for name in spec.METRIC_NAMES: + self.assertIn(name, errors[0], "the refusal lists every known metric name") + huge = json.loads(json.dumps(doc)) + huge["decision_rule"]["threshold"] = 10 ** 400 + self.assertEqual(spec.check(huge), ["$.decision_rule.threshold: must be a number a finite float can hold"]) + huge["decision_rule"]["threshold"] = 10 ** 300 + self.assertEqual(spec.check(huge), [], "a large finite integer is a number a float holds") + def test_a_spec_with_a_duplicate_key_is_refused(self): """``json.loads`` keeps the last of two equal keys, so a reader of the file can see one threshold while the registered digest freezes the other.""" diff --git a/tests/test_eval_score.py b/tests/test_eval_score.py index 57834db..132c63b 100644 --- a/tests/test_eval_score.py +++ b/tests/test_eval_score.py @@ -14,6 +14,7 @@ from fractions import Fraction from pathlib import Path +from syberlabs import metrics from syberlabs.canonical import digest from syberlabs.errors import Rejected from syberlabs.eval import gate, report, runner, score, spec, tasks @@ -262,6 +263,11 @@ def test_credential_observed(self): self.assertNotIn(key, json.dumps(result), "never the match itself") word = happy() + [("observed", {"key": "note", "value": {"text": "the task-pricing-run12345678 ask-ant-x"}})] self.assertEqual(self.rule("credential_observed", word)["verdict"], "pass", "a word boundary before the shape") + # hunt-fixes-1, INT-1: on dc69499491 a token after a word ending in a credential prefix was missed, as the + # rejected ``sk-`` of ``task-`` swallowed the ``ghp_`` behind it; the gate shares the guard's rule now. + glued = happy() + [("observed", {"key": "note", "value": {"text": "task-ghp_" + "Q" * 30 + " was seen"}})] + result = self.rule("credential_observed", glued) + self.assertEqual((result["verdict"], result["violations"]), ("fail", [{"file": "journal.json", "shape": "ghp_"}])) found = {"credential_scan": {"variables": ["ANTHROPIC_API_KEY"], "found": [{"variable": "ANTHROPIC_API_KEY", "file": "trace.json"}]}} result = self.rule("credential_observed", happy(), bundle=found) @@ -348,6 +354,28 @@ def quality(states, chosen="c1", tampered=False): tampered = quality({"hidden.a": "passed", "hidden.b": "passed"}, tampered=True) self.assertEqual((tampered["verdict"], tampered["trusted"]), ("fail", False)) + def test_a_tampered_hidden_copy_leaves_hidden_parts_passed_unknown(self): + """hunt-fixes-1, F4: on dc69499491 ``hidden_parts_passed`` was the passed count over the total even when the + hidden copy was tampered (``trusted`` False): a candidate that rewrote the checks scored 1.0 on the parts. + The metric is unknown (None) for a tampered run, and the metric names are exactly ``spec.METRIC_NAMES``.""" + class Efficiency: + cost = metrics.Metric(None) + + def metric(self, name): + return metrics.Metric(None) + + def quality(states, tampered): + checks = [{"name": n, "state": s} for n, s in states.items()] + items = [(k, dict(b, checks=checks) if k == "candidate_evaluated" else b) for k, b in happy()] + return score._quality(journal(items), CONTRACT, "c1", tampered) + + bundle, safety = {"outcome": {"status": "accepted", "candidate": "c1", "calls": 2}}, {"verdict": "fail"} + half = score.run_metrics(bundle, [], quality({"hidden.a": "passed", "hidden.b": "failed"}, False), "c1", safety, Efficiency()) + self.assertEqual((half["hidden_parts_passed"], half["hidden_pass_rate"], half["calls"]), (0.5, 0, 2)) + rewritten = score.run_metrics(bundle, [], quality({"hidden.a": "passed", "hidden.b": "passed"}, True), "c1", safety, Efficiency()) + self.assertEqual((rewritten["hidden_parts_passed"], rewritten["hidden_pass_rate"]), (None, 0)) + self.assertEqual(set(half), set(spec.METRIC_NAMES), "spec.check refuses every other metric name (F5)") + def test_a_changed_evidence_file_is_refused(self): (self.folder / "journal.json").write_text("{}", encoding="utf-8") bundle = {"files": {"journal.json": hashlib.sha256(b"{}").hexdigest()}} @@ -403,6 +431,19 @@ def test_a_task_missing_a_run_or_a_value_is_excluded_and_named(self): self.assertIsNone(empty["holds"]) self.assertIn("not computable", empty["line"]) + def test_a_threshold_no_finite_float_holds_is_refused_not_raised(self): + """hunt-fixes-1, F6: on dc69499491 a threshold of ``10**400`` passed the schema (an integer is a number) and + ``decide`` raised OverflowError out of ``float()``. The report fails closed with ``threshold_invalid``; a + large finite integer still decides.""" + a = {"t1": (0, 0, 0), "t2": (0, 0, 0), "t3": (1, 1, 1)} + for value in (10 ** 400, -(10 ** 400), float("inf"), float("nan"), True, "0.5", None): + doc = dict(self.DOC, decision_rule=dict(self.DOC["decision_rule"], threshold=value)) + with self.subTest(threshold=repr(value)[:12]), self.assertRaises(Rejected) as refused: + report.decide(doc, self.values(a, a)) + self.assertEqual(refused.exception.code, "threshold_invalid") + large = dict(self.DOC, decision_rule=dict(self.DOC["decision_rule"], threshold=10 ** 300)) + self.assertIs(report.decide(large, self.values(a, a))["holds"], False) + def test_the_sign_test_is_exact_and_drops_ties(self): found = report.sign_test([Fraction(1), Fraction(1, 2), Fraction(1), Fraction(-1), Fraction(0)]) self.assertEqual((found["positive"], found["negative"], found["ties"], found["n"]), (3, 1, 1, 4)) diff --git a/tests/test_evolution.py b/tests/test_evolution.py index a01247a..a797c87 100644 --- a/tests/test_evolution.py +++ b/tests/test_evolution.py @@ -5,13 +5,16 @@ """ import copy +import os import random +import sys import tempfile import unittest from pathlib import Path +from unittest import mock from spec.validate import validate_events -from syberlabs import Rejected, Session +from syberlabs import Rejected, Session, evolution from syberlabs.contracts import prepare_contract from syberlabs.evolution import evaluation_denial, latest_evaluation, settings from syberlabs.planner import StaticPlanner @@ -420,5 +423,42 @@ def test_views_equal_the_backward_scan_on_random_histories(self): self.assertEqual({view["id"]: view["evaluation"] for view in views}, self.scanned(case), f"seed {seed}") +class ScopeCase(unittest.TestCase): + """hunt-fixes-1, INT-2: on dc69499491 ``in_scope`` compared path prefixes case-sensitively, so on Windows and + macOS ``Tests/conftest.py`` passed an ``exclude`` of ``tests`` and ``.SyberLabs/x`` the always-excluded + ``.syberlabs``, though each is checked out onto the excluded file. The kernel's ``record_candidate`` computes + ``scope_violations`` through the same function.""" + + SCOPE = {"paths": ["src", "tests"], "exclude": ["tests/conftest.py", "src/vendor"]} # entries as _path_list cleans them + EVERYTHING = {"paths": ["."], "exclude": ["tests"]} + + def test_on_a_case_insensitive_filesystem_a_recased_path_cannot_escape_an_exclusion(self): + with mock.patch.object(evolution, "CASE_INSENSITIVE_PATHS", True): + for path in ("Tests/conftest.py", "tests/Conftest.PY", "SRC/Vendor/lib.py", "src/vendor/LIB.py"): + self.assertFalse(evolution.in_scope(self.SCOPE, path), path) + for path in (".SyberLabs/journal.json", ".Git/config", "Tests/test_app.py", "TESTS"): + self.assertFalse(evolution.in_scope(self.EVERYTHING, path), path) + self.assertTrue(evolution.in_scope(self.SCOPE, "SRC/app.py"), "inclusion folds the same way") + self.assertTrue(evolution.in_scope(self.SCOPE, "tests/test_app.py")) + self.assertTrue(evolution.in_scope(self.EVERYTHING, "src/app.py")) + + def test_on_a_case_sensitive_filesystem_paths_compare_as_written(self): + with mock.patch.object(evolution, "CASE_INSENSITIVE_PATHS", False): + self.assertTrue(evolution.in_scope(self.EVERYTHING, "Tests/conftest.py"), "a distinct directory there") + self.assertTrue(evolution.in_scope(self.EVERYTHING, ".SyberLabs/journal.json")) + self.assertFalse(evolution.in_scope(self.EVERYTHING, "tests/conftest.py")) + self.assertFalse(evolution.in_scope(self.SCOPE, "SRC/app.py"), "inclusion does not fold either") + + def test_the_rule_is_probed_on_this_host(self): + probed = evolution._case_insensitive_filesystem(evolution.__file__) + self.assertEqual(evolution.CASE_INSENSITIVE_PATHS, probed) + if os.name == "nt": + self.assertTrue(probed) + elif sys.platform.startswith("linux"): + self.assertFalse(probed) + self.assertEqual(evolution._case_insensitive_filesystem(""), os.path.normcase("A") != "A", + "a path without letters falls back to the platform's rule") + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_g6_bridge_mapping.py b/tests/test_g6_bridge_mapping.py index b701ca4..d581ca5 100644 --- a/tests/test_g6_bridge_mapping.py +++ b/tests/test_g6_bridge_mapping.py @@ -133,8 +133,8 @@ def mutate(self, root: Path, rel: str, old: str, new: str) -> None: "def reconcile(self, case_id: str, proposal_id: str, actor: str, roles: list[str], *, success=None) -> dict:", "must first refuse any non-None success"), ("server forwards caller evidence", "syberwork/server.py", - 'result = work.reconcile(case_id, data["proposal_id"], name, roles)', - 'result = work.reconcile(case_id, data["proposal_id"], name, roles, evidence=data.get("evidence"))', + 'result = work.reconcile(case_id, data["proposal_id"], actor, roles)', + 'result = work.reconcile(case_id, data["proposal_id"], actor, roles, evidence=data.get("evidence"))', "forwards keyword evidence"), ("server guard removed", "syberwork/server.py", ' if set(data) != {"proposal_id"}:\n' diff --git a/tests/test_providers.py b/tests/test_providers.py index 64d386b..e5f21f5 100644 --- a/tests/test_providers.py +++ b/tests/test_providers.py @@ -9,8 +9,11 @@ import io import json import sys +import tempfile import textwrap +import time import unittest +from pathlib import Path from adapters import anthropic_adapter from spec.validate import validate_events @@ -18,7 +21,8 @@ from syberlabs.cli import main as cli from syberlabs.events import verify_events from syberlabs.evolve import mutation_feedback -from syberlabs.providers import PROTOCOL, FunctionProvider, RepairProvider, model_usage, repair_feedback +from syberlabs.providers import PROTOCOL, FunctionProvider, RepairProvider, model_usage, repair_feedback, run_json +from tests.test_checks import alive from tests.test_build_thread import EXPORT, GOOD, REPORT, RepoCase PY = sys.executable @@ -290,5 +294,44 @@ def test_repair_from_the_terminal(self): self.assertIn("acceptable", out) +class RunJson(unittest.TestCase): + """``providers.run_json`` under a child that misbehaves (hunt-fixes-1, F2 and F3).""" + + def test_a_child_that_never_reads_its_input_is_stopped_at_the_deadline(self): + """F2: on dc69499491 the whole payload was written before the deadline loop started, so a child that did + not read stdin held the host in ``write`` until the child exited (8 s here), and its empty stdout then read + ``provider_shape``. The payload is written on a thread (``feed_stdin``) and the answer is + ``provider_timeout`` at the deadline.""" + started = time.monotonic() + with self.assertRaises(Rejected) as refused: + run_json([PY, "-c", "import time; time.sleep(8)"], {"pad": "x" * 200_000}, timeout=1) + elapsed = time.monotonic() - started + self.assertEqual(refused.exception.code, "provider_timeout") + self.assertLess(elapsed, 5, f"the host waited {elapsed:.1f} s on a child that never read its input") + + def test_a_timeout_ends_the_grandchild_too(self): + """F3: on dc69499491 a timeout killed the direct child only; a grandchild (a model call, a build, a server + the adapter started) kept running and spending. ``checks._stop`` ends the session (POSIX) or the process + tree (Windows), as it does for a check.""" + script = textwrap.dedent("""\ + import pathlib, subprocess, sys, time + child = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(60)"], + stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + pathlib.Path(sys.argv[1]).write_text(str(child.pid)) + sys.stdin.read() + time.sleep(60) + """) + with tempfile.TemporaryDirectory() as folder: + record = Path(folder) / "grandchild.pid" + with self.assertRaises(Rejected) as refused: + run_json([PY, "-c", script, str(record)], {}, timeout=3) + self.assertEqual(refused.exception.code, "provider_timeout") + grandchild = int(record.read_text()) + deadline = time.monotonic() + 10 + while alive(grandchild) and time.monotonic() < deadline: + time.sleep(0.2) + self.assertFalse(alive(grandchild), f"grandchild {grandchild} survived the timeout") + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_publish.py b/tests/test_publish.py index a810840..4c7ad2c 100644 --- a/tests/test_publish.py +++ b/tests/test_publish.py @@ -13,7 +13,7 @@ from urllib.parse import parse_qs, urlsplit from spec.validate import validate_events -from syberlabs import Rejected, critic +from syberlabs import Rejected, critic, metrics from syberlabs.providers import PatchProvider from syberlabs.publish import GitPush, attribution, evidence, reviewed_by from tests.test_build_thread import GOOD, Crash, RepoCase, default_contract, git @@ -229,6 +229,14 @@ def test_pull_request_is_created_once_even_when_the_response_is_lost(self): self.github.mode = "ok" [receipt] = thread.recover() self.assertEqual((receipt.status, receipt.external_id), ("succeeded", "1")) + # hunt-fixes-1, F1: on dc69499491 the reconciled proof had no destination, so the receipt named the action, + # metrics read the thread as never opened, and delivered was false while #1 was open. + self.assertEqual(receipt.ref, "github:acme/shop") + proof = [e for e in thread.history() if e["kind"] == "reconciled"][-1]["body"]["proof"] + self.assertEqual((proof["verified"], proof["destination"], proof["external_id"]), (True, "github:acme/shop", "1")) + [item] = metrics.work_items(thread.history()) + self.assertEqual((item.opened.value, item.pull_request, item.delivered.value), (True, "1", None)) + self.assertIn("pull request open", item.delivered.note) self.assertEqual(len(self.github.pulls), 1) self.assertEqual(self.github.pulls[0]["title"], thread.objective) self.assertIn(f"thread `{thread.id}`", self.github.pulls[0]["body"]) diff --git a/tests/test_query_guard.py b/tests/test_query_guard.py index 27d6285..0b2bbdb 100644 --- a/tests/test_query_guard.py +++ b/tests/test_query_guard.py @@ -48,6 +48,26 @@ def test_each_rule_and_a_plain_public_question(self): self.assertEqual(guard.judge("x" * 600, max_chars=700), "pass", "the threshold is the host's policy") self.assertEqual(guard.judge(CODE, max_lines=5), "pass") + def test_a_credential_after_a_word_ending_in_a_prefix_is_still_a_credential(self): + """hunt-fixes-1, INT-1: on dc69499491 ``task-ghp_...`` passed. The regex found ``sk-`` inside ``task-``, the + word-start rule rejected that match, and the scan resumed after the whole token, so the ``ghp_`` behind the + hyphen was never seen (likewise ``disk-sk-ant-``, ``see ask-AKIA``); ``key_sk-ant-`` failed on the ``_``. A + prefix now counts wherever it is not glued to a preceding letter or digit, and a rejected start resumes one + character on (``_patterns.credential_prefixes``). Ordinary words ending in ``sk`` are still not keys.""" + hidden = [("task-ghp_" + "Z" * 30, "ghp_"), ("disk-sk-ant-api03-" + "a" * 24, "sk-ant-"), + ("see ask-AKIA" + "Z" * 16, "AKIA"), ("key_sk-ant-api03-" + "a" * 24, "sk-ant-"), + ("ghp_" + "Z" * 30, "ghp_"), ("token=xoxb-" + "1" * 12, "xoxb-")] + for text, prefix in hidden: + with self.subTest(text=text[:20]): + self.assertEqual(guard.judge(text), "secret") + self.assertEqual(_patterns.credential_prefixes(text), [prefix], "the prefix, never the key itself") + for text in ("risk-assessment library for python", "task-list-manager for teams", "disk-cleanup-tool for windows", + "ask-anything-bot source", "the flask-sqlalchemy-utils package", "mask-generation-model weights", + "the task-pricing-run12345678 ask-ant-x"): + with self.subTest(text=text): + self.assertEqual(guard.judge(text), "pass") + self.assertEqual(_patterns.credential_prefixes(text), []) + def test_a_line_copied_from_a_tracked_file_is_blocked_by_its_digest(self): with tempfile.TemporaryDirectory() as folder: root = Path(folder) diff --git a/tests/test_server.py b/tests/test_server.py index f9fc374..9608ef3 100644 --- a/tests/test_server.py +++ b/tests/test_server.py @@ -146,6 +146,35 @@ def test_counts_follow_the_store(self): "effects_pending": 0, "effects_settled_applied_total": 1, "effects_settled_not_applied_total": 0, "queue_depth": 0}) +class Principals(ServiceCase): + def test_two_principals_with_one_display_name_are_two_actors(self): + """hunt-fixes-1, INT-3: on dc69499491 the server passed ``user["name"]`` as the actor, so two principals named + ``bot`` were one actor to the kernel: the second could commit the first's proposal (``commit`` compares + actors) and could not give an independent approval. The actor is the principal id now (``actor_of``); + ``/api/me`` keeps the display name.""" + self.service.install() + for number in ("1", "2"): + self.service.work.register_principal(f"bot-{number}", "agent", "bot", "acme", ["operator"], + token=f"bot-token-{number}") + status, _, me = self.service.call("/api/me", token="bot-token-2") + self.assertEqual((status, me["name"], me["principal_id"]), (200, "bot", "bot-2")) + status, _, created = self.service.call("/api/cases", {"contract_id": "note", "version": 1, "inputs": {"name": "n"}}, + token="bot-token-1") + self.assertEqual(status, 200, created) + case = created["id"] + status, _, proposed = self.service.call(f"/api/cases/{case}/proposals", {"action": "note", "args": {}}, + token="bot-token-1") + self.assertEqual((status, proposed["decision"]["status"]), (200, "allowed"), proposed) + self.assertEqual(proposed["proposal"]["actor"], "bot-1", "the recorded actor is the principal id, not the name") + proposal = proposed["proposal"]["id"] + status, _, refused = self.service.call(f"/api/cases/{case}/commit", {"proposal_id": proposal}, token="bot-token-2") + self.assertEqual((status, refused["error"]), (409, "actor_mismatch"), refused) + status, _, committed = self.service.call(f"/api/cases/{case}/commit", {"proposal_id": proposal}, token="bot-token-1") + self.assertEqual((status, committed["status"]), (200, "succeeded"), committed) + actors = {e["body"]["actor"] for e in self.service.work.inspect(case)["events"] if "actor" in e["body"]} + self.assertEqual(actors, {"bot-1"}, "no event records the display name as its actor") + + class WorkerMetrics(ServiceCase): effects = "worker"