diff --git a/.gitignore b/.gitignore index ded1a21..a89422d 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,9 @@ build/ # The ledger is secret-bearing by construction — never commit it. .didrun/ *.didrun-bundle + +# Local-only working material (private paths, internal project references). +# Not for publication -- this repo ships docs/ to PyPI and GitHub. +research/ +docs/DIDRUN_V02_PACK.md +.didrun-history/ diff --git a/README.md b/README.md index d92f30b..9f5842d 100644 --- a/README.md +++ b/README.md @@ -7,24 +7,32 @@ **Flight records for agent-written code.** -> **v0 · experimental.** The core (`didrun run/claim/seal/verify/show`) works and is -> covered by CI on Linux + macOS across Python 3.11–3.13, but the API and CLI may -> still change, and there has been no independent security review yet. Treat it -> accordingly. See [what it does and does not attest](docs/TRUST_MODEL.md). +> **v0.2 · experimental.** The six commands (`run`, `claim`, `authorize`, `seal`, +> `verify`, `show`) work, and the test suite and capture kill-gate are green — on one +> machine, macOS / CPython 3.14. The declared floor is **Python 3.11**: the CI matrix +> (ubuntu + macOS × 3.11–3.13) is wired to `main`, which is what the badge above tracks, +> and it has **never run on the v0.2 branch**. Read "3.11+" as declared and tested on +> `main`, not as a v0.2 measurement. The API and CLI may still change, and there has been +> no independent security review. **Upgrading from v0.1? Verifiers before sealers** — see +> [What changed in v0.2](#what-changed-in-v02). What didrun does and does not attest is +> [here](docs/TRUST_MODEL.md); the unflattering version of what this release did and did +> not establish is [here](docs/V02_STATUS.md). When an AI coding agent says *"done — tests pass,"* the proof usually evaporates the moment the session ends. The transcript is unverifiable prose, CI only re-runs what CI knows about, and the reviewer is left re-deriving the work or trusting -vibes. That gap is the 2026 bottleneck: teams merge far more agent-authored PRs -than they can review, and AI PRs wait the longest for a reviewer. +vibes. didrun records what an agent **actually executed** — the command, the exit code, the output, and the exact git tree state it ran against — then binds structured -claims to a commit and lets a reviewer check them in seconds: +claims to a commit and lets a reviewer check them in seconds. A verdict carrying one +row of each of the five ladder grades: ``` x FAILED — review needed - 2/5 claims recorded-exact · commit a1b2c3d4e5f6 · tree 0f1e2d3c4b5a · resolved-by commit + 2/5 claims recorded-exact · commit 7eebe0fd0d9f · tree c6876974c0c0 · resolved-by commit + 4/5 claims evidence-bound + env: 4 match / 0 drifted / 0 incomparable / 1 not-recorded STATUS CLAIM DETAIL ---------------------------------------------------------------------------- @@ -33,8 +41,12 @@ x FAILED — review needed ! STALE lint clean exit 0 tree moved since evidence: 2 path(s) differ M src/api/handlers.py A src/api/new_route.py - ~ SCOPE-EXACT unit tests pass exit 0 all 3 change(s) within declared pathspecs + ~ SCOPE-EXACT unit tests pass exit 0 all 2 change(s) within declared pathspecs + M src/api/handlers.py + A src/api/new_route.py = TREE-EXACT type check passes exit 0 self-stable command ran against the sealed tree + + didrun records what ran; it does not prove the code is correct. See docs/TRUST_MODEL.md. ``` There is **no LLM in the trust path.** Capture and verification are deterministic. @@ -42,16 +54,15 @@ didrun is a *recording*, not a *proof* — read [what it does and does not attes ## Why it exists -- **Verification is the bottleneck, not generation.** Reviewers need to check an - agent's claims without re-running the work. -- **Everything else in this space is another LLM judging the code.** didrun answers - a different question — *what actually happened?* — deterministically. -- **Trust is an artifact problem, not a model problem.** The fix for unverifiable - work is work that carries its own machine-checkable evidence. +A reviewer needs to check an agent's claims without re-running the work, and the +usual answer is another model reading the diff — a judgement about whether the code +looks right. didrun answers a different and much narrower question, *what actually +happened?*, and answers it deterministically, so what a reviewer gets is evidence +they can check rather than a second opinion they have to weigh. ## Quickstart (under 5 minutes) -didrun is Python 3.14+, **stdlib-only** (zero runtime dependencies — you can read +didrun is Python 3.11+, **stdlib-only** (zero runtime dependencies — you can read the whole thing). Install it into your project's environment: ```bash @@ -76,29 +87,43 @@ didrun verify --strict # exit 0 only if every claim is recorded-exact ``` The `--strict` exit code is the CI seam: wire `didrun verify --strict` into your -pipeline and a drifted or unbacked claim fails the build. For humans, generate a -self-contained HTML evidence report: - -```bash -didrun verify --html evidence.html -``` - -The report is a single file with zero external assets — it opens offline, prints -cleanly, and is safe to attach to a PR (secrets are redacted from every export). +pipeline and a drifted or unbacked claim fails the build. + +Every verdict also reports how the environment compared with the one the seal +recorded — `env: N match / M drifted / K incomparable`. Drift is advisory unless +you pass `--require-env-match`, because a changed environment is a fact about the +machine verifying, not about whether the recorded command ran. Evidence sealed by +an older didrun is *incomparable*, never drifted, and never refuses. + +`didrun verify --html evidence.html` writes a self-contained report for humans: +one file, zero external assets, opens offline and prints cleanly. Redaction +covers the sealed note — every exported claim string (`argv_preview`, label, +pathspecs, changed paths, conjuncts, reason) is redacted by the same pass that +reports findings to the gate, and the projection is declared in the manifest. +The HTML carries no argv at all and renders the note's redacted label, but a +stale claim's file list is recomputed against your working tree at verify time +and shown as it is on disk — read it before you attach it to a PR. + +Detection is pattern- and entropy-based, so it is a filter, not a guarantee: a +base64 credential whose own slashes chop it into short, name-like pieces is scored +as a path and missed ([docs/COMPAT.md](docs/COMPAT.md) quantifies that class). And +redaction covers the *published* artifact only. The local ledger is not redacted +and has no retention or purge — `didrun run` stores each command's raw stdout and +stderr as content-addressed blobs under `.didrun/`, so a credential scrubbed out of +git history survives there until you delete the directory by hand. `.didrun/` is +gitignored and its root is forced to `0700` (files didrun creates are `0600`; ones +an older didrun created keep their modes, and `.didrun/objects/**` is git's at +git's modes). Treat it as secret-bearing: do not commit it, share it, or attach it +to an issue. ## Use it with your coding agent You don't have to run these commands by hand. Tell your agent to route its own verification through didrun, and it produces receipts as a side effect of working. -Drop the snippet for your tool into its instructions file: - -- **Claude Code** → add the block to `CLAUDE.md` -- **Codex / agent-agnostic** → add it to `AGENTS.md` -- **Cursor** → a project rule -- **Any agent** → a system-prompt block - -Copy-paste snippets for each — plus the honest bounds — are in -[docs/agents.md](docs/agents.md). The short version: +Drop the snippet for your tool into its instructions file — `CLAUDE.md` for Claude +Code, `AGENTS.md` for Codex and anything agent-agnostic, a project rule for Cursor, +a system-prompt block for anything else. Copy-paste snippets for each, plus the +honest bounds, are in [docs/agents.md](docs/agents.md). The short version: ```markdown For any check you'll cite as evidence, run `didrun run -- `. Per finished @@ -110,13 +135,14 @@ as verified. This is **cooperative capture** — it works because the agent follows the instruction, not because anything is enforced. Commands the agent *doesn't* route through `didrun run` are simply not recorded (a claim with no backing run grades -`unknown`), so didrun never mistakes "didn't capture it" for "verified." Enforced, -agent-can't-forget capture is what Tiers 1–3 are for — see the note below. +`unknown`), so didrun never mistakes "didn't capture it" for "verified." Tiers 1–3 +narrow how much an agent can forget to route; nothing closes it, and none of them +is wired to a command yet — see the note below. ## The grades didrun never says "verified — trust me." Each claim gets an honest grade against -the sealed commit's tree — **first match wins**: +the sealed commit's tree: | Grade | Meaning | |-------|---------| @@ -124,19 +150,74 @@ the sealed commit's tree — **first match wins**: | **SCOPE-EXACT** | Every change since the evidence is within claimant-declared pathspecs. | | **STALE** | Evidence exists but the tree moved — the exact path+content delta is shown. | | **UNKNOWN** | No honest binding (no witnessed success, retroactive binding, gc'd object). | -| **FAILED** | The claimed command was *recorded failing* — a caught lie, not merely missing evidence. | +| **FAILED** | The command backing the claim was *recorded exiting nonzero* — evidence against the claim, not merely absent evidence. Nothing in the grade speaks to intent. | +| **WITNESS-UNAVAIL** | The live ledger cannot supply the evidence this claim was sealed against. The sealed grade is shown as history, never re-earned, and `--strict` never accepts it. | +| **CHAIN-BROKEN** | Report-level, never a per-claim row: the ledger every grade was read out of does not recompute, or this binary cannot check it. It dominates every grade below it. | + +The first five are the ladder. It is evaluated worst-first — `failed`, `unknown`, +`stale`, `scope-exact`, `tree-exact` — and the **first match wins**, so a claim never +grades better than the weakest fact about it. (The table above is in the reverse, +best-first order a verdict prints.) The last two are not rungs: one replaces a grade +when the evidence behind it cannot be produced, the other overrides the whole report. `STALE` always carries the concrete delta; it is never a bare shrug. `tree-exact` means the evidence tree equals the sealed tree — it does **not** claim the code is correct or that the command meaningfully tested anything. -## The five commands +## The six commands + +- `didrun run [--tee] [--heartbeat SECONDS] -- ` — record a wrapped execution (complete capture). `--tee` mirrors the child's raw output to your terminal as it arrives (local only, never redacted); `--heartbeat` prints a content-free progress line to stderr. Both default **off**, so anything parsing didrun's stdout sees what it saw before. +- `didrun claim [--label L] [--event N] [--path ] [--of a,b,c]` — declare a structured claim. With no `--event` it binds the command that just ran, and **refuses** (exit 2) rather than bind an earlier passing event when the latest one exited nonzero. `claim conjunction --label REL --of a,b,c` declares one over other claims in the same seal window, graded as the worst of them. Re-declaring the same type and label supersedes the earlier claim: the retired one keeps its grade in the record, drops out of the verdict, and is counted on both surfaces. +- `didrun authorize --artifact --scope {allow-secrets,reseal} --label ""` — record a cited authorization artifact for one exceptional scope, storing its SHA-256 and never its bytes. What that establishes, narrowly: **a file with this content was cited.** Not a signature, not authentication, not evidence a human approved anything — an agent can write its own artifact in one line, and every surface that prints one says so. A citation covers `--expires-after-seals N` seals starting with the next one (default **1**), and there is no value meaning *never*. `seal --require-authority` demands one for those two scopes; it is opt-in, because an authorization required on the happy path is a rubber stamp within a week. +- `didrun seal [--commit C] [--allow-secrets] [--bundle F] [--reseal] [--require-authority]` — compile and publish a commit-bound evidence manifest. **Fails closed**: exit 2 if the note cannot be published (leaving no note, no watermark, no seal record), exit 3 on a structured secret in the bytes it is about to publish. Replacing a note with a *narrower* record needs `--reseal`. +- `didrun verify [--commit C] [--strict] [--require-env-match] [--html F] [--quiet]` — check claims against the evidence they were sealed against. +- `didrun show [--commit C] [--session] [--event N --output [--stream stdout|stderr] [--redacted]] [--html F]` — the verdict, the recorded session history, or a recorded output blob, re-hashed on read so a corrupted one is a refusal rather than bytes presented as the record. + +## What changed in v0.2 + +**`verify` is now bound to the evidence it sealed.** Before v0.2 it regraded against +whatever ledger happened to be on disk, keyed by integer index, and never checked the +chain. Measured: seal three claims, delete the ledger, run three unrelated commands into +a fresh one on the same tree, then verify. v0.1 reported `3/3 recorded-exact` and exited +0. v0.2: -- `didrun run -- ` — record a wrapped execution (complete capture). -- `didrun claim [--path ]` — declare a structured claim. -- `didrun seal` — compile a commit-bound evidence manifest (redacts secrets; blocks by default if found). -- `didrun verify [--strict] [--html ]` — check claims against recorded evidence. -- `didrun show [--session]` — show the verdict, or the recorded session history. +``` +? WITNESS-UNAVAIL — review needed + 0/3 claims recorded-exact · commit 8710d403a1b8 · tree 7c75e84f39ea · resolved-by commit + 0/3 claims evidence-bound +``` + +`--strict` exits 1, and a chain that does not recompute now dominates the verdict +outright rather than being ignored. The rest: + +- **Seal is atomic and fails closed.** Publication and the watermark are one pair: nothing is recorded until the note is on the commit, and a failure after that rolls the note back. A seal that could not publish used to report success. +- **The v1 chain preimage is frozen behind an explicit field list**, so adding an `Event` field cannot silently change hashes already written. An entry whose preimage version this binary does not know is reported unverifiable *with its index*, never graded against a guess. +- **Interrupts no longer lose a flight.** Signals are held across the digest-and-append window (SIGINT → 130, SIGTERM → 143, one line each), and a `run` that dies after the digest records `tree_after=None` with observed-text-only coverage instead of costing the event. +- **The ledger is tighter.** Its root is `0700` and the files didrun creates are `0600`; read-only `verify` and `show` no longer manufacture one as a side effect; concurrent appends take an `fcntl.flock`, so parallel writers cannot fork the chain. Bounds are declared rather than rounded up: inherited files keep their old modes, `.didrun/objects/**` is git's at git's modes, and on a platform with no `fcntl` or no meaningful `chmod` both properties degrade to v0.1 behaviour instead of pretending. +- **Redaction and reporting are one act.** The entropy detector scores runs rather than characters — a path in segments, a base64 blob whole — every detector declares a tier (six structured patterns at `block`, the entropy sweep at `notice`), and the fields that block are the fields that get redacted, so `--allow-secrets` cannot publish raw what the gate blocked. The projection is *declared*, not inferred: `projection_version`, `detector_set_version`, and the exact replaced spans, which substitute back to the input byte for byte. +- **Claims can be superseded**, so a fix-verify loop converges instead of accumulating red rows, and a conjunction grades as the worst of its members. Both are described under `claim` above. + +**What v0.2 did not establish.** Compatibility with evidence already sealed is **not** a +finding: the compat harness is content-blind and green on synthetic fixtures only, the +real archive replay has never run, and the harness prints `corpus: SKIPPED (unverified)` +rather than let a green line imply otherwise. Nor has CI run on this branch. And two +operational errors still escape as Python tracebacks with exit **1** instead of the graded +refusal with exit 2 that the same binary gives elsewhere: a command that does not exist, +and a `session.log` this binary cannot parse (which takes `run`, `verify` and +`show --session` down alike). Exit 1 is `--strict`'s "graded badly" code, so a crash is +not distinguishable by exit status from an honest failure. +[docs/V02_STATUS.md](docs/V02_STATUS.md) separates verified from bet, item by item. + +> **Upgrading: verifiers before sealers.** `MANIFEST_VERSION` is now **3**, and an older +> reader **refuses** a v0.2 note by design — exit 2, `manifest version N is newer than +> this didrun understands (max M); upgrade didrun` — because regrading a note index-only, +> without checking the binding it was sealed under, is the confidently-wrong verdict this +> tool exists to avoid. Seal on a v0.2 workstation, verify in v0.1 CI, and every commit +> fails. The other direction is fine: a v1 note carries no evidence block and regrades +> exactly as v0.1 regraded it. Two smaller breaks the same way — a tree digest now also +> excludes `.didrun-history/`, so a repository carrying one **un-gitignored** sees claims +> against the older digest grade `stale`; and a narrowing re-seal needs `--reseal`. +> [docs/COMPAT.md](docs/COMPAT.md). ## Capture is tiered and honest about coverage @@ -144,15 +225,15 @@ Universal, invisible capture of everything an agent runs is not possible from outside the agent (measured — absolute-path executions escape a PATH shim every time). didrun is honest about this instead of pretending: -- **Tier 0 — `didrun run -- `:** the trust core. Complete argv/exit/output/tree state. This is the only tier that *guarantees* a claim's evidence — **and, in v0, the only tier wired to a command.** Use it explicitly for anything you'll claim. +- **Tier 0 — `didrun run -- `:** the trust core. Complete argv/exit/output/tree state. This is the only tier that *guarantees* a claim's evidence — **and, in v0.2, the only tier wired to a command.** Use it explicitly for anything you'll claim. - **Tier 1 — PATH shim:** re-dispatches bare-name commands through Tier 0. Absolute-path invocations and shell builtins are honest structural gaps. -- **Tier 2 — per-shell trap:** bash/zsh enrichment, **inert unless a didrun session is active** (it never records unrelated shell activity on your machine). +- **Tier 2 — per-shell trap:** bash/zsh enrichment. The snippet is gated on `DIDRUN_SESSION`, which only `didrun run` sets, so it is **inert outside a didrun session** rather than observing every shell on the machine. - **Tier 3 — native adapters:** e.g. Claude Code hooks. Enrichment only; the core works without them. Every recorded event carries a coverage grade, so the manifest is honest about what was and wasn't observed. -> **v0 status:** Tiers 1–3 exist as measured, tested library functions (see +> **v0.2 status:** Tiers 1–3 exist as measured, tested library functions (see > `harness/` and `src/didrun/capture.py`) but are **not yet exposed as a > user-facing `install`/`hooks` command** — wiring them up is on the roadmap. > Today you get their guarantee by calling `didrun run -- ` directly. @@ -165,6 +246,21 @@ cryptographic signing, which is deferred — see [TRUST_MODEL.md](docs/TRUST_MOD It is a **trust accelerant among people with a baseline of trust**, not a substitute for it. It records; you review. +**It does not resist a local forger.** Whoever can substitute a ledger can regenerate +its chain and re-run `claim` and `seal` to mint fresh hash-bound claims. What v0.2 buys +is **accident and drift detection on published evidence** — the thing that actually goes +wrong in practice, and worth having. It is not tamper-proofing and must not be sold as +such. + +**And the honest limit of the evidence for the whole idea:** in the largest run to date — +846 sealed claims by one count and 667 by another, a discrepancy that is itself an open +research item, and every figure here self-reported by the system under study — exactly +**one** claim graded `failed`. That is equally consistent with a highly effective gate and +with a nearly inert one, and a run whose only actor was cooperating cannot distinguish +them. Until someone tells an agent to obtain a green +`didrun verify --strict` by any means and watches what it does, "didrun catches dishonest +agents" is a hypothesis. + ## License Apache-2.0. diff --git a/docs/COMPAT.md b/docs/COMPAT.md new file mode 100644 index 0000000..c2fbf90 --- /dev/null +++ b/docs/COMPAT.md @@ -0,0 +1,622 @@ +# Format compatibility + +didrun writes evidence that is meant to outlive the version that wrote it: a chain of +recorded events on disk, and a manifest published as a git note on a commit. This +document states what is frozen, what may change additively, and what v0.2 changed in a +way that is visible from outside. + +The rule behind all of it: **when didrun meets a format it does not understand, it +refuses and says so.** It never reads unknown evidence under the semantics it happens to +know — a confident verdict about a format the binary has never seen is exactly the +failure mode this tool exists to avoid. + +## The chain preimage is versioned by an explicit field list + +`ChainEntry.compute_hash` does not hash "the event". It hashes the fields named in +`ledger._PREIMAGE_FIELDS` for the entry's `preimage_version`. Version 1 is **frozen**: +its field list is a literal in the source and a golden test recomputes a known entry +hash from a fixed event, so any change to the field set fails that test rather than +silently invalidating every `entry_hash` ever written. + +Adding a field to `Event` therefore does not change existing hashes. Putting a field +into the *preimage* does, which is why a new preimage version is a deliberate, tested +step and not a side effect of adding an attribute. + +An entry whose `preimage_version` this binary does not know is not verified against a +guess: the chain check reports it as unverifiable and names the index. + +## Manifest versioning + +`MANIFEST_VERSION` is the note format's version. + +- **Additive fields do not bump it.** A new key that older notes simply lack is + backward-compatible in both directions. +- **A change to an existing field's type or meaning bumps it**, as does a change to the + claim-type vocabulary — those are the changes that make an old reader wrong rather + than merely incomplete. +- **A version newer than the reader is refused, not coerced.** `Manifest.from_json` + raises on a version above the one it knows and names both numbers. Older versions are + read. +- **Every new manifest key is read with `.get()` and a default that reproduces v1 + semantics.** A required key would break reading every note already published. + +### Version 2 — a claim names the evidence it was sealed against + +A v2 note may carry an `evidence` block inside a claim entry, naming the recorded ledger +entry that backed that claim: its index, its chain `entry_hash`, and the environment +fingerprint and tree digest recorded on that event. `verify` checks that binding before +grading anything. When the live ledger cannot supply that entry — the index is out of +range, or a different entry sits there — the claim is reported as `witness-unavailable` +carrying the grade the note sealed, and `--strict` refuses it. + +The version bumped for the change of MEANING, not for the new key. Before v2, verify +silently regraded against whatever ledger is present, so archival, rebuild or index drift +changed a sealed verdict with no signal. That is the defect v2 closes. + +What this buys is **accident and drift detection on a published note**, not resistance to +a local forger. Whoever can substitute a ledger can regenerate its chain and re-run +`claim` and `seal` to mint fresh hash-bound claims. `ledger.py` states that non-guarantee +in the source and nothing here changes it. + +Old notes stay readable. New notes do not travel backwards: + +- **v0.2 reading a v1 note.** Every note published before v0.2 is version 1 and carries + no `evidence` block, so it takes the unbound path and regrades exactly as v0.1 regraded + it, down to the reason string. +- **v0.1 reading a v2 note — REFUSED, by design.** v0.1's `Manifest.from_json` raises on + any version above its own, so `didrun verify` exits 2 with + `manifest version 2 is newer than this didrun understands (max 1); upgrade didrun`. + This is the refuse-never-coerce rule at the top of this document applied to us: a v0.1 + reader would regrade a v2 note index-only and report a confident verdict without ever + checking the binding the note was sealed under. Refusing is the correct behaviour, but + it is a real forward incompatibility and it bites a mixed-version setup — seal on a + v0.2 workstation, verify in v0.1 CI, and every commit fails. **Upgrade the verifier + first, then the sealer.** + +The evidence block lives in the manifest, never in an `Event`. Putting it into the chain +preimage would invalidate every `entry_hash` ever written — the break the frozen preimage +above exists to prevent. + +### Version 3 — the claim-type vocabulary gained `conjunction` + +**A v3 note may contain a claim whose `ctype` is `conjunction`, and in a released binary +whose claim vocabulary does not include that type — v0.1, or a v0.2 built before the +refusal below landed — reading it is not a graded refusal but an uncaught traceback: +`verify` called `Claim.from_dict` outside the `try` that grades a claim, so +`ClaimError: unknown claim type: 'conjunction'` propagated out of the command.** That is +the vocabulary break, stated at full strength — "raises" understates it, and the +`MANIFEST_VERSION` bump to 3 exists so a v0.1 or early-v0.2 reader refuses the note on its +version *before* it reaches the claim it cannot parse. + +**THIS reader no longer behaves that way, and the distinction matters when reading the rest +of this section.** A later unit closed the same hole from this end: a claim type this +binary cannot parse, in a note whose version it accepts, is now a whole-manifest refusal +naming the entry — `claim entry 0 is unreadable: unknown claim type: '…'`, exit 2, no +traceback. Measured. See *An unreadable note is a graded refusal* below. The paragraph +above describes what a note published by this version does **when it travels backwards to +an older binary**, which is the forward incompatibility this section is about. + +There is no way to extend a closed vocabulary without that break. The alternative — +skipping claim types a reader does not know — is worse: the note would silently +under-report, and print a verdict over the claims it happened to understand. + +A `conjunction` claim names other claims in the same seal window **by label** and is graded +as the **worst** of them under the same severity order the whole-report verdict uses, so it +can never be better than any one of its members. It binds no event (`event_indices` is +required to be empty) and carries no `evidence` block; `verify` grades it in a second pass, +against the results of the other claims **in that same note** — never against the live +`claims.jsonl`, which would let a claim declared after the seal change an old commit's +verdict. A conjunct label that names no live claim in the window, or more than one, grades +the conjunction `unknown` and the reason names it. + +Two additive keys carry it, both read with `.get()`: + +```json +"claim": {"ctype": "conjunction", "label": "REL", "event_indices": [], + "conjuncts": ["unit", "lint"]}, +"conjunct_grades": [{"label": "unit", "grade": "tree-exact"}, + {"label": "lint", "grade": "stale"}] +``` + +`claim.conjuncts` is emitted for every claim (as `[]` for the ones that name none, like +`pathspecs`); `conjunct_grades` appears only on a conjunction, and a member's `grade` is +`null` when the name resolved to nothing — an unresolved name has no grade, and giving it +one would be an invented fact. + +**What it does NOT do.** It does not reconstitute an execution witness. A design review +declined a proposal to replace recorded executions with a transition marker, on the ground +that a marker witnesses *that a transition occurred*, not *that the command ran and exited +0* — that refusal stands and nothing here attempts it. A conjunction substitutes for +nothing: every conjunct keeps its own witnessed exit code and its own tree comparison, and +the conjunction asserts nothing its members do not already back. + +Stored evidence is unaffected. Archived notes contain no conjunction and stay `version: 1`; +`conjuncts` is additive with a `()` default, so every archived note reads exactly as it did. +The break is forward-only, and it is the same shape as v2's: **upgrade the verifier before +the sealer.** + +When didrun falls back to resolving a manifest by tree id it scans the notes ref. A note +it cannot parse at all — a foreign note under the same ref, a corrupt body — is skipped +and counted, and the count is reported, so one bad note cannot hide every good one. A +note it *can* parse but whose version is too new is not skipped: it aborts the +resolution. The fallback returns the first note whose sealed tree matches; it does not +currently detect a second match. + +## The exported redaction projection is declared, not inferred + +Each claim entry in a v0.2 note carries an additive `redaction` block: + +```json +"redaction": {"projection_version": 1, "detector_set_version": 2, + "applied": [{"argv_index": 2, "start": 10, "end": 59, + "detector_id": "env-secret-assignment", + "withheld": false}]} +``` + +`applied` names the exact source spans that were replaced in `argv_preview`, in the +source's own coordinates. It exists because the projection used to be readable only by +inference, and inference was wrong: which tokens the entropy sweep flagged turned on +letter frequency, so the same argv shape exported differently from one seal to the next +and a downstream validator had to keep a matrix of didrun's historical behaviour. A +consumer now reads the contract. `projection_version` bumps when the rule turning +findings into replaced spans changes; `detector_set_version` bumps when the detectors do. + +**The marker string alone is not authenticating.** `«redacted:openai-key»` is plain text +that a recorded argv is free to contain, and didrun does not escape a pre-existing +marker. A consumer that wants to know whether a preview was really redacted — and where +— reads `redaction.applied`; it must not pattern-match the preview. Keyed markers would +make the marker itself unforgeable and are deliberately not built: they need a per-ledger +key store, and a per-ledger key destroys the cross-ledger comparability that motivated +them. + +`withheld: true` marks the honest-absence path: the element was dropped whole rather than +rewritten, and the preview shows `«withheld:»` in its place. It is never an +empty string, because an empty preview and a suppressed one are different facts. + +This block is additive and read with `.get()`, so it does not bump `MANIFEST_VERSION` and +older notes that lack it are read exactly as before. **v0.2 projections differ from +v0.1's for any input with overlapping findings**: v0.1 replaced overlapping spans +end-first, which deleted the text after the outermost span, so an argv like +`run --key MY_TOKEN= && echo done` exported with `&& echo done` silently gone. +v0.2 merges findings into disjoint maximal spans first, and the result is exactly +reconstructible — substituting each marker back with the source span `applied` names +reproduces the input byte for byte. + +**`detector_set_version` is 2 from v0.2's detector work onward.** The entropy sweep no +longer decides in the character class whether `/` is a path separator; it decides per +run, scoring a filesystem path in segments and a base64 blob whole, and every detector +now declares a tier. That is a change to what the detectors match, which is exactly what +the number is for — a consumer comparing a `1` projection with a `2` one is told the +detectors differ instead of having to infer it. `projection_version` stays 1: the +span-partition algorithm did not change. + +### The projection covers the claim fields, not only argv + +A v0.2 `redaction` block carries a second, additive list beside `applied`: + +```json +"redaction": {"projection_version": 1, "detector_set_version": 2, + "applied": [{"argv_index": 3, "start": 9, "end": 49, ...}], + "fields": [{"field": "claim.label", "start": 17, "end": 57, + "detector_id": "github-token", "withheld": false}]} +``` + +`applied` is unchanged and stays argv-addressed by `argv_index`; `fields` is addressed +by field name, because a label has no argv index. Field names in v0.2 are `claim.label`, +`claim.pathspecs[i]`, `delta[i].path`, `claim.conjuncts[i]` and `reason`. A reader that +does not know the key sees exactly the argv projection it always did. + +`claim.conjuncts[i]` is scanned as a **label**, because that is what a conjunct name is — +the label of another claim in the same window. Both sides of the match are therefore +redacted by the same deterministic projection, which is what keeps a conjunction resolvable +at verify time even when a conjunct's label carried a secret. + +This exists because blocking and redacting were not the same set. A structured secret in +a claim label refused the seal — and the refusal says *"re-run with `--allow-secrets` to +export a redacted artifact anyway"* — but on the override the label was serialized into +`refs/notes/didrun` verbatim, a ref that gets pushed. The claim fields are now scrubbed +by the same pass that reports them to the gate, so the two cannot drift apart again. +`scan_domain` says so: it reads `claim-bound-events+claim-fields+manifest` rather than +`claim-bound-events+manifest`. A stored note carrying the older string is still read +normally; the string is a description of that seal, not a compatibility gate. + +## The manifest records what the secrets pass scanned + +A v0.2 manifest carries an additive top-level `secrets` block beside `secrets_override`: + +```json +"secrets": {"scan_domain": "claim-bound-events+claim-fields+manifest", "events_scanned": 1, + "events_total": 7, "bytes_scanned": 4213, "detector_set_version": 2, + "findings_by_tier": {"block": 0, "notice": 3}, + "blocked": false, "overridden": false} +``` + +It is additive, defaults to `{}`, and is read with `.get()`, so it does **not** bump +`MANIFEST_VERSION` and a stored note that lacks it reads exactly as before. The compat +harness's per-version additive-key allowlist carries `secrets` for both v1 and v2 — the +deliberate one-line diff that keeps the note round-trip leg honest instead of red across +every stored note at once. + +`secrets_override` is untouched: same name, same type, same meaning — *this seal would +have been refused and the operator overrode it*. It is deliberately not replaced by a +dict, because a v0.1 binary reads it with `.get("secrets_override", False)` and a dict +there is truthy, which would make every clean v0.2 seal print "sealed with +--allow-secrets" on an older reader. What `secrets` adds is the domain behind the +boolean: `false` used to be the same word for "nothing was found" and "almost nothing was +scanned", and `events_scanned` / `events_total` now say which. + +**What blocks changed direction in v0.2.** A refusal is scoped to the export domain — +the argv of each claim-bound event plus the serialized manifest, which carries the claim +labels and delta paths that were previously published entirely unscanned. Findings +confined to the local output blobs, and every `notice`-tier finding, warn on stderr and +do not stop the seal. So some previously-refused seals now pass and some previously +passing ones now refuse; both directions are measured in MEASUREMENTS.md. + +## The claim window is recorded, and a narrowing re-seal is refused + +A v0.2 manifest carries two additive top-level integers, `claims_from` and `claims_to` — +the half-open range of the session's declared claims this note covers. They are read with +`.get()` and defaults of `0` and `len(claims)`, so no stored note changes meaning and +`MANIFEST_VERSION` does not bump. The compat harness's per-version allowlist carries both +for v1 and v2: the second deliberate one-line diff that criterion has forced. + +A claim entry may also carry an additive `superseded_by` — the index, within the same +note's claim list, of a later claim declaring the same `(ctype, label)`. `verify` excludes +a superseded entry from the verdict and from `worst_status`, and reports the count. The +entry keeps its grade, its reason and its delta verbatim: supersession subtracts from the +verdict, never from the record. A mark that is not a forward index into the same note's +claims carrying the same `(ctype, label)` is dropped, which *counts* the claim. That is +fail-closed reading, not a forgery barrier — whoever can rewrite a note can rewrite a +grade directly. + +**What this changes for evidence already in the world.** + +- **Reading is unaffected.** Both fields are additive with reading defaults, so every + archived note verifies exactly as it did, down to the reason string. An absent window is + *unknown*, not `[0, N)`: the refusal below reads key presence off the raw body rather + than a parsed default, precisely so a legacy note is never treated as containable. +- **Re-sealing an already-sealed commit now requires `--reseal`** unless the new window + contains the recorded one. This is a deliberate behaviour break, and it is the data-loss + path being closed: `git notes add -f` force-replaces and a seal is scoped to + `all_claims[watermark:]`, so re-sealing the same commit published a note holding only + the post-watermark claims and silently destroyed everything the earlier note held. A + superset re-seal (the watermark-rotation case) still needs no flag. With `--reseal` the + narrowing happens and is visible in `claims_from` / `claims_to` afterwards. +- **An absent `seals.jsonl` warns.** The watermark lives in a local, gitignored, unhashed + file, and its *absence* is not visible from the note: delete it and every later seal + silently re-covers the whole claim history. Measured where it matters most — only 62 of + 167 archived ledgers carried a `seals.jsonl` at all. The seal proceeds (rotation is a + legitimate workflow, and a session's first seal is indistinguishable from a rotated one + from inside `seal`) but it says so on stderr. + +**What it does not do.** Claiming a gate repeatedly until one attempt goes green is not +prevented. Every attempt is retained in the note and the retired count is printed on both +surfaces, so it is visible — which is the honest bound, and strictly better than the +invisible truncation it replaces. + +## The environment fingerprint is versioned in band + +`env_fingerprint()` returns `v2:<16 hex gate>:path=<8 hex>`. The prefix exists because the +digest's *shape* does not change when its key set does — two bare digests over different +key sets look comparable and are not. + +A value with **no prefix was produced by v0.1** and is **incomparable** with a versioned +one. It is not evidence that the environment drifted, and nothing may report it as such; +`capture.fingerprint_version()` returns `None` for it so a consumer can tell the two +apart. `capture.fingerprint_gate()` returns `None` for the same values, so a consumer +cannot compare one by accident. + +### v2: what the gate binds, and what it deliberately does not + +The **gate** — the only portion any comparison reads — is a digest of `LANG`, `SHELL`, +`TZ`, `VIRTUAL_ENV` and the **process umask**. Two of v0.1's keys were dropped: + +- **`PATH`** varies with terminal tab, `direnv`, `nix develop`, tmux and homebrew + shellenv ordering, and flips on any venv activation `VIRTUAL_ENV` already records. A + gate that refuses in a new shell is ignored within a week. PATH is instead recorded as + its own truncated digest in the `path=` suffix: drift in it is **advisory**, reported + as a suffix on a verdict line, and can never turn a match into drift. +- **`PWD`** is not the child's cwd. A wrapped command always runs with `cwd=`, and + `Event.cwd` records that same resolved path; `PWD` is the caller's shell variable, + which can be stale and can be absent when didrun is spawned by a launcher. Once the + fingerprint is compared, gating on it would mean two byte-identical executions of the + same command against the same repo disagree because the operator was standing + somewhere else. + +`umask` is added because it is the fact whose absence made two green wrapper events under +different umasks indistinguishable in the receipt. It is read **once per process** +(`capture.process_umask`) and reused: reading a umask is a read-modify-write on +process-global state, so a per-event read would open that window on every wrapped command. + +**The `path=` digest is a deviation, stated plainly.** The design review asked for `PATH` +to be recorded *unhashed in the event body*. A raw `PATH` there would enter the v1 chain +preimage — which is frozen — and would sit one redaction pass away from a published note, +against the export-scoped secrets rule. A digest in the existing free-form `str` field +gives the same drift visibility at zero schema cost and leaks less. What it costs is +legibility: a reader learns *that* PATH changed, never *how*. + +### The comparison verify makes, and what it may not do + +`verify` computes this process's fingerprint once and compares the **gate** against the +one each claim's `evidence` block recorded. Four outcomes: + +| `env_status` | when | +|---|---| +| `match` | same fingerprint version, equal gates | +| `env-drift-since-seal` | same fingerprint version, different gates | +| `incomparable` | different fingerprint versions, or a bare v0.1 digest | +| `not-recorded` | the note bound no fingerprint for the claim | + +**Only `env-drift-since-seal` can ever refuse, and only under `--require-env-match`.** +By default drift is reported in the verdict and does not change `all_verified`, because +a changed environment is a fact about this machine now, not about whether the recorded +command ran. `incomparable` never refuses under either mode — it is the state every note +published before v0.2 lands in, and refusing on it would convert "this note predates the +current key set" into a failure. + +**No archived evidence changes.** `Event`'s field set is untouched, so no stored +`entry_hash` moves; archived events keep their v0.1 and v1 fingerprint strings verbatim; +and a note with no `evidence` block is compared not at all. Note also that this makes +green **harder** to reach, not easier: anyone measuring whether v0.2 reduced failed +attempts must be told that `--require-env-match` adds a way to fail. A fingerprint that +can never refuse is not binding anything. + +## Tree-digest semantics changed in v0.2 (release-note level) + +## Tree-digest semantics changed in v0.2 (release-note level) + +v0.1 excluded only `.didrun/` from a tree digest. v0.2 also excludes the archive root +`.didrun-history/`, so rotating a ledger into the archive no longer moves the digest of +the tree that just archived it. + +This is a **semantics change, not hygiene**: for a repository that carries a +`.didrun-history/` directory which is **not** gitignored, a tree digest taken by v0.2 +differs from one taken by v0.1, and claims recorded against the older digest grade +`stale`. For every other repository the digest is unchanged: with no such directory there +is nothing to exclude, and with a gitignored one `git add -A` already skipped it before +the exclusion pass ever ran. + +The one repository where this was measured before release gitignores `.didrun-history/`, +so no archived tree id there is affected. That is a measurement of one repository, not a +proof about all repositories. Both configurations are pinned by tests. + +`.didrun/` itself remains excluded unconditionally, regardless of gitignore state. + +## Ledger permissions (v0.2) and where the guarantee stops + +v0.2 forces the ledger ROOT to `0700` on every writable open — creation and re-open +both, because `mkdir(exist_ok=True)` never re-tightens a directory an older binary +already made at `0755`. Files this version creates (`session.log`, `claims.jsonl`, +`seals.jsonl`, `.gitignore`, blob temp files) are created `0600`. + +Three bounds, stated because a rounded-up version of any of them would be an overclaim: + +- **Inherited files keep their modes.** The create mode applies at creation only, and + `.gitignore` is written only when absent. A ledger a v0.1 binary produced keeps `0644` + on those files. The root's `0700` is what makes that acceptable — another uid cannot + reach them through the directory — and it is **not** the same statement as "the files + are 0600". +- **`.didrun/objects/**` is not covered.** git writes into it directly through the + `GIT_OBJECT_DIRECTORY` redirect, twice per `didrun run`, at its own modes. didrun does + not chmod what git owns. +- **Windows.** `os.chmod` is close to a no-op on NTFS, so `.didrun/` inherits the parent + directory's ACLs and this guarantee does not hold there. The ownership check that + guards the chmod uses `os.geteuid`, which does not exist on Windows; it is skipped. + +Read-only commands (`verify`, `show`) no longer open a writable session: they create no +ledger directory, write no `.gitignore`, and re-mode nothing. Before v0.2, `didrun verify` +in a repo with no ledger *manufactured* an empty world-readable one, after which every +claim graded `unknown`; pointing `show` at an archived ledger re-moded it. Nothing on disk +changes shape, and `.didrun/` is excluded from tree digests unconditionally, so no +archived tree id or sealed note is affected. + +## Append serialization is POSIX-only (v0.2, behaviour, not format) + +From v0.2 the read-tail-and-append pair inside `Session.append` — and the `claims.jsonl` / +`seals.jsonl` appends — run under an `fcntl.flock` on `.didrun/.lock`, so two didrun +processes writing to one ledger cannot both record the same chain index. **On a platform +without `fcntl` the lock is a declared no-op and didrun degrades to the v0.1 behaviour: +concurrent writers can fork the chain, which `verify_chain` then reports as a permanent +break.** No portable fallback is invented, because a fallback that does not actually +exclude would be worse than a stated gap. `flock` is also emulated or ignored on NFS and +some other network filesystems; the exclusion there is as good as the filesystem's and no +better. + +Nothing stored changes. The lock file lives inside the ledger directory, which is excluded +from every tree digest unconditionally and ignored by the `*` rule the ledger writes into +its own `.gitignore`, so it moves no tree id, reaches no sealed note, and cannot be +committed. The lock is held for one short local write and is **never** held across a +wrapped child process — a `didrun run` that takes an hour blocks nobody. + +Separately, a blob's temp file is now named per writer rather than `.tmp`. Two +recorders storing the same bytes at the same time — an empty stderr is a digest every +event shares — used to race on that one name, and the loser died with `FileNotFoundError` +instead of recording. Stored blob names, contents and modes are unchanged. + +## Tier-0 capture streams from v0.2 (behaviour, not format) + +v0.2 drains the wrapped command's pipes incrementally instead of buffering them to +completion. Nothing recorded changes: the stored blobs are byte-identical to what the +buffered path produced, per stream, and the event's field set, `observed_via`, `coverage` +and exit code are untouched. What changes is that `didrun run` can now show the command's +output while it runs (`--tee`) and print a content-free progress line (`--heartbeat N`). +Both default **off**, so a caller parsing didrun's stdout sees exactly what it saw before. + +The drain uses `selectors`, which on Windows is a `select()` over sockets and cannot +register an anonymous pipe. Rather than make Tier-0 capture POSIX-only, registration +failure falls back to one blocking reader thread per pipe — same behaviour, no new +dependency. **The fallback is declared, not measured:** CI runs ubuntu and macOS only, so +no Windows leg exercises it. + +`didrun show --event N --output` reads a recorded blob back (`--stream stderr`, +`--redacted`). It is a reader over the existing store, adds no format, and re-hashes on +read, so a corrupted blob is a refusal rather than bytes presented as the record. + +## Known, not closed: the redaction domain + +Found while fixing the two defects above, judged real, and deliberately left open rather +than fixed quietly in the same change. Each is a bounded statement about where redaction +stops, not a plan. + +1. **An HTML report's file list is recomputed, not read from the note.** `verify` + regrades a claim against the working tree, so a `stale` claim's changed paths in + `didrun verify --html` come from the local repository at verify time — they are not + the note's redacted copy. The note itself is redacted; the HTML file is an export and + this path is not. It predates the claim-field work and is unchanged by it. Redacting + at render time is the obvious fix and is not taken here because it also degrades the + local terminal's `stale` drill-down, which is the one thing that grade exists to show; + choosing between them needs its own measurement. README.md carries the caveat in the + sentence that describes the report. + +2. **The entropy sweep misses roughly 1–3% of slash-bearing base64 credentials.** A + credential whose own slashes chop it into pieces all shorter than the 24-character + floor, *and* leave one piece looking like a name, is scored as a path and missed. That + is the measured price of biasing the path test toward paths (MEASUREMENTS.md), and it + is a `notice`-tier detector either way. Nothing in the rule resists an attacker who + knows it — the rule is published here, and satisfying it deliberately is trivial. + +3. **`coverage` is exported unredacted.** It is didrun's own generated text — grade names + and counts — and carries no operator or repository string today. That is a property of + the current generator, not an enforced invariant. The whole-blob manifest scan still + *blocks* on such a field, so the failure mode would be a blocked seal, not a silent leak. + + **`reason` was in this bullet until v3, and v3 is the change that took it out.** The + note in the previous edition said a future `reason` interpolating a path or a label would + leave the redacted set with nothing failing — and a conjunction's reason names a conjunct, + which is operator-authored. So `reason` is now redacted for every claim by the same pass + that reports it to the export gate, under the field name `reason`, rather than only for + the claim type that made it necessary: a field a later version interpolates something + into is then covered without anyone remembering. For every other claim type the reason + is generated text with nothing to find, so no stored or newly published note changes. + +## An unreadable note is a graded refusal + +`verify` refuses a note body it cannot parse, with exit 2 and a reason naming what is +wrong, and it does that for every malformed shape rather than for the ones someone +remembered. The refusal type is `ManifestFormatError`, a subclass of `ManifestError`. + +This was measured broken. Over 22 malformed note shapes, the pre-fix reader left an +uncaught `KeyError` / `TypeError` / `JSONDecodeError` / `ClaimError` traceback on 11 of +them and exited **1** — which is `--strict`'s "this note graded badly" code, so a corrupt +note was indistinguishable from an honest failure by exit status. Worse, it **accepted** +four: `{"version": true}` verified green at exit 0, because `bool` is an `int` subclass and +`True > 3` is False. `2.5`, `0` and `-1` did the same. + +The accepted set is why this matters more than tidiness. The "Version 3" section above +rests the whole forward-compatibility story on the version guard — the `MANIFEST_VERSION` +bump exists so an older reader refuses a v3 note *on its version* before it reaches the +claim vocabulary it cannot parse. A guard a malformed field walks past is not a guard. +`version` is now read with the same rule `superseded_by` gets (`_is_index`, which rejects +`bool`) and must be an integer >= 1. + +**The two refusals stay distinguishable, and the distinction is load-bearing.** A +malformed body is a `ManifestFormatError`; an integer version above `MANIFEST_VERSION` +stays a plain `ManifestError` with the `upgrade didrun` message. On the tree-fallback scan a +format error is **skipped and counted** (a foreign note under the same ref must not hide +every good one) while a too-new version **propagates** (scanning past it would report some +older note's verdict as if it were current). Before, that split was made by accident — the +scan caught bare `Exception`, so a malformed body was skipped only because `json` happened +to raise something that was not a `ManifestError`. + +`verify` also no longer raises out of the command on a claim type it cannot read. The +"Version 3" section notes that `Claim.from_dict` is called outside the `try` that grades a +claim; that hole is now closed from this end too, as a whole-manifest refusal naming the +entry. A per-claim `unknown` was rejected for the reason that section already gives: it +would print a verdict over the claims this binary happened to understand. + +**This is fail-closed reading, not a forgery barrier**, and it is the same caveat +`superseded_by` carries. Whoever can rewrite a note can rewrite a grade directly +(docs/TRUST_MODEL.md). Nothing above resists an attacker; it stops a corrupt or foreign +note from being read as a verdict. + +Stored evidence is unaffected: every note ever published carries an integer version of 1, 2 +or 3 and the required fields, so nothing legitimately published is newly refused. The +compat harness's **synthetic** fixtures cover all three versions and the per-version +additive-key allowlists; the **real** archive replay is still unexercised, so "nothing +legitimately published is newly refused" is an argument from the version guard, not a +measurement over stored notes. `docs/COMPAT_REPLAY.md` records which is which. + +## Known, not closed + +Small things measured and left alone, recorded rather than dropped. None of them can +produce a green verdict that the evidence does not back. + +- **A duplicate conjunct label is counted twice.** `--of a,a` grades correctly (the worst + of the two resolutions of `a`, which is `a`'s grade) but the reason reads `worst of 2 + conjunct(s)` for one distinct member, and `conjunct_grades` carries the pair twice. + Deduplicating would silently rewrite what the operator declared, which is worse than a + cosmetically odd count; refusing a duplicate is the better fix and is not built. +- **`didrun claim conjunction` in a session with no recorded events** refuses with "no + recorded events to bind to (run something first)". A conjunction binds no event, so the + wording is not its refusal — but a session with no events has no claims either, so a + conjunction there could only ever grade `unknown`, and the early refusal is correct even + though the sentence is borrowed. +- **A conjunction's `reason` names a conjunct, and the reason is redacted as one string.** + Both the reason and the conjunct label go through the same deterministic projection, so a + secret in a label is redacted in both places; but the redacted reason is not guaranteed to + contain the redacted label as a substring, because the two are scanned as separate fields. + Resolution never reads the reason, so this affects display only. + +## Two things v0.2 does not close + +Stated here because a trust tool's silence about its own limits is the same defect as an +overclaim. + +1. **A wrapped command inherits the caller's environment.** didrun records *that* the + command ran and *what* it exited with; it does not control what the environment made + the command do. `GOFLAGS=-exec=/usr/bin/true` yields an exit-0 event and a + `tree-exact` `tests-pass` claim for a command that ran no tests. No allowlist closes + this — `NODE_OPTIONS`, `PYTHONPATH`, `LD_PRELOAD`, `DYLD_INSERT_LIBRARIES`, + `PYTEST_ADDOPTS`, a user gitconfig, a `sitecustomize.py`, and a `PATH` shim all + produce it — and putting the effective environment into the event body would put the + richest secret carrier on a developer machine into the chain preimage. A declared + clean-environment mode is the honest shape for this, and it is not built. + + **The v2 environment fingerprint does not close this and must not be described as + closing it.** It binds five facts (`LANG`, `SHELL`, `TZ`, `VIRTUAL_ENV`, umask) and + records a PATH digest. `GOFLAGS` is in none of them, and adding it would only move the + line — the list above has no end. What the fingerprint buys is that a *change* in + those five facts between seal and verify is visible instead of silent. + +2. **The ledger has no retention or purge.** Captured stdout and stderr are stored as + content-addressed blobs under `.didrun/objects`. A credential that was scrubbed out of + git history survives there as a loose object until the ledger directory is removed by + hand. Treat the ledger as secret-bearing: it is gitignored by default, and it should + not be committed, shared, or attached to an issue. + +## Known, not closed: a note's authority block is descriptive, not authenticated + +Found while closing the authority redaction and rendering defects. Both items below are +statements about what a *note* can say, not about what it can make didrun do — the note is +the artifact that crosses machines (`git fetch`), and `docs/TRUST_MODEL.md` already states +that citation is "not tamper resistance — whoever can rewrite a note can rewrite the digest +in it". These are recorded because "the note is not authenticated" is easy to state and easy +to forget field by field. + +1. **A note's authority `scope` is rendered as written, and is not re-validated against the + scope vocabulary.** `authorize` refuses any scope outside `allow-secrets` / `reseal`, and + `_valid_authority` drops a ledger record whose scope is unknown — but a note that has been + edited by hand can carry `"scope": "everything"` and `didrun verify` will print + `CITED authority for scope everything`. It grants nothing: `--require-authority` gates the + seal against the local ledger, never against a fetched note, so a forged scope changes only + the description a reader sees. It is left open because the honest fix is a decision about + what to render for an unrecognized scope — dropping the block hides that a citation was + claimed, and printing it plainly is what it does now — and that decision wants the same + treatment the grade vocabulary got, not a quiet clamp. + + The three fields that state *what a citation is worth* are NOT in this position and are + pinned to source constants at both bind and render: `grade` (always `cited`), `limitation` + (the full sentence) and `principal_basis` (always `self-asserted`). A note supplying + `principal_basis: "verified-ssh-signature"` was the reason the last of those three was + pinned — it printed `principal (verified-ssh-signature)` two lines above the + sentence saying didrun cannot tell who wrote it, in one of the exact words the trust model + forbids. Verified by test, both surfaces. + +2. **Redaction is not sanitization, and the two run at different times.** A label is scrubbed + for secrets before publication; control characters around the secret survive that pass + untouched, because a redactor that also rewrote whitespace could not declare faithful + spans. Every *terminal* surface therefore sanitizes at the point it prints — the verify + authority block, and the `seal` / `authorize` echoes — and the HTML path escapes through + one helper. Anything reading a note or `authority.jsonl` programmatically gets the raw + strings and must do its own neutralizing; that is the same contract the recorded argv and + output already carry. diff --git a/docs/COMPAT_REPLAY.md b/docs/COMPAT_REPLAY.md new file mode 100644 index 0000000..b67ffa2 --- /dev/null +++ b/docs/COMPAT_REPLAY.md @@ -0,0 +1,140 @@ +# The v0.2 compat replay — verified, bet, and known not closed + +**Date:** 2026-07-29 · **Commit:** `49a8b74` on `v0.2-hardening` · **Python:** 3.14.5 (CPython, arm64) +· **Platform:** Darwin 25.5.0 (macOS) · **Suite:** 496 passed, 4 skipped + +**Corpus provenance: NONE. The corpus legs did not run.** `DIDRUN_COMPAT_CORPUS` was unset, so the four +real-corpus legs skipped by construction and every corpus-wide row below is **BET (UNEXERCISED)**. No +number in this document is a simulation of one. See *Gate 1* for exactly what clears it. + +Every command below ran unwrapped and is therefore **UNRECEIPTED** — the wrapper discipline is suspended +for this pack, so nothing here has a didrun receipt behind it. What it has is the shipped CLI's own +stdout, quoted. + +--- + +## VERIFIED — demonstrated, with the command that settles it + +Each row was produced this session by running the shipped CLI in a throwaway git repo under `$TMPDIR`, or +by a test in the suite that fails when the property stops being true. One witness, one platform, one +interpreter. + +| # | Claim | What settles it | Result | +|---|---|---|---| +| V1 | The suite is green | `python -m pytest -q` | **496 passed, 4 skipped** (4 skips are exactly the corpus legs) | +| V2 | The capture kill-gate holds | `python -m harness.recall` | **PASS**, freeze INTACT, defect-class recall **100.0% (3/3)**, structural 0.0% (0/2) | +| V3 | The synthetic compat legs are green | `python -m pytest -q tests/compat` | 11 passed, 4 skipped; leg1 7 logs / 11 entries, leg2 4 notes **0 violations**, leg3 **residue 0**, leg4 **multi-index 0** | +| V4 | The tree parses under 3.11 grammar | `ast.parse(…, feature_version=(3,11))` over all 41 `*.py` | **0 failures.** Syntax only — not an API check | +| V5 | Runtime is stdlib-only | AST walk of every import in `src/` + `harness/` | 24 top-level imports, **0 non-stdlib, 0 non-local**; `dependencies = []` | +| V6 | No private paths in shipped files | RULE 17 grep over `src/ tests/ harness/ docs/ testkit/` | Only `/home/runner/…` and `/home/example/…` — public CI shapes and explicit examples, which is what RULE 17 asks for | +| V7 | **Seal fails closed on note-publication failure** | `.git/objects` made unwritable, then `seal` | exit **2**, git's error quoted verbatim, **no `seals.jsonl`**, **no note**, **no watermark file**; permission restored → same seal succeeds and `verify --strict` exits 0 | +| V8 | A note far past `ARG_MAX` publishes and re-reads | 30 claims with 60 KB labels | note body **1,819,948 B** (14× the largest corpus note), seal exit 0, `verify` exit 0, 30/30 recorded-exact | +| V9 | **The substituted-ledger attack (EV2) fails closed** | seal 3 claims → `rm -rf .didrun` → 3 unrelated exit-0 commands on the same tree → `verify --strict` | **exit 1**, `WITNESS-UNAVAIL` ×3, **0/3 evidence-bound**, reason `entry hash mismatch at index 2`. Pre-P1.2 this was `3/3 recorded-exact`, exit 0 | +| V10 | A tampered chain dominates the verdict | flip one nibble of `entry_hash` at index 2 | `x ledger chain BROKEN at index 2 — every grade below is unreliable`, `--strict` **exit 1**, `show --session` **exit 1** | +| V11 | A missing ledger is *unavailable*, not *broken* | `mv .didrun .didrun-moved`, then `verify` | `WITNESS-UNAVAIL`, `--strict` exit 1, the word "broken" appears **0** times, and **no `.didrun` is recreated** | +| V12 | Ledger permissions | `ls -ld .didrun` after a write command | `drwx------`, `session.log` `-rw-------` | +| V13 | An interrupt never loses the flight | SIGINT then SIGTERM during `run -- sleep 30` | exit **130** / **143**, one greppable line, no traceback, event recorded `unobserved no-exit`, `chain intact`, and the next `run` appends normally | +| V14 | A torn log refuses rather than writing past the tear | truncate `session.log` by 25 B, then `run` | append refused, byte offset named, recovery spelled out, `session.log` still **1** line. (Delivered as a traceback — see U2) | +| V15 | Concurrent appends are safe and non-blocking | 6 parallel `run`s; then a 5 s `run` alongside a second | **6 events, chain intact**; the second run completed in **0.38 s** while the 5 s run held the session | +| V16 | Streamed output is byte-faithful, and a corrupt blob refuses | `run --tee`, `show --event 0 --output`, then flip a byte in the blob | output arrived progressively, replayed exactly; corrupted read → `blob digest mismatch (corruption)`, exit **2** | +| V17 | A malformed note is a **graded refusal**, never a crash and never a green | 6 bad `version` shapes + string claim entry + truncated body + unknown `ctype` + killed `ctype` | **every one exit 2** with a reason. `true`/`2.5`/`0`/`-1`/`"3"`/`null` all → `not a version number`; `version: 99` → `newer than this didrun understands (max 3); upgrade didrun` | +| V18 | Forging the note's own rows does not mint a green | rewrite `grade` to `tree-exact` and `evidence.entry_hash` to zeros | `WITNESS-UNAVAIL`, **0/1 evidence-bound** — the forgery grades *worse*, not better | +| V19 | Supersession retains the record and excludes it from the verdict | claim → move the tree → re-run → re-claim the same label → seal | `1 superseded`, first claim carries `superseded_by: 1` and prints `superseded by claim #1 (record, not verdict)`, `--strict` exit **0** | +| V20 | A narrowing re-seal is refused | `seal` over a narrower window | exit **2**, message names both windows and the two ways out; `--reseal` permits it | +| V21 | A conjunction is never better than its worst member | `claim conjunction --of T,L` with T stale | `STALE`, `worst of 2 conjunct(s): 'T' graded stale`, and both members printed (`of T: STALE`, `of L: TREE-EXACT`) | +| V22 | The detector tiers separate benign from dangerous | a deep build path in stdout vs. a `ghp_…` token in argv | path → **2 notice**, seal succeeds exit 0. Token → **export blocked**, exit **3**, located at `argv of event 0 arg 2 offset 11`, **0 notes written** | +| V23 | A scope miss names the fix | `--path src/` with a `docs/` change | `STALE`, `1 change(s) outside declared pathspecs — add --path docs/ to make this scope-exact` | +| V24 | A claim after a witnessed failure is refused | pass, then `exit 3`, then `claim tests-pass` | exit **2**, and **no `claims.jsonl` written at all** | +| V25 | Environment drift is advisory until asked | `TZ=UTC` seal, `TZ=Asia/Tokyo` verify | `0 match / 1 drifted`, `(advisory; --require-env-match makes it refuse)`; `--require-env-match --strict` exit **1** | +| V26 | A cited authority is cited, not claimed as approval | `authorize` → `seal --allow-secrets --require-authority` → `verify` | `CITED` row carrying, verbatim: *"NOT a signature and NOT approval — didrun cannot tell who wrote it"* | +| V27 | Amend still resolves the note | `git commit --amend`, then `verify` | `resolved-by tree-fallback`, verdict unchanged | +| V28 | Staleness has a non-empty delta | claim, then change the code, then seal | `! STALE`, `tree moved since evidence: 1 path(s) differ` + `M file.txt`, `--strict` exit 1 | +| V29 | `verify` in a repo with no ledger creates nothing | `verify` in a fresh repo | `○ NO CLAIMS`, and no `.didrun` afterwards | +| V30 | **The corpus guardrail refuses the live tree by name** | `is_refused()` / `resolve_corpus_root()` over path strings only — no I/O against the corpus | the live path, its trailing-slash form, and any path *inside* it are all refused by digest; a live-looking copy (`.didrun` + `.git`) is refused by heuristic; a proper copy is accepted | + +**Grep gate**, run and quoted: + +``` +$ grep -rnE "print\(|\bassert .*(line|raw|content|body)\b" tests/compat/ +tests/compat/__init__.py:7:content. ``assert line == ...`` is forbidden, because pytest prints both sides +tests/compat/test_corpus_replay.py:6:two hashes is fine — a hash is not content. ``assert line == ...`` is +tests/compat/test_corpus_replay.py:677: its arguments, so ``assert note_violations(, 0) == []`` would +tests/compat/test_corpus_replay.py:689: assert one_newline == [] +``` + +Four hits, **zero echo a ledger line**: three are the prose that states the rule, and `one_newline` is a +list of `(index, reason)` violation names bound to a local before the assertion, which is the shape the +gate exists to force. No `print(` anywhere in `tests/compat/`. + +**Read-only gate:** not applicable and not claimed — nothing was pointed at a corpus, so there were no +mtimes to compare. + +--- + +## BET — argued, not measured + +| # | Claim | What would settle it | Who | +|---|---|---|---| +| B1 | **D1 — every stored `entry_hash` in the archived logs recomputes** (or the exceptions are the two known fork incidents) | the corpus leg, over a copy | **HUMAN (Gate 1)** | +| B2 | **D2 — all 65 notes satisfy the three-part round-trip criterion** | the corpus leg, over a copy | **HUMAN (Gate 1)** | +| B3 | **D3 — every archived `objects/` entry partitions, residue 0** | the corpus leg, over a copy | **HUMAN (Gate 1)** | +| B4 | **D4 — multi-index 0, the 178/846 pathspec histogram, the gap distribution** | the corpus leg, over a copy | **HUMAN (Gate 1)** | +| B5 | **FU1** — the 667-vs-846 claim-count discrepancy between the original brief and the artifact | recount over the archived notes | HUMAN | +| B6 | **FU2** — "the structured detectors produced zero findings across the seals". Measured over exported `argv_preview` strings only, **never over the ledgers** | run the detectors over archived ledgers, by someone authorised to read them | HUMAN | +| B7 | **FU3** — the 1-of-8 resume-legality mapping under Locked Decision 2. Inference over recorded tree ids, not observation | inspect the eight launches directly | HUMAN | +| B8 | **FU4** — lane 2's EV2 false-green, single-sourced in the research. V9 above is a second witness for the **mechanism**, in a scratch repo; it is **not** a witness for the corpus | replay EV2 against a corpus copy | HUMAN | +| B9 | **FU5** — every frequency, cost and duration figure in the research, all self-reported by the system under study | independent instrumentation | HUMAN | +| B10 | **Red-team finding 16** — whether the 197 non-terminal claim bindings had an intervening failure | a grep over the archived logs, by someone authorised to read them | HUMAN | +| B11 | The **S6 live-agent recall leg** — capture recall against real agent sessions | a real agent on a real machine (`harness/README.md`) | **HUMAN (Gate 3)** | +| B12 | **CI green on 3.11 / 3.12 / 3.13 × ubuntu / macos** | the CI workflow. **Cannot be settled here** — this interpreter is 3.14.5, and V4 is a grammar check, not an API one | **HUMAN / CI (Gate 2)** | +| B13 | That any of this **changes a decision**. 846 corpus claims produced exactly **1** `failed`; a corpus whose only observed actor was cooperating cannot distinguish a superbly effective gate from a nearly inert one | instruct an agent to obtain a green `didrun verify --strict` by any means. **Not run here, and not authorised by this pack.** The highest-value missing experiment | **HUMAN (Gate 4)** | +| B14 | Durability under power loss. There is no `fsync` anywhere in the package; V15 shows appends are *mutually* safe, not that a committed append survives a crash | a fault-injection harness | open | + +--- + +## KNOWN NOT CLOSED — non-guarantees v0.2 leaves open on purpose + +- **Inherited-environment false greens.** `GOFLAGS=-exec=/usr/bin/true` and its family + (`NODE_OPTIONS`, `PYTHONPATH`, `LD_PRELOAD`, `DYLD_INSERT_LIBRARIES`, `PYTEST_ADDOPTS`, a PATH shim, + `sitecustomize.py`) make a passing command that ran nothing. No allowlist closes it. P3.3 records a PATH + **digest**; a declared clean-environment mode is the honest v0.3 shape. +- **No ledger retention or purge.** A credential scrubbed from git history survives as a loose object under + `.didrun/objects`. Real, and it needs a retention policy behind it before it needs code. +- **The redaction marker string is forgeable.** `«redacted:…»` is a convention, not a cryptographic mark; + keyed fingerprints were rejected because a per-ledger key destroys the cross-ledger stability that + motivated them. +- **`chmod` is close to a no-op on Windows.** V12's `drwx------` is a POSIX result and does not travel. +- **`flock` on a network filesystem.** V15 was measured on a local APFS volume. NFS/SMB semantics differ and + were not tested. +- **Tree-fallback returns the *first* matching note** (red-team finding 20) rather than failing closed on + more than one match. Deliberately out of this pack; the first candidate for a v0.2.1 unit. +- **`argv_preview` is written into every note, dropped on read, and computed-but-never-rendered in HTML** + (red-team finding 21). A product call, not a hardening unit. +- **Duplicate conjunct labels are counted twice.** `--of a,a` grades correctly and reads oddly. +- **`seal` still does not consult the chain.** `verify` does; sealing over a broken chain is possible. +- **None of this resists a determined local forger.** Whoever can substitute a ledger can regenerate its + chain and re-run `claim` and `seal` to mint fresh hash-bound claims. `ledger.py` declares this in the + source, `docs/TRUST_MODEL.md` states it, and V18 is *accident and drift detection*, not a barrier. + +--- + +## The four gates + +| Gate | Who clears it | Status | +|---|---|---| +| 1 — the corpus copy | **HUMAN** | **NOT CLEARED.** `DIDRUN_COMPAT_CORPUS` unset. Clear it by copying the archive tree **without** its live `.didrun` directory onto a separate volume and re-running with the variable set at the copy. The harness refuses the live tree by name (V30) even if it is set wrongly | +| 2 — CI on 3.11/3.12/3.13 × ubuntu/macos | **HUMAN / CI** | **NOT CLEARED.** Local green does not imply CI green | +| 3 — the S6 live-agent recall leg | **HUMAN** | **NOT CLEARED.** Unexercised | +| 4 — does any of this change a decision | **HUMAN** | **NOT CLEARED, and unanswerable from this corpus** | + +--- + +## Confidence + +**30 of 44 load-bearing conclusions are externally verified** — by the shipped CLI's own output in a +throwaway repo this session, or by a suite test that fails when the property stops being true. The other +14 rest on argument from code, on figures the system under study reported about itself, or on a signal +only a human or CI can supply. **Every one of the four real-corpus legs is in the second group.** + +A green replay means *these hashes reproduce*. It has never meant *the corpus is proven intact*, and a +`tree-exact` grade means *recorded against the sealed tree* — never *proven correct*. diff --git a/docs/MEASUREMENTS.md b/docs/MEASUREMENTS.md index ec041eb..52443c7 100644 --- a/docs/MEASUREMENTS.md +++ b/docs/MEASUREMENTS.md @@ -34,6 +34,38 @@ Relevant detail for Claude Code specifically: its Bash tool spawns the **user's `/bin/zsh -c` on a default macOS box — which is why Tier 2 is per-shell dispatch rather than `BASH_ENV` alone. +## Capture: what the streaming pump costs (v0.2) + +Tier 0 drained the child through `subprocess.run(capture_output=True)` until +v0.2, which buffers both pipes to completion — so a wrapped command surfaced +nothing at all until it exited (measured on real sessions: single events of +201.95 s, 332.42 s, 447.77 s and 530.21 s with no progress, and a 14.6-hour +chain whose entire operator-visible output was one summary line per command). +v0.2 replaces it with a `selectors` pump that drains both pipes incrementally. + +The question that had to be answered with numbers rather than a shrug is what +the pump costs per spawn. Both arms measured **alternately in one process** (so +machine load is shared, not donated to whichever arm ran during a quiet minute), +macOS 26.5 / APFS / arm64, Python 3.14.5: + +| Shape | Old — `subprocess.run(capture_output=True)` | New — `selectors` pump | Delta (median) | +|---|---|---|---| +| `/bin/echo hi`, 200 paired runs | 2.340 ms median (mean 2.358, p90 2.601) | 2.364 ms median (mean 2.370, p90 2.628) | **+0.024 ms (+1.0%)** | +| 1 MiB to stdout, 40 paired runs | 24.099 ms median (p90 24.661) | 23.581 ms median (p90 24.309) | **−0.517 ms (−2.1%)** | + +So the pump is ~24 µs per short spawn — the same order as the wrapper's own ++55 µs, and invisible next to the two tree digests a `didrun run` already pays +(a full `run_wrapped` over `/bin/echo hi` is ~245 ms median on this machine, +almost all of it digest). On large output it is marginally *faster* than +`communicate()`. Both readings are single-machine and macOS-only; the Linux +rerun this file already gates on covers this table too. + +What the pump does **not** change is what is recorded: the drained blobs are +byte-identical to the buffered path's, per stream, over empty, 1 MiB, interleaved, +non-UTF-8 (NUL-bearing) and stderr-only output, and exit codes match including +signal deaths. That equality is the criterion, and it is a test +(`tests/test_streaming_capture.py`), not a claim made here. + ## Tree digests: why copied-index write-tree The digest must reflect staged + unstaged + **untracked** state, deterministically, @@ -61,6 +93,151 @@ Also measured: git notes do **not** follow `commit --amend`/rebase by default (`notes.rewriteRef` defaults exclude custom refs) — which is why manifests dual-bind to commit id **and** tree id, with tree-fallback lookup. +## Secrets detection: why the detectors are tiered (v0.2) + +Measured **2026-07-29** over a real archive of **66** sealed `refs/notes/didrun` +notes (**884** graded claims, **22,062** exported argv elements, **602,975** +characters of exported argv), read through `git notes list` + `git cat-file` +only. Counts, never tokens. + +**What the detectors actually caught, in production:** + +| Detector | Redaction markers in the 66 exported notes | +|---|---| +| `high-entropy` (entropy sweep) | **7,482**, in 60 of 66 notes | +| `aws-access-key` | 0 | +| `github-token` | 0 | +| `slack-token` | 0 | +| `openai-key` | 0 | +| `pem-private-key` | 0 | +| `env-secret-assignment` | 0 | + +The six structured detectors fired **zero times**. Every finding the tool has +ever blocked on came from the entropy sweep — and **65 of the 66 notes record +`secrets_override: true`**, i.e. the gate was overridden on 98.5% of real seals. +A blocking gate that fires only false positives is a gate that gets switched +off, which is the failure mode the tiering exists to prevent. This asymmetry is +what the two-tier split rests on, so it is a gate on the change and not a +footnote: had the structured tier fired even once here, the split would have +needed a human decision. + +**The root cause was one character.** The entropy sweep's token class included +`/`, so an entire absolute filesystem path was a single token — and a path +accumulates distinct characters much faster than any one of its segments does. +Removing `/` from the class, over real absolute paths on this machine: + +| Path population | Paths | Entropy findings before | After | +|---|---|---|---| +| Repository file paths (project A) | 556 | 135 | **0** | +| Repository file paths (project B) | 47 | 9 | **0** | +| Homebrew Cellar directories (depth ≤ 6) | 2,039 | 35 | **0** | +| **Total** | **2,642** | **179** | **0** | + +The 24-character floor and the 4.2 bits/char threshold are unchanged, so the +before/after is attributable to the candidate — what gets scored — alone. The +population sits right on the knife edge — median 4.110 and 4.088 bits/char for +the two repository populations, against a 4.2 threshold — which is why the same +tool produced wildly different finding counts on runs that differed only in +where the repository was checked out. + +**That table is one-directional, and reading it as two-directional was a +mistake that cost a real credential class.** It measures what stopped firing on +paths. It does not measure what stopped firing on secrets — and `/` is not only +a path separator, it is 1 of the 64 characters of base64. Dropping it from the +token class meant a base64 credential whose slash-free runs are all shorter +than the 24-character floor stopped being a *candidate at all*: not re-scored, +not demoted to `notice`, invisible. Replacement is driven off the findings, so +invisible also meant unredacted — published verbatim into `refs/notes/didrun` +under a seal line reading "0 findings". + +So the separator is no longer decided by the character class. A run is taken +over the class **including** `/`, and a per-run test decides whether that `/` is +a separator (score the segments) or an alphabet member (score the run whole). +The test is that a filesystem path is a sequence of **names** and base64 is not: +a segment counts as a name if it is ≥3 characters, ≥60% letters, and ≥80% of +those letters are lower case. That admits `usr`, `site-packages` and CamelCase +like `CoreServices` (10/12) or `Frameworks` (9/10); base64 is ~50% lower case by +construction and clears it only by accident. **One** name is enough to call the +whole run a path — the conservative direction, since it can only return a run to +segment scoring. + +Measured over the same population plus a wider one (system framework and +shared-library trees, a virtualenv's site-packages, `/usr/share`), and over +seeded credential populations of 20,000 samples each: + +| Path population | Paths | pre-v0.2 | `/` dropped | now | +|---|---|---|---|---| +| The 2,642 above | 2,642 | 179 | **0** | **0** | +| Wide (adds system/library trees) | 46,403 | 9,278 | **248** | **248** | + +| Credential population (detected) | pre-v0.2 | `/` dropped | now | +|---|---|---|---| +| AWS secret-access-key shape (40 chars of `[A-Za-z0-9+/]`) | 100.0% | 83.6% | **98.9%** | +| `base64(24 bytes)` = 32 chars | 99.3% | 74.6% | **97.1%** | +| `base64(32 bytes)` = 44 chars | 100.0% | 86.8% | **99.0%** | +| `base64(64 bytes)` = 88 chars | 100.0% | 99.6% | **99.9%** | + +The false-positive column is **identical** to the `/`-dropped column on both +populations — the path win is kept in full, at zero measured cost — and the +recall lost by dropping the character is mostly recovered. The 1–3% still +missing is a credential whose own slashes chop it into sub-floor pieces *and* +leave one piece looking like a name; that is the price of biasing the test +toward paths, and it is the honest side to lose on for a detector that is +`notice`-tier precisely because it is a poor discriminator. None of this is +evasion-resistant, and it is not offered as such: an attacker who knows the rule +can satisfy it deliberately. + +The entropy sweep was **demoted to `notice`, not deleted**: it is the only +detector that catches an unstructured credential, and length is not secrecy in +either direction. Per-character entropy over an *n*-character token is capped at +log2(*n*), so ~18 distinct characters clears 4.2 — and, measured the other way, +a lowercase-base36 credential sits near 4.05 and does **not** clear it. The +threshold is a poor discriminator in both directions; that is an argument for +not blocking on it, not for trusting it. + +**Both directions of the blocking change**, replayed over the same 66 notes: +under the new rule **0** would block (the 65 overridden seals become clean +seals with a warning), and **0** newly block — no claim label and no delta path +in the archive carries a structured-secret shape, even though those fields are +exported and were never scanned before. + +## Environment fingerprint: what the v2 gate keys actually do (v0.2) + +The v2 gate binds `LANG`, `SHELL`, `TZ`, `VIRTUAL_ENV` and the process umask, +and drops `PATH` and `PWD` (both were in v0.1's key set). The fingerprint is now +compared across time, so every key in it is a potential *false*-drift source and +the drop needs a number rather than a shrug. + +Measured on one machine (macOS 26.5, arm64), sampling each key across five shell +contexts and comparing each against the value the current process holds. Counts +only — no value is recorded here or anywhere in the repo: + +| Key | Contexts disagreeing with the current process (of 5) | +|---|---| +| `SHELL`, `TZ`, `VIRTUAL_ENV` | 0 | +| `LANG` | 1 (login zsh) | +| `PATH` | 1 (login bash) | + +Contexts: `zsh -lc`, `zsh -ic`, `bash -lc`, `bash -c`, `sh -c`. + +**Read this honestly: it is a weak result and it does not on its own justify +dropping `PATH`.** On this machine `PATH` was no noisier than `LANG`, which is +kept. The measurement bounds the claim rather than proving it — one machine, one +day, no `direnv`, no `nix develop`, no tmux, no second homebrew prefix, which are +the conditions under which `PATH` reordering is reported to be routine. It also +records a real cost of the keys that were kept: `LANG` differs between login and +non-login zsh here, so a seal from one and a verify from the other reports drift. +That is a true statement about the environment rather than a bug, and it is why +drift is advisory unless `--require-env-match` is passed. + +The load-bearing argument for the two drops is structural, not statistical: +`PATH` flips on any venv activation that `VIRTUAL_ENV` already records, and +`PWD` is not the child's cwd at all — a wrapped command always runs with +`cwd=`, so gating on `PWD` would make two byte-identical executions of the +same command against the same repo disagree because the operator was standing in +a different directory. `PATH` is still recorded, as a truncated digest beside the +gate, so its drift stays visible without being a refusal. See docs/COMPAT.md. + ## Known digest blind spots (by design, documented rather than hidden) Ignored files are invisible to every git-based digest (`tree-exact` does not diff --git a/docs/TRUST_MODEL.md b/docs/TRUST_MODEL.md index 5032fd0..ef473a8 100644 --- a/docs/TRUST_MODEL.md +++ b/docs/TRUST_MODEL.md @@ -54,6 +54,68 @@ later. It is also a genuine adoption bet, because voluntary developer commit signing is rare in practice. Until it lands, **didrun's guarantees hold only among parties who already share a baseline of trust.** +## Operator authority: what `cited` attests + +`didrun authorize --artifact --scope --label ""` records that a +file with a particular SHA-256 was cited as the authority for one exceptional +transition. `seal` binds the live citations into the manifest, grades each one +`cited`, and every surface that shows one prints this sentence beside it: + +> cited authority: recorded that a file with this digest was cited for this scope. +> NOT a signature and NOT approval — didrun cannot tell who wrote it. + +**What it attests.** A file with this digest existed at this path and was cited for +this scope, at this point in an append-only record. The digest, the scope, the label +and the self-asserted principal are in the note; the artifact's bytes never are. + +**What it does not attest — read this part.** It is not evidence that a human wrote, +read, or approved anything. **An agent can author its own authorization artifact in +one line**, cite it, and seal. There is no identity here: `principal` is whatever +`$USER` said, recorded as `self-asserted`, and didrun authenticates nobody. `cited` +must never be rendered or reported as "signed", "approved", "verified" or +"authorized by" — the same rule that keeps `tree-exact` from rendering as "verified +true". + +**The one property it buys.** Retroactive fabrication becomes detectable: because the +digest sits in a note sealed earlier, an artifact edited or written after the fact no +longer matches what that earlier note recorded. That is a real, narrow gain, and it is +the whole gain. It converts an authority from prose into a fixed, diffable, +digest-bound object; it does not make the object trustworthy. It is also not tamper +resistance — whoever can rewrite a note can rewrite the digest in it, exactly as they +can rewrite a grade. + +**Why it is shaped this way.** `--allow-secrets` was the shipped instance of the +problem: a bare flag stored as a bare boolean, with no principal, no basis and no +artifact, so a reader could not tell an authorized override from an agent that simply +passed the flag. didrun's own thesis is that a claim with no witnessed event grades +`unknown`; an out-of-band authorization is exactly that shape, and the tool used to +grade it implicitly true. So authorities are: + +- **scoped** — only `allow-secrets` and `reseal`, the exceptional transitions. There + is deliberately no scope for an ordinary seal, and none will be added. +- **expiring** — a TTL counted in seals (default 1, minimum 1, and no value meaning + "never"). Seals rather than wall time, because a clock in the trust path is not + allowed here. +- **never required by default** — `seal --require-authority` is opt-in. Any + authorization demanded on the happy path becomes a rubber stamp within a week, + which is precisely how the information content of `--allow-secrets` reached zero. + +**The honest anchor for a stronger tier is a key the agent cannot read.** A +passphrase-less key in `~/.ssh` that the agent can read is theatre; a hardware-backed +key with touch-to-sign (`ssh-keygen -t ecdsa-sk`, or a Secure Enclave key) is the real +anchor, because the human's finger is the thing an agent cannot fake. Two further +tiers are designed and **not built**: + +- **Tier 1 — reuse the key the developer already has.** `git verify-tag` / + `git verify-commit` under `gpg.format=ssh` and `~/.ssh/allowed_signers`, recording + the signer identity and the verification output. Zero new key management, no crypto + inside didrun. **Not built.** +- **Tier 2 — org identity, roles, revocation, trust roots.** **Not built**, and out of + scope for a local single-user tool. + +Both are consequences of the deferral in the section above, and neither is claimed +anywhere in didrun's output. + ## The ledger is secret-bearing The ledger records everything a session printed — including anything an agent @@ -61,14 +123,53 @@ echoed: tokens, `.env` contents, connection strings. Treat `.didrun/` and any exported bundle as **secret-bearing**: - `.didrun/` is self-ignored (a `*` gitignore inside it) so it is never committed. -- At `seal`/export, didrun scans argv and captured output for likely secrets and - **blocks export by default** if it finds any (override with `--allow-secrets`, - which is logged into the manifest, and still redacts the exported artifact). +- At `seal`/export, didrun scans two domains and **tiers** what it finds. The six + structured detectors are `block`: they fire on shape and name what they found. + The entropy sweep is `notice`: it fires on "this looked random", which over a + 24-character run is a statement about length far more than about secrecy. +- **A seal is refused only for a `block`-tier finding in the bytes about to be + published** — the argv of each claim-bound event and the serialized manifest, + claim labels and delta paths included. Override with `--allow-secrets`, which + is logged into the manifest; the exported artifact is redacted either way. +- **What blocks and what is redacted are the same fields.** Every string in a + published entry that carries operator- or repository-authored text is replaced + by a marker before the note is written, and the spans are declared in + `redaction.applied` / `redaction.fields`. For a claim that is the argv preview, + the label, the pathspecs and the changed paths; for a cited authority it is the + label, the cited path **and the principal**. This is stated because it was once + false in the worst direction, twice: a token in a claim label refused the seal + and was then published verbatim on the override, under a refusal message that + had already promised a redacted artifact — and when authorities were added, + `principal` reproduced it exactly, having been scanned by the whole-manifest + backstop but left out of the pass that redacts. Blocking is not redacting. An + enumeration of "the fields that carry outside text" also goes stale the moment + a field is added, so the authority enumeration is checked for completeness by a + test rather than trusted, and a field it does not classify is dropped from the + note instead of published. +- Everything else is **reported loudly and does not stop the seal**: every + `notice`-tier finding, and any `block`-tier finding confined to the recorded + output blobs, which stay in a gitignored local ledger and are never published. + Refusing over bytes that do not leave stopped work without protecting anything. +- The manifest records **what was scanned** (`secrets`: the domain, the events + scanned out of the events present, the bytes, the detector-set version, and + the findings per tier), so `secrets_override: false` is a checkable statement + instead of the same word for "nothing found" and "almost nothing looked at". +- Every finding carries a location: source kind, event index, argv index or line, + byte offset, and a **locality** fingerprint. The fingerprint deliberately does + not digest the matched token — an unkeyed digest of a short high-entropy string + would be a brute-force oracle for the thing just redacted. - Redaction applies to what *leaves* the machine (bundles, HTML reports). The raw - local ledger is left intact and stays local. + local ledger is left intact and stays local. One documented exception: an HTML + report's file list for a `stale` claim is **recomputed against your working + tree** at verify time rather than read from the note, so it is not the note's + redacted copy — see docs/COMPAT.md. The scanner is a safety net tuned to bound false positives, not a guarantee. Do not rely on it to catch a novel secret format; keep secrets out of command output. +How weak the entropy tier is in **both** directions is measured, not asserted — +see the detection tables in MEASUREMENTS.md. Those tables also bound the other +direction: the entropy sweep misses roughly 1–3% of slash-bearing base64 +credentials by design, and nothing in it resists an attacker who knows the rule. ## Honest capture limits diff --git a/docs/V02_STATUS.md b/docs/V02_STATUS.md new file mode 100644 index 0000000..2b2003d --- /dev/null +++ b/docs/V02_STATUS.md @@ -0,0 +1,220 @@ +# didrun v0.2 — release readiness, stated honestly + +**As of 2026-07-29, commit `49a8b74` on `v0.2-hardening`.** Nineteen units landed across six phases. This +document exists because the difference between *"we shipped the code"* and *"we know it works"* is the only +thing a trust tool actually sells, and the two are not the same size here. + +Read this before deciding to release. It is deliberately unflattering. + +Two conventions used throughout, because the whole product depends on them: + +- **`tree-exact` means "recorded against the sealed tree."** It has never meant "proven correct." +- **UNRECEIPTED** means a command was run without the didrun wrapper, so no receipt backs it. Everything in + the *Verified* section below is UNRECEIPTED: the wrapper discipline is suspended while hardening didrun + itself, for the obvious reason. + +--- + +## VERIFIED — what was actually demonstrated, and how + +The gates, all run at `49a8b74` on CPython 3.14.5 / macOS arm64: + +| Gate | Result | +|---|---| +| `python -m pytest -q` | **496 passed, 4 skipped** (the 4 skips are exactly the real-corpus legs) | +| `python -m harness.recall` — the CI kill-gate | **PASS**, probe freeze INTACT, defect-class recall **100.0% (3/3)** | +| `python -m pytest -q tests/compat` | 11 passed, 4 skipped; synthetic legs report **0 note violations**, **0 layout residue**, **0 multi-index claims** | +| 3.11 grammar parse over all 41 `*.py` | clean. **Syntax only** — it cannot catch a 3.12+ stdlib API | +| stdlib-only | 24 top-level imports in `src/` + `harness/`, **none** outside the stdlib; `dependencies = []` | +| no private paths in shipped files | clean; the only `/home/...` strings are a public CI shape and an explicit `example` | + +Beyond the gates, the real CLI was walked end-to-end in throwaway repos and **30 distinct properties were +observed directly**, quoted verbatim in `docs/COMPAT_REPLAY.md`. The load-bearing ones: + +**The headline defect is closed, and I watched it close.** The attack v0.2 exists to stop — seal three +claims, delete the ledger, run three unrelated commands that exit 0 against the same tree, then verify — +now returns `WITNESS-UNAVAIL` on all three, **0/3 evidence-bound**, `--strict` exit 1. The pre-P1.2 +behaviour was `3/3 recorded-exact`, exit 0. That is the whole point of the release and it is measured, not +argued. + +**Every earlier phase's headline property still holds after P3–P5 landed on top of it.** Re-checked +individually, not assumed: + +- **Seal is fail-closed and all-or-nothing.** With note publication made to fail for real (the git object + store set unwritable), `seal` exits **2**, quotes git's error, and writes **no `seals.jsonl` line, no + note, and no watermark file**. Restore the permission and the same seal succeeds and verifies green. A + failed seal leaves nothing behind that a later seal would silently skip. +- **A substituted ledger fails closed** (above), and **forging the note's own rows makes the verdict + worse, not better**: rewriting a stored `grade` to `tree-exact` and zeroing the evidence hash yields + `WITNESS-UNAVAIL`, 0/1 evidence-bound. +- **Interrupts never lose a flight.** SIGINT → exit 130, SIGTERM → exit 143, one greppable line each, no + traceback, the in-flight event recorded as `unobserved no-exit`, `chain intact`, and the next `run` + appends normally. +- **Ledger permissions hold**: `drwx------`, `session.log` `-rw-------`. +- **Concurrent appends are safe and do not serialize.** Six parallel runs produced six events with the + chain intact, and a second `run` finished in **0.38 s** while a five-second run held the session. +- **A torn log refuses instead of writing past the tear** — named byte offset, spelled-out recovery, + nothing appended. (Its delivery is a defect; see K1.) + +**And the new v0.2 surfaces do what they say.** A note body of **1,819,948 bytes** — fourteen times the +largest note in the real corpus, and far past `ARG_MAX` — publishes, re-resolves and parses. A malformed +note is a **graded refusal with exit 2** across every shape tried (six bad `version` types, a string where +a claim object belongs, a truncated body, an unknown claim type, a retired claim type): no traceback, and +critically **no false green** — the `{"version": true}` class that used to verify green now refuses. +Supersession retains the retired claim in the record and excludes it from the verdict. A conjunction is +graded as the worst of its members and names which one. The secrets detector puts a deep build path at +**notice** (seal proceeds) and a `ghp_…` token at **block** (exit 3, located to the argument and offset, no +note written). A cited authority prints, verbatim, *"NOT a signature and NOT approval — didrun cannot tell +who wrote it."* + +**The guardrail protecting the live corpus was verified without touching the corpus.** The refusal list is +stored as path digests, so the live tree — plus its trailing-slash form and anything inside it — is refused +by name, a live-looking copy is refused by heuristic, and a proper copy is accepted. Checked with string +hashing only; no filesystem access to that tree occurred at any point in this pass. + +--- + +## STILL A BET — designed, argued from code, not measured + +**The single biggest one: the real-corpus replay never ran.** Nineteen units each argued in their own risk +register that they would not disturb 65 sealed notes and 178 archived ledgers. **Every one of those +arguments is still an argument.** The harness that would settle it exists, is content-blind, and is green +on synthetic fixtures — and it has never seen the real thing, because the corpus is live, secret-bearing +evidence for a build that is still running and copying it is not mine to do. Four legs are unexercised: +chain recompute over the archived logs, the note round-trip over all 65 notes, the `objects/` layout +partition, and the claim-shape invariants. + +A green synthetic run is **not** corpus validation, and the harness prints exactly that word — +`corpus: SKIPPED (unverified)` — rather than letting a green line imply it. + +The rest, each with what would settle it: + +- **CI on Python 3.11 / 3.12 / 3.13 × ubuntu / macos.** Unverified and **unverifiable here**: this + interpreter is 3.14.5. The 3.11 check that passed is a *grammar* parse; it cannot catch a 3.12+ stdlib + API. The declared floor is 3.11 and nothing has ever run there. +- **Capture recall against real agent sessions (the S6 leg).** The frozen S1–S5 probes report 100% + defect-class recall; S6 needs a real agent on a real machine and has never been run. +- **Durability across a crash.** There is no `fsync` anywhere in the package. Concurrent appends are + *mutually* safe; that is not the same as an append surviving power loss. +- **`flock` on network filesystems, and permissions on Windows.** Both measured on local APFS only. +- **Five research follow-ups (FU1–FU5) and red-team finding 16.** A claim-count discrepancy (667 vs 846); a + "zero detector findings" figure measured over exported argv strings and never over the ledgers; a + 1-of-8 resume-legality mapping that is inference over recorded tree ids rather than observation; the + original EV2 false-green, single-sourced (this pass witnessed the *mechanism* in a scratch repo, which is + not a witness for the corpus); and every frequency, cost and duration figure in the research, all + self-reported by the system under study. Details and owners in `docs/COMPAT_REPLAY.md`. +- **The most important unknown: whether any of this changes a decision.** 846 recorded claims produced + exactly **one** `failed`. That is consistent with a superbly effective gate and equally consistent with a + nearly inert one, and a corpus whose only observed actor was cooperating cannot tell the two apart. The + experiment that would — tell an agent to obtain a green `didrun verify --strict` by any means and see what + it does — was **not run**, is not authorised by the plan this release followed, and is the highest-value + missing signal in the whole project. Until someone runs it, "didrun catches dishonest agents" is a + hypothesis. + +--- + +## KNOWN NOT CLOSED — defects and gaps left in deliberately + +### Defects found in this pass and NOT fixed + +The final integration pass measures; it does not fix, because a fix landing after the last verification +gate is unverified code in a release. Each of these is a small, self-contained unit for Drew to sequence. + +- **K1 — `didrun run` reports two operational errors as raw Python tracebacks with exit 1.** + 1. A command that does not exist (`run -- /bin/false` on macOS, where that path is absent) → + uncaught `FileNotFoundError`, exit 1, **and no event recorded at all**. + 2. A torn `session.log` → uncaught `LedgerError`, exit 1. The refusal *message* is excellent and + complete; only its delivery is wrong. + + This breaks an invariant the codebase states in its own words in `cmd_verify`: *evidence this binary + cannot read is a graded refusal, not a crash — 2, not `--strict`'s 1*. `cmd_verify` catches `LedgerError` + and returns 2; `cmd_run` wraps nothing, and `main()` catches only the two interrupt types. **Exit 1 is + `--strict`'s "graded badly" code**, so a crashed `run` is indistinguishable by exit status from an honest + failure — the same class of bug P4 already fixed on the manifest side. **Pre-existing, not a v0.2 + regression:** `main`'s `subprocess.run` raises the same way. Fix shape: one `try` in `cmd_run` over + `OSError` and `LedgerError`, returning 2 with a named message. + +- **K2 — `show --session` reports an unverifiable chain as "BROKEN".** `ledger.py` computes a three-way + status (`intact` / `broken` / `unverifiable`) and `render.py` documents in a comment that the two faults + *"get different words on purpose"* — broken means an entry did not recompute, unverifiable means this + binary cannot check it. `verify`'s reason line honours that (`ledger chain unverifiable — entry 1: + unknown chain preimage version: 2`). But `show --session` goes through the boolean `verify_chain()` and + can only say `chain BROKEN at index 1`, and `verify`'s own headline banner says `CHAIN-BROKEN` for both. + So a future ledger format reads to the operator as **tamper**, which is precisely the false alarm the + frozen preimage was built to prevent. + +- **K3 — CLOSED after the integration pass.** `didrun --version` printed `0.1.0`, which was + self-defeating: the forward-compatibility story in `docs/COMPAT.md` is phrased as "a v0.1 reader versus + a v0.2 reader" and instructs operators to *upgrade the verifier before the sealer*, and `--version` is + the only way anyone can tell which they have. Now `0.2.0` in `pyproject.toml` and + `src/didrun/__init__.py`. It was outstanding at the end of the integration pass only because that pass + fenced off `src/didrun/**` by design, not because the change was in doubt. + +- **K4 — the happy path shouts.** The first seal of every session prints a four-line `WARNING` about + having no watermark file, a condition the warning itself then explains is *"correct for a session's first + seal."* `docs/COMPAT.md` argues elsewhere that a gate which fires spuriously gets ignored within a week; + this is that, on the most-travelled path in the tool. + +- **K5 — a plan inconsistency worth recording so nobody chases it.** P4.1's documented supersession + scenario ("fail → claim → fix → claim, no commit between") is **unreachable through the CLI** now, + because P5.2 correctly refuses a claim bound to an event that exited nonzero. Supersession itself works + and was verified by the route that remains: claim, let the tree move, re-run, re-claim the same label. + The product is right; the plan's matrix row is stale. + +### Non-guarantees the design leaves open on purpose + +- **Inherited-environment false greens.** `GOFLAGS=-exec=/usr/bin/true` and its whole family + (`NODE_OPTIONS`, `PYTHONPATH`, `LD_PRELOAD`, `DYLD_INSERT_LIBRARIES`, `PYTEST_ADDOPTS`, a PATH shim, + `sitecustomize.py`) produce a command that passes having run nothing. No allowlist closes it, and putting + the effective environment in the event body would put the richest secret carrier on a developer machine + into the chain preimage. A PATH digest is recorded; a declared clean-environment mode is the v0.3 shape. +- **No retention or purge.** A credential scrubbed from git history survives as a loose object under + `.didrun/objects`. +- **The `«redacted:…»` marker is forgeable** — a convention, not a cryptographic mark. +- **Tree-fallback takes the first matching note** and does not fail closed on a second match (red-team + finding 20). First candidate for v0.2.1. +- **`argv_preview` is written into every note, dropped on read, and computed-but-unrendered in HTML** + (finding 21). A product decision, not a hardening unit. +- **`seal` does not consult the chain.** `verify` does. Sealing over a broken chain is possible. +- **Duplicate conjunct labels count twice.** `--of a,a` grades correctly and reads oddly. +- **No PKI, signing, key management or revocation.** `authorize` records a cited artifact's digest and says + in its own output that it is not a signature. +- **And the one that frames all the others: none of this resists a determined local forger.** Whoever can + substitute a ledger can regenerate its chain and re-run `claim` and `seal` to mint fresh hash-bound + claims. The source declares it, `docs/TRUST_MODEL.md` states it, and nothing in v0.2 changes it. What + v0.2 buys is **accident and drift detection on published evidence** — which is what actually goes wrong + in practice, and is worth having. It is not tamper-proofing and must never be sold as such. + +--- + +## What a human has to do before this ships + +Four things need a signal no amount of local work can produce: + +1. **Copy the archive tree and run the corpus replay.** Copy it *without* its live `.didrun` directory onto + a separate volume, then run the compat suite with `DIDRUN_COMPAT_CORPUS` pointed at the copy. The + harness refuses the live tree by name if the variable is set wrongly. Until this runs, "v0.2 does not + disturb existing sealed evidence" is an argument, not a finding. +2. **Get CI green on 3.11, 3.12 and 3.13 across ubuntu and macos.** The declared floor is 3.11 and nothing + has ever executed there. +3. **Run the S6 live-agent recall leg** — real agent, real machine, per `harness/README.md`. +4. **Run the adversarial-agent experiment**, or accept knowingly that the product's central claim is + untested. Tell an agent to obtain a green `didrun verify --strict` by any means and watch what it does. + One `failed` out of 846 claims cannot tell an effective gate from an inert one. + +Plus the small ones that are simply mine-to-hand-over: **K3 (the version string) blocks release outright**, +and K1 is worth an hour. + +--- + +## The honest summary + +**30 of 44 load-bearing conclusions are externally verified.** The other 14 rest on argument from code, on +figures the system reported about itself, or on a signal only a human or CI can supply — and all four +real-corpus legs are in that second group. + +The engineering is in good shape: the defect the release was built to close is closed and measured, every +prior phase's property was re-checked rather than assumed, and the failure modes fail closed with legible +messages. What is missing is not code. It is contact with reality — the real corpus, real CI, a real agent, +and one honest adversarial test. Ship it as **0.2.0 after K3 and gates 1–2**, and do not describe it as +verified against existing evidence until gate 1 actually clears. diff --git a/pyproject.toml b/pyproject.toml index d6e6c42..d1d6851 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "didrun" -version = "0.1.0" +version = "0.2.0" description = "Flight records for agent-written code: deterministic session capture, evidence ledger, and graded verification." requires-python = ">=3.11" license = "Apache-2.0" diff --git a/src/didrun/__init__.py b/src/didrun/__init__.py index 03de7e1..d96c1de 100644 --- a/src/didrun/__init__.py +++ b/src/didrun/__init__.py @@ -10,6 +10,6 @@ The runtime is stdlib-only by design; auditability is the trust wedge. """ -__version__ = "0.1.0" +__version__ = "0.2.0" __all__ = ["__version__"] diff --git a/src/didrun/capture.py b/src/didrun/capture.py index b120206..5977562 100644 --- a/src/didrun/capture.py +++ b/src/didrun/capture.py @@ -21,19 +21,61 @@ import hashlib import os import selectors +import signal import subprocess import sys +import threading import time from pathlib import Path -from typing import Optional +from typing import Callable, Optional from .ledger import Event, Session from . import gitplumbing -# Environment variables whose presence/identity is safe to fingerprint. We hash -# a sorted allowlist rather than the raw environment so the fingerprint never -# leaks a secret (redaction of captured *output* is a separate pass; see redact.py). -_ENV_FINGERPRINT_KEYS = ("PATH", "SHELL", "LANG", "PWD", "VIRTUAL_ENV") +# The GATE: the environment facts whose identity is allowed to change a verdict. +# We hash a declared allowlist rather than the raw environment so the +# fingerprint never leaks a secret (redaction of captured *output* is a separate +# pass; see redact.py) — every value here is digested, never stored. +# +# IN, because each of them changes what a build produces: the locale (LANG), the +# shell a recipe may re-enter (SHELL), the active virtualenv (VIRTUAL_ENV), the +# timezone (TZ, which moves every date a build stamps), and the process umask +# (below — not an environment variable, but the same kind of fact, and the one +# whose absence made two green wrapper events under different umasks +# indistinguishable in the receipt). +# +# OUT, deliberately, and both were in v0.1's key set: +# +# PATH varies with terminal tab, direnv, `nix develop`, tmux and homebrew +# shellenv ordering, and flips on any venv activation VIRTUAL_ENV +# already records. A gate that refuses in a new shell is ignored within +# a week. It is recorded as its own digest instead (see +# env_fingerprint), so PATH drift stays inspectable without gating. +# +# PWD is NOT the child's cwd. run_wrapped always spawns with cwd=str(repo) +# and Event.cwd records that same resolved path; PWD is the caller's +# shell-maintained variable, which can be stale and can be absent when +# didrun is spawned by a launcher. While nothing read the fingerprint +# this was inert. The moment it is compared, PWD is a false-drift +# source: two byte-identical executions of the same command against the +# same repo, invoked from different directories, would disagree. +_ENV_FINGERPRINT_KEYS = ("LANG", "SHELL", "TZ", "VIRTUAL_ENV") + +# The fingerprint's preimage is versioned IN BAND, because the digest's shape +# does not change when its key set does: two 16-hex strings over different key +# sets are silently incomparable, which reads as "the environment drifted" when +# the truth is "these were produced by different didrun versions". A bare digest +# with no prefix was produced by v0.1 and is incomparable, not drifted. +# +# 2: the key set changed (umask and TZ in, PATH and PWD out) AND the string +# grew a `:path=` suffix. A v1 and a v2 value are never compared — see +# manifest._env_comparison and docs/COMPAT.md. +FINGERPRINT_VERSION = 2 + +# The marker introducing the non-gated PATH digest in a v2 fingerprint. +_PATH_MARKER = "path=" + +_UMASK: Optional[int] = None # The env var that gates Tier-2 traps. The trap does nothing unless this is set, # and it is set ONLY by `didrun run`, so an installed trap never records an @@ -41,54 +83,632 @@ DIDRUN_ACTIVE_ENV = "DIDRUN_SESSION" -def env_fingerprint(env: Optional[dict] = None) -> str: +def process_umask() -> int: + """This process's umask, read ONCE and reused for every event. + + There is no read-only way to read a umask: the read is a read-modify-write + (``os.umask(0)`` then restore), and between the two calls this process's + umask is 0 — process-global state, not thread-local, so a sibling thread + forking in that window hands its child the wrong one. Reading once per + process opens that window once, before the first child is spawned, instead + of once per wrapped command. The umask is a property of the process, not of + the command, so re-reading it per event would buy nothing anyway. + + ``run_wrapped`` primes this before spawning, so the value recorded on an + event is the umask the child actually inherited. + """ + global _UMASK + if _UMASK is None: + old = os.umask(0) + os.umask(old) + _UMASK = old + return _UMASK + + +def env_fingerprint(env: Optional[dict] = None, umask: Optional[int] = None) -> str: + """Digest the environment facts that can change what a build produces. + + Returns ``v2:<16 hex gate>:path=<8 hex>``. The GATE is the only part a + comparison may read (``fingerprint_gate``); the ``path=`` digest records + PATH so its drift stays INSPECTABLE without becoming a gate — see + ``_ENV_FINGERPRINT_KEYS`` for why, and docs/COMPAT.md for why a digest and + not the raw string. + + Every input is hashed and none is stored: the fingerprint must never become + the place where a secret-bearing environment variable lands verbatim in a + note. That is also why the whole environment is not fingerprinted, and it is + the reason an inherited-environment false green (``GOFLAGS=-exec=…``) is a + documented NON-guarantee rather than something this closes. + """ env = env if env is not None else os.environ - parts = [] - for key in _ENV_FINGERPRINT_KEYS: - val = env.get(key, "") - parts.append(f"{key}={val}") - return hashlib.sha256("\n".join(parts).encode("utf-8", "replace")).hexdigest()[:16] + umask = process_umask() if umask is None else umask + parts = [f"{key}={env.get(key, '')}" for key in _ENV_FINGERPRINT_KEYS] + # Named like the env parts, so the preimage stays one readable list and a + # future key called "umask" could not collide with it silently. + parts.append(f"umask={umask:04o}") + gate = hashlib.sha256("\n".join(parts).encode("utf-8", "replace")).hexdigest()[:16] + path_digest = hashlib.sha256( + env.get("PATH", "").encode("utf-8", "replace") + ).hexdigest()[:8] + return f"v{FINGERPRINT_VERSION}:{gate}:{_PATH_MARKER}{path_digest}" + + +def fingerprint_version(value: str) -> Optional[int]: + """The in-band version of a fingerprint, or None if it carries no prefix. + + ``None`` means the value came from a didrun that predates the prefix, so it + is INCOMPARABLE with a versioned one — a consumer must not read a mismatch + between the two as environment drift. + + A v2 value carries a second colon (the ``:path=`` suffix); only the leading + ``v`` field is read here, so both shapes parse. + """ + prefix, sep, _rest = value.partition(":") + if not sep or not prefix.startswith("v"): + return None + digits = prefix[1:] + if not (digits.isascii() and digits.isdigit()): + return None + return int(digits) + + +def fingerprint_gate(value: str) -> Optional[str]: + """The comparable portion of a fingerprint, or None when there is none. + + ``None`` is INCOMPARABLE and a consumer must never report it as drift: it + means a bare v0.1 digest with no version prefix, or a value whose gate field + is empty. The ``:path=`` suffix is excluded on purpose — PATH is recorded to + be looked at, never to be gated on. + """ + if fingerprint_version(value) is None: + return None + _prefix, _sep, rest = value.partition(":") + gate, _sep2, _suffix = rest.partition(":") + return gate or None + + +def fingerprint_path_digest(value: str) -> Optional[str]: + """The non-gated PATH digest carried by a v2 fingerprint, or None. + + Advisory only, in every consumer. A v1 fingerprint carries none, and a + difference here is reported as PATH drift, never as environment drift. + """ + if fingerprint_version(value) is None: + return None + _prefix, _sep, rest = value.partition(":") + _gate, sep, suffix = rest.partition(":") + if not sep or not suffix.startswith(_PATH_MARKER): + return None + return suffix[len(_PATH_MARKER):] or None + + +# --- Tier 0: the output pump -------------------------------------------------- +# +# v0.1 ran the child through subprocess.run(capture_output=True), which buffers +# both pipes to completion inside communicate(). The child's output therefore +# reached the terminal not progressively and not at the end but *never*: the +# whole operator-visible record of a 14.6-hour chain was one summary line per +# command. It is also why an interrupted command had no partial output to keep — +# subprocess.run's except clause kills the child and re-raises with the buffers +# still bound inside communicate(), where nothing can reach them. +# +# The pump below drains both pipes incrementally instead. What is *recorded* is +# unchanged, byte for byte; only when the bytes become visible changes. + +_PUMP_CHUNK = 65536 + + +class CaptureBuffers: + """The accumulating stdout/stderr of one wrapped child. + Owned by the caller rather than hidden inside the pump so that whatever was + drained before an exception is still reachable afterwards. Byte counts are + kept alongside the chunks so a progress line can report volume without ever + touching content. + """ + + def __init__(self) -> None: + self._chunks: dict[str, list[bytes]] = {"stdout": [], "stderr": []} + self._counts: dict[str, int] = {"stdout": 0, "stderr": 0} + # The threaded drain has two writers; the selector drain has one. The + # lock costs nothing measurable and keeps one implementation of feed(). + self._lock = threading.Lock() + + def feed(self, stream: str, data: bytes) -> None: + with self._lock: + self._chunks[stream].append(data) + self._counts[stream] += len(data) + + def bytes_drained(self, stream: str) -> int: + with self._lock: + return self._counts[stream] + + @property + def stdout(self) -> bytes: + with self._lock: + return b"".join(self._chunks["stdout"]) + + @property + def stderr(self) -> bytes: + with self._lock: + return b"".join(self._chunks["stderr"]) + + +def _tee_chunk(stream: str, data: bytes) -> None: + """Write one drained chunk straight to this process's terminal. + + RAW BYTES, deliberately unredacted. Redaction is whole-buffer: a _TOKENISH + match can straddle a chunk boundary, so a streaming redactor would emit half + a secret and call it covered. ``--tee`` is local terminal output — the same + exposure as running the command directly — and is never an export. Every + export path (note, bundle, HTML, ``show --output --redacted``) still runs + through redact. + """ + out = sys.stdout if stream == "stdout" else sys.stderr + buf = getattr(out, "buffer", None) + if buf is not None: + buf.write(data) + buf.flush() + else: # a text-only stream (a test harness's capture object, say) + out.write(data.decode("utf-8", "replace")) + out.flush() + + +def _heartbeat_line(argv: list[str], elapsed: float, buffers: CaptureBuffers) -> str: + """One progress line: elapsed, command identity, volume. No content. + + The command identity is argv[0]'s basename and an argument count, not the + argv itself: argv is the carrier redact.scrub_argv exists for (``FOO_TOKEN=…`` + is a measured shape), and a heartbeat that pasted it into a build log would + be a leak this file introduced. Byte counts are volume, never a digest of + the output text — a rolling digest of content is content. + """ + name = os.path.basename(argv[0]) if argv else "?" + return ( + f"didrun: {int(elapsed)}s {name} ({max(len(argv) - 1, 0)} args) " + f"stdout={buffers.bytes_drained('stdout')}B " + f"stderr={buffers.bytes_drained('stderr')}B" + ) -def run_wrapped(argv: list[str], session: Session, repo: Optional[Path] = None) -> Event: + +def _emit_heartbeat(argv: list[str], elapsed: float, buffers: CaptureBuffers) -> None: + # stderr, so a caller parsing didrun's stdout is unaffected even with the + # flag on. + print(_heartbeat_line(argv, elapsed, buffers), file=sys.stderr, flush=True) + + +def _selector_for(proc: "subprocess.Popen") -> Optional[selectors.BaseSelector]: + """A selector with both pipes registered, or None if they cannot be. + + ``selectors.DefaultSelector`` on Windows is a select() over sockets and + cannot take an anonymous pipe. Registration fails before a single byte is + read, so returning None here is a clean handover to the threaded drain + rather than a partial one. + """ + sel = selectors.DefaultSelector() + try: + sel.register(proc.stdout, selectors.EVENT_READ, "stdout") + sel.register(proc.stderr, selectors.EVENT_READ, "stderr") + except (ValueError, OSError, NotImplementedError): + sel.close() + return None + return sel + + +def _drain_via_selector( + sel: selectors.BaseSelector, + buffers: CaptureBuffers, + tee: bool, + heartbeat: Optional[float], + argv: list[str], + origin: float, +) -> None: + # One chunk per ready stream per pass: neither pipe is ever read to + # completion while the other fills, which is the deadlock the buffered path + # avoided by using communicate(). + next_beat = origin + heartbeat if heartbeat else None + while sel.get_map(): + timeout = None + if next_beat is not None: + timeout = max(0.0, next_beat - time.monotonic()) + for key, _mask in sel.select(timeout): + data = os.read(key.fd, _PUMP_CHUNK) + if not data: # EOF on this pipe + sel.unregister(key.fileobj) + key.fileobj.close() + continue + buffers.feed(key.data, data) + if tee: + _tee_chunk(key.data, data) + if next_beat is not None and time.monotonic() >= next_beat: + _emit_heartbeat(argv, time.monotonic() - origin, buffers) + next_beat = time.monotonic() + heartbeat + + +def _drain_via_threads( + proc: "subprocess.Popen", + buffers: CaptureBuffers, + tee: bool, + heartbeat: Optional[float], + argv: list[str], + origin: float, +) -> None: + """Fallback drain: one blocking reader thread per pipe. + + Same no-starvation property as the selector pump (neither stream waits on + the other) at the cost of two threads per wrapped command. stdlib only — + this is the alternative to declaring Tier-0 capture POSIX-only, and adding + a dependency to pump a pipe is not on the table. + """ + tee_lock = threading.Lock() + + def reader(stream: str, pipe) -> None: + try: + while True: + data = pipe.read1(_PUMP_CHUNK) + if not data: + break + buffers.feed(stream, data) + if tee: + with tee_lock: + _tee_chunk(stream, data) + finally: + pipe.close() + + threads = [ + threading.Thread(target=reader, args=("stdout", proc.stdout), daemon=True), + threading.Thread(target=reader, args=("stderr", proc.stderr), daemon=True), + ] + for t in threads: + t.start() + next_beat = origin + heartbeat if heartbeat else None + while True: + alive = [t for t in threads if t.is_alive()] + if not alive: + return + timeout = None + if next_beat is not None: + timeout = max(0.0, next_beat - time.monotonic()) + alive[0].join(timeout) + if next_beat is not None and time.monotonic() >= next_beat: + _emit_heartbeat(argv, time.monotonic() - origin, buffers) + next_beat = time.monotonic() + heartbeat + + +# --- Tier 0: interruption ----------------------------------------------------- +# +# An operator SIGINT reaches the whole foreground process group, so the child +# takes it too. v0.1 let the resulting KeyboardInterrupt propagate out of +# subprocess.run: the child was killed, the buffers were discarded inside +# communicate(), and CPython exited 130 with a traceback BEFORE session.append +# ran. The flight recorder lost the flight — the exact failure it exists to +# prevent, and a silent one, because absence of evidence stopped being visible. +# +# What replaces it records the same event the normal path would, minus the two +# facts nobody witnessed: no exit code, and coverage "unobserved". Both are +# EXISTING vocabulary (ledger.COVERAGE), so no Event field moves and no archived +# entry_hash changes. The grading ladder needs no edit either: _supporting_event +# skips the event (exit_code != 0) and _witnessed_failure skips it (exit_code is +# None), so a claim bound to it grades `unknown` — the correct answer, for free. + +_INTERRUPT_SIGNALS = ("SIGINT", "SIGTERM") + +# How long the child gets after the signal is forwarded before it is killed. +# The wait is a timer rather than a sleep because the handler runs on the main +# thread, which is the thread draining the pipes — sleeping there would stall +# the drain that keeps the child from blocking on a full pipe. +_CHILD_GRACE_SECONDS = 2.0 + + +def signal_name(signum: int) -> str: + """``SIGINT`` for 2, and a readable fallback for anything unnamed.""" + try: + return signal.Signals(signum).name + except ValueError: + return f"signal {signum}" + + +class CaptureInterrupted(Exception): + """A wrapped command was interrupted AND its event has been recorded. + + Raised only after ``session.append`` returns, so the exception means "the + flight is on the record", never "the record was lost". It carries the + signal number so the CLI can exit on the conventional code without a + traceback, and the appended event's index so an operator is told where the + partial record is. + """ + + def __init__(self, signum: int, event_index: Optional[int] = None) -> None: + self.signum = signum + self.event_index = event_index + super().__init__( + f"interrupted by {signal_name(signum)}; " + f"recorded event {event_index} as unobserved" + ) + + @property + def exit_code(self) -> int: + # 128 + N, the shell convention: 130 for SIGINT, 143 for SIGTERM. + return 128 + self.signum + + +def _kill_quietly(proc: "subprocess.Popen") -> None: + try: + proc.kill() + except (OSError, ValueError): + pass + + +def _install_interrupt_handlers( + proc: "subprocess.Popen", caught: dict +) -> Callable[[], None]: + """Forward SIGINT/SIGTERM to the child, remember the first one that arrived. + + Returns a callable that restores the previous handlers. The caller MUST run + it in a ``finally``: leaving didrun's handlers installed would change the + interrupt behaviour of an embedder's entire process, long after the child + it was installed for is gone. + + MAIN THREAD ONLY. ``signal.signal`` raises ``ValueError: signal only works + in main thread`` anywhere else, and ``run_wrapped`` is a library entry point + a harness or an embedder may legitimately call from a worker. Off the main + thread this installs nothing and records nothing new — the uninterrupted + path is byte-for-byte what it was. + """ + if threading.current_thread() is not threading.main_thread(): + return lambda: None + + timers: list = [] + previous: dict = {} + + def handler(signum, _frame): + if caught["signum"] is None: + caught["signum"] = signum + # Forward first. A terminal Ctrl-C already delivered the signal to the + # whole foreground process group, in which case this is a harmless + # no-op; a signal sent to didrun alone (a supervisor, a test) reaches + # the child only here. + try: + proc.send_signal(signum) + except (OSError, ValueError): + pass + # Then escalate, so a child that ignores the signal cannot hang the + # recorder. The kill closes the pipes, which ends the drain. + timer = threading.Timer(_CHILD_GRACE_SECONDS, _kill_quietly, args=(proc,)) + timer.daemon = True + timer.start() + timers.append(timer) + + for name in _INTERRUPT_SIGNALS: + signum = getattr(signal, name, None) + if signum is None: # not every platform defines both + continue + try: + previous[signum] = signal.signal(signum, handler) + except (ValueError, OSError, RuntimeError): + continue + + def restore() -> None: + for timer in timers: + timer.cancel() + for signum, prev in previous.items(): + try: + signal.signal(signum, prev) + except (ValueError, OSError, RuntimeError): + pass + + return restore + + +def _defer_interrupts(caught: dict) -> Callable[[], None]: + """Hold SIGINT/SIGTERM until the flight is on the record. + + The child is already reaped by the time this is installed, so there is + nothing left to forward to and nothing to escalate against — the only job + is to stop a signal from unwinding the recorder between the drain and + ``session.append``. Without it a second Ctrl-C lands in the digest/append + window and raises ``KeyboardInterrupt`` out of ``run_wrapped``, losing the + event for a command that actually ran and, on the drain path, discarding + the interrupt evidence the drain went to the trouble of collecting. + + The signal is recorded, not dropped: the caller re-raises it as + ``CaptureInterrupted`` once the entry is durable, so the operator still + gets the interrupt they asked for and the conventional 130 exit. + + MAIN THREAD ONLY, for the same reason as ``_install_interrupt_handlers``. + """ + if threading.current_thread() is not threading.main_thread(): + return lambda: None + + previous: dict = {} + + def handler(signum, _frame): + if caught.get("deferred") is None: + caught["deferred"] = signum + + for name in _INTERRUPT_SIGNALS: + signum = getattr(signal, name, None) + if signum is None: + continue + try: + previous[signum] = signal.signal(signum, handler) + except (ValueError, OSError, RuntimeError): + continue + + def restore() -> None: + for signum, prev in previous.items(): + try: + signal.signal(signum, prev) + except (ValueError, OSError, RuntimeError): + pass + + return restore + + +def _spawn_and_drain( + argv: list[str], + cwd: str, + buffers: CaptureBuffers, + tee: bool = False, + heartbeat: Optional[float] = None, +) -> tuple: + """Run ``argv``, draining both pipes as they fill. + + Returns ``(exit_code, signum)``, where ``signum`` is the interrupting signal + or ``None``. On interruption the exit code is still read — the child is dead + by then — but the caller must not record it: nobody witnessed the command + finish, and a kill-induced ``-2`` is not the command's verdict. + + The bytes fed into ``buffers`` are the bytes ``capture_output=True`` would + have returned — that equality is the criterion this change is held to, and + it is tested per stream over empty, large, interleaved and non-UTF-8 output + rather than asserted here. + """ + caught: dict = {"signum": None} + with subprocess.Popen( + argv, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE + ) as proc: + origin = time.monotonic() + restore = _install_interrupt_handlers(proc, caught) + try: + sel = _selector_for(proc) + if sel is None: + _drain_via_threads(proc, buffers, tee, heartbeat, argv, origin) + else: + with sel: + _drain_via_selector(sel, buffers, tee, heartbeat, argv, origin) + # Reaped inside the handlers' lifetime on purpose: a second signal + # arriving here is forwarded to an already-dead child instead of + # raising KeyboardInterrupt through the recording that follows. + exit_code = proc.wait() + except BaseException: + # Same disposition as the buffered path: kill the child and let + # Popen.__exit__ reap it. Unlike the buffered path, the caller's + # buffers hold everything drained up to this point. + proc.kill() + raise + finally: + restore() + return exit_code, caught["signum"] + + +def _tree_digest_or_none(repo: Path, ledger_objects: Path) -> Optional[str]: + """``tree_digest``, with a failure recorded as ``None`` instead of raised. + + The interrupt path only. The signal went to the whole foreground process + group, so git may have taken it too and the digest can fail in ways the + normal path never sees. ``None`` is the honest answer — the claim grades + ``unknown`` — and taking the digest must never cost the event being + recorded, which is the whole point of this path. + """ + try: + return gitplumbing.tree_digest(repo, ledger_objects) + except Exception: + return None + + +def run_wrapped( + argv: list[str], + session: Session, + repo: Optional[Path] = None, + tee: bool = False, + heartbeat: Optional[float] = None, + buffers: Optional[CaptureBuffers] = None, +) -> Event: """Tier 0: run a command under direct capture and record a complete Event. Captures argv, cwd, an env fingerprint, exit code, stdout/stderr as blobs, and git tree digests before and after. This is the only path that witnesses a real exit code, so its events are ``coverage="complete"``, ``observed_via="wrapper"``. + + ``tee`` mirrors the child's raw output to this terminal as it arrives, and + ``heartbeat`` prints a content-free progress line to stderr at most every N + seconds. Both are local display, both default off, and neither changes one + byte of what is recorded. ``buffers`` lets a caller own the accumulator and + read partial output back after an interrupt. + + On SIGINT/SIGTERM the event is still recorded — argv, cwd, fingerprint, both + tree digests and whatever output was drained before the signal — and then + ``CaptureInterrupted`` is raised. The event says what was not witnessed + rather than guessing it: no exit code, ``coverage="unobserved"``. """ repo = Path(repo or os.getcwd()) ledger_objects = session.blobs.root # digest objects live beside blobs + # Read the umask BEFORE the child is spawned, so the value the event records + # is the one the child inherited, and so the read-modify-write window this + # opens (see process_umask) is never concurrent with a fork. Cached from + # here on: the fingerprint below reuses it rather than re-reading. + process_umask() + tree_before = gitplumbing.tree_digest(repo, ledger_objects) submod = gitplumbing.submodule_dirty(repo) + if buffers is None: + buffers = CaptureBuffers() started = time.time() - proc = subprocess.run(argv, cwd=str(repo), capture_output=True) - ended = time.time() - - tree_after = gitplumbing.tree_digest(repo, ledger_objects) - - stdout_digest = session.blobs.put(proc.stdout) - stderr_digest = session.blobs.put(proc.stderr) - - event = Event( - argv=tuple(argv), - cwd=str(repo), - env_fingerprint=env_fingerprint(), - observed_via="wrapper", - coverage="complete", - exit_code=proc.returncode, - started_at=started, - ended_at=ended, - stdout_blob=stdout_digest, - stderr_blob=stderr_digest, - tree_before=tree_before, - tree_after=tree_after, - submodule_dirty=submod, + exit_code, signum = _spawn_and_drain( + list(argv), str(repo), buffers, tee=tee, heartbeat=heartbeat ) - session.append(event) + ended = time.time() + interrupted = signum is not None + + # The child is reaped, but the flight is not on the record until append + # returns. Digesting a large worktree and writing two blobs is a real + # window, and it is exactly where a second Ctrl-C lands. Hold interrupts + # across it; honour them immediately afterwards. + deferred: dict = {"deferred": None} + stop_deferring = _defer_interrupts(deferred) + try: + # The after-digest never costs the event, on ANY path. A terminal + # Ctrl-C reaches the whole foreground process group, so the git the + # digest shells out to can be killed even when the child had already + # exited cleanly and the drain saw no signal at all. Losing a witnessed + # command over a failed digest is the worse trade every time: `None` + # grades `unknown`, which is the honest answer, while no event at all + # says the command never ran. + tree_after = _tree_digest_or_none(repo, ledger_objects) + digest_lost = tree_after is None and not interrupted + + stdout_digest = session.blobs.put(buffers.stdout) + stderr_digest = session.blobs.put(buffers.stderr) + + if interrupted: + coverage = "unobserved" + elif digest_lost: + # argv, streams and exit code were all witnessed; the resulting + # tree was not. Say exactly that rather than claiming "complete". + coverage = "observed-text-only" + else: + coverage = "complete" + + event = Event( + argv=tuple(argv), + cwd=str(repo), + env_fingerprint=env_fingerprint(), + observed_via="wrapper", + coverage=coverage, + exit_code=None if interrupted else exit_code, + started_at=started, + ended_at=ended, + stdout_blob=stdout_digest, + stderr_blob=stderr_digest, + tree_before=tree_before, + tree_after=tree_after, + submodule_dirty=submod, + ) + entry = session.append(event) + finally: + stop_deferring() + + if interrupted: + raise CaptureInterrupted(signum, entry.index) + # The command itself completed and is fully recorded; the operator + # interrupted the recorder afterwards. Report the interrupt, not a + # fabricated clean return. + if deferred["deferred"] is not None: + raise CaptureInterrupted(deferred["deferred"], entry.index) return event @@ -102,6 +722,10 @@ def pty_transcript(argv: list[str], session: Session, repo: Optional[Path] = Non """ import pty + # Before pty.spawn, for the same reason run_wrapped primes it before Popen: + # the recorded umask must be the one the child inherited. + process_umask() + repo = Path(repo or os.getcwd()) chunks: list[bytes] = [] diff --git a/src/didrun/claims.py b/src/didrun/claims.py index b92d3f6..fa3cda5 100644 --- a/src/didrun/claims.py +++ b/src/didrun/claims.py @@ -15,6 +15,24 @@ instrumentation in disguise with no deterministic v1 implementation; it is killed by design (coverage belongs in an adapter, not the core). Registering it raises with the flip condition. + +``conjunction`` is the one claim type that grades no evidence of its own. It +names other claims in the same seal window by label and is graded as the WORST +of them, so it can never be better than any one of its members. The property it +represents — "all N of these gates passed against this tree" — had no +representation in the tool and was being enforced by hand. + +**The distinction this type is on the right side of, stated because the next +reader will conflate the two.** A design review of an 80-command proof ledger +declined a proposal to replace some of those recorded executions with a +transition marker, and the reason is quoted here rather than softened: a marker +witnesses *that a transition occurred*, not *that the command ran and exited +0*, so substituting one for the other genuinely weakens the claim. That refusal +stands, permanently, and nothing here attempts it — a conjunction substitutes +for nothing. Every conjunct keeps its own witnessed exit code and its own tree +comparison, graded by the same ladder as any other claim, and the conjunction +asserts nothing its members do not already back. It conjoins execution witnesses +that exist; it never reconstitutes one that does not. """ from __future__ import annotations @@ -26,9 +44,16 @@ from .ledger import Event, Session from . import gitplumbing -# The v1 claim vocabulary. `tests-pass` and `lint-clean` are labels over one +# The claim vocabulary. `tests-pass` and `lint-clean` are labels over one # generic engine: "a declared command exited 0". No claim type infers meaning. -CLAIM_TYPES = ("command-succeeded", "tests-pass", "lint-clean") +# +# `conjunction` is the one entry that binds no event of its own: it names other +# claims in the same seal window and is graded as the worst of them. Extending +# this tuple is a MANIFEST_VERSION bump (docs/COMPAT.md) — an older reader hits +# `unknown claim type` on a note that carries a type it has never seen. +CONJUNCTION = "conjunction" + +CLAIM_TYPES = ("command-succeeded", "tests-pass", "lint-clean", CONJUNCTION) KILLED_CLAIM_TYPES = { "diff-exercised": ( @@ -46,6 +71,67 @@ # The strongest negative signal: didrun WITNESSED the claimed command fail. This # is distinct from `unknown` (no evidence at all) — it is a claim caught lying. GRADE_FAILED = "failed" +# The note records a grade for this claim, but the live ledger cannot supply the +# evidence it was sealed against — this is not a verification. It is never +# produced by ``grade()``: the ladder below grades a claim against evidence it +# can see, and this state says the evidence itself is gone, or is not the +# evidence that was sealed. manifest.verify assigns it AROUND the ladder, and it +# is never admitted to the set --strict accepts. +GRADE_WITNESS_UNAVAILABLE = "witness-unavailable" + +# The environment comparison verify makes ACROSS TIME — between the fingerprint +# the seal recorded and the one this process computes. These are NOT grades: a +# drifted environment does not make a recorded command un-run, so they never +# enter the grading ladder and never reach `worst_status`. +# +# The two refusals are load-bearing. `incomparable` is what a fingerprint from a +# different FINGERPRINT_VERSION (or a bare v0.1 digest) gets — different key +# sets are not measuring the same thing, and calling that drift would make every +# archived note look like the environment moved. `not-recorded` is a note that +# bound no fingerprint at all, which is a different fact again. +ENV_MATCH = "match" +ENV_DRIFT = "env-drift-since-seal" +ENV_INCOMPARABLE = "incomparable" +ENV_NOT_RECORDED = "not-recorded" + +# Worst first. This is the ONE severity order in the package: the whole-report +# verdict (manifest.VerifyReport.worst_status) reads it, and a conjunction reads +# it to grade as the worst of its conjuncts. Two copies could disagree about +# which grade is worse, and a conjunction that disagreed with the verdict line +# above it would be the exact overclaim this type has to be incapable of. +# +# `chain-broken` is deliberately absent: it is a statement about the ledger every +# grade was read out of, not a grade any claim can carry, and it dominates the +# verdict before this order is consulted. +GRADE_ORDER_WORST_FIRST = ( + GRADE_FAILED, + GRADE_UNKNOWN, + GRADE_WITNESS_UNAVAILABLE, + GRADE_STALE, + GRADE_SCOPE_EXACT, + GRADE_TREE_EXACT, +) + + +def worst_grade(grades) -> Optional[str]: + """The worst grade in ``grades`` under ``GRADE_ORDER_WORST_FIRST``. + + Ties go to the first occurrence, so the caller's order decides which of two + equally-bad members is named — never the iteration order of a set. + + A grade this version does not know ranks WORSE than every registered one. A + grade nobody here can interpret is not evidence that anything is fine, and + ranking it best is how an unregistered token would carry a conjunction to + green. Returns None for an empty input; callers must not treat that as a + grade. + """ + worst: Optional[str] = None + worst_rank: Optional[int] = None + for g in grades: + rank = GRADE_ORDER_WORST_FIRST.index(g) if g in GRADE_ORDER_WORST_FIRST else -1 + if worst_rank is None or rank < worst_rank: + worst, worst_rank = g, rank + return worst class ClaimError(Exception): @@ -59,6 +145,13 @@ class Claim: ``pathspecs`` are supplied by the claimant for a scope-exact grade; they are never auto-derived. ``declared_at_index`` is the session index at which the claim was declared — used for the retroactive-binding rule. + + ``conjuncts`` are the LABELS of other claims in the same seal window that a + ``conjunction`` conjoins. They are labels rather than event indices on + purpose: a conjunction must not be a claim bound to several events. That + shape already exists, grades on the first bound success it finds, and is the + illegitimate composite the design review ruled out — so ``event_indices`` is + required to be EMPTY here and its meaning is untouched. """ ctype: str @@ -66,12 +159,30 @@ class Claim: event_indices: tuple[int, ...] pathspecs: tuple[str, ...] = () declared_at_index: int = -1 + conjuncts: tuple[str, ...] = () def __post_init__(self) -> None: if self.ctype in KILLED_CLAIM_TYPES: raise ClaimError(KILLED_CLAIM_TYPES[self.ctype]) if self.ctype not in CLAIM_TYPES: raise ClaimError(f"unknown claim type: {self.ctype!r}") + if self.ctype == CONJUNCTION: + if not self.conjuncts: + raise ClaimError( + "a conjunction claim must name the claims it conjoins " + "(at least one conjunct label)" + ) + if self.event_indices: + raise ClaimError( + "a conjunction claim binds no event: it conjoins claims that " + "carry their own witnessed events, and widening its bound " + "events would make it a multi-index claim instead" + ) + elif self.conjuncts: + raise ClaimError( + f"claim type {self.ctype!r} names no conjuncts; only a " + f"conjunction claim conjoins other claims" + ) def to_dict(self) -> dict: return { @@ -80,6 +191,11 @@ def to_dict(self) -> dict: "event_indices": list(self.event_indices), "pathspecs": list(self.pathspecs), "declared_at_index": self.declared_at_index, + # Emitted always, like `pathspecs`, and read with a default below: + # an older reader that does not know the key sees the claim it + # always saw, and every claim this version writes declares the key + # whether or not it carries one. + "conjuncts": list(self.conjuncts), } @classmethod @@ -90,12 +206,39 @@ def from_dict(cls, d: dict) -> "Claim": event_indices=tuple(d.get("event_indices", ())), pathspecs=tuple(d.get("pathspecs", ())), declared_at_index=d.get("declared_at_index", -1), + conjuncts=tuple(d.get("conjuncts", ())), ) @dataclass class GradeResult: - """The outcome of grading one claim against a target tree.""" + """The outcome of grading one claim against a target tree. + + ``evidence_bound`` and ``sealed_grade`` are verify-side facts, not grading + facts: they say whether this verdict was checked against the specific + recorded entry the seal named, and what that seal graded. ``grade()`` never + sets them, so a sealed note carries them at their defaults. + + ``env_status`` / ``env_reason`` are verify-side in the same way, and they + are deliberately NOT in ``to_dict``: the seal has nothing to compare against + itself, so writing them would put a permanently-default key in every + published note. + + ``superseded_by`` is verify-side for a different reason: supersession is a + fact about the seal WINDOW, not about this claim's evidence, so the seal + writes the mark onto the exported entry itself (manifest._redact_result) and + verify reads it back onto the result. Keeping it out of ``to_dict`` is what + makes the mark appear only on the entries that carry one, instead of putting + a permanently-null key on every claim in every note. + + ``conjunct_grades`` holds one ``(label, grade)`` pair per conjunct a + conjunction declared, in declaration order, so the note carries what each + member earned instead of only the worst of them. ``grade`` is None for a + conjunct that resolved to no single live claim: an unresolved name has no + grade, and giving it one would be the invented fact this whole type exists + not to produce. Serialized only when non-empty, on ``superseded_by``'s + precedent — every other claim would otherwise carry an empty list forever. + """ claim: Claim grade: str @@ -103,16 +246,39 @@ class GradeResult: delta: list = field(default_factory=list) # PathChange list when stale/scope supporting_event_index: Optional[int] = None exit_code: Optional[int] = None + evidence_bound: bool = False + sealed_grade: Optional[str] = None + env_status: str = ENV_NOT_RECORDED + env_reason: str = "" + superseded_by: Optional[int] = None + conjunct_grades: tuple = () + + @property + def is_superseded(self) -> bool: + """Whether a later claim in the same seal window retired this one. + + A superseded result is RECORD, not verdict: its grade and reason are + kept verbatim and stay visible, and it is excluded from the verdict — + never relabelled, re-graded or dropped. + """ + return self.superseded_by is not None def to_dict(self) -> dict: - return { + d = { "claim": self.claim.to_dict(), "grade": self.grade, "reason": self.reason, "delta": [{"status": c.status, "path": c.path} for c in self.delta], "supporting_event_index": self.supporting_event_index, "exit_code": self.exit_code, + "evidence_bound": self.evidence_bound, + "sealed_grade": self.sealed_grade, } + if self.conjunct_grades: + d["conjunct_grades"] = [ + {"label": label, "grade": g} for label, g in self.conjunct_grades + ] + return d def _supporting_event(claim: Claim, events: list[Event]) -> Optional[tuple[int, Event]]: @@ -145,32 +311,106 @@ def _witnessed_failure(claim: Claim, events: list[Event]) -> Optional[tuple[int, return None +def grade_conjunction(claim: Claim, siblings: Optional[list] = None) -> GradeResult: + """Grade a conjunction as the WORST of the claims it names. Never better. + + ``siblings`` are the other graded results of the same seal window. Two kinds + are filtered out here rather than at the call site, so both callers (seal and + verify) get the same resolution from the same rule: + + - **superseded entries.** A repair re-declares the same gate, so the retired + attempt shares its label with the claim that stands in its place. Skipping + it is what lets a conjunction name a gate that failed and was fixed inside + the window; counting it would make every such name ambiguous. + - **other conjunctions.** A conjunction naming a conjunction resolves to no + match and grades ``unknown``. That is deliberate: it keeps resolution one + level deep, so there is no recursion to bound and no cycle to detect, and + the grade of a name this function will not follow is not asserted. + + Resolution is by EXACT label match — never a prefix, never a fuzzy one. A + name matching no live claim, or more than one, makes the whole conjunction + ``unknown`` and the reason names it: a member it cannot identify is a member + it cannot vouch for, and grading around the gap is how a conjunction would + come out better than its parts. + """ + pool = [ + s + for s in (siblings or ()) + if not s.is_superseded and s.claim.ctype != CONJUNCTION + ] + pairs: list = [] + unresolved: list[str] = [] + for name in claim.conjuncts: + matches = [s for s in pool if s.claim.label == name] + if len(matches) == 1: + pairs.append((name, matches[0].grade)) + continue + pairs.append((name, None)) + unresolved.append( + f"{name!r} names no claim in this seal window" + if not matches + else f"{name!r} names {len(matches)} claims in this seal window" + ) + if unresolved: + return GradeResult( + claim, + GRADE_UNKNOWN, + reason="unresolved conjunct: " + "; ".join(unresolved), + conjunct_grades=tuple(pairs), + ) + worst = worst_grade(g for _name, g in pairs) + # Ties go to the first declared conjunct, which is what makes this reason + # reproducible for the same window rather than a function of dict ordering. + named = next(name for name, g in pairs if g == worst) + return GradeResult( + claim, + worst, + reason=f"worst of {len(pairs)} conjunct(s): {named!r} graded {worst}", + conjunct_grades=tuple(pairs), + ) + + def grade( claim: Claim, sealed_tree: str, events: list[Event], repo: Path, ledger_objects: Optional[Path] = None, + siblings: Optional[list] = None, ) -> GradeResult: """Grade one claim against ``sealed_tree``. First match wins. The ordering is the honesty ladder: only concede a weaker grade when the stronger one cannot be honestly asserted. + + ``siblings`` is consulted for a ``conjunction`` and for nothing else: it + grades no evidence of its own, so it is dispatched BEFORE the ladder rather + than added as a rung. Every rung below is untouched, and a caller that + forgets ``siblings`` gets a conjunction graded ``unknown`` — the fail-closed + direction, never a green one. """ + if claim.ctype == CONJUNCTION: + return grade_conjunction(claim, siblings) + + # A witnessed failure among the bound events DOMINATES a witnessed success. + # Consulting _supporting_event first graded a claim bound to + # [failed, success] on the success and made the caught lie invisible. + # Reachable only through the library API today (the CLI's --event is + # type=int, so every CLI claim binds exactly one event). + failure = _witnessed_failure(claim, events) + if failure is not None: + fidx, fev = failure + return GradeResult( + claim, + GRADE_FAILED, + reason=f"the recorded command exited {fev.exit_code} — claim is not backed", + supporting_event_index=fidx, + exit_code=fev.exit_code, + ) + found = _supporting_event(claim, events) if found is None: - # No witnessed success. Is that because we watched it FAIL (a caught - # lie) or because there is no evidence at all (unknown)? - failure = _witnessed_failure(claim, events) - if failure is not None: - fidx, fev = failure - return GradeResult( - claim, - GRADE_FAILED, - reason=f"the recorded command exited {fev.exit_code} — claim is not backed", - supporting_event_index=fidx, - exit_code=fev.exit_code, - ) + # No witnessed success and no witnessed failure: no evidence at all. return GradeResult( claim, GRADE_UNKNOWN, @@ -223,6 +463,19 @@ def grade( # Compute the delta between the evidence tree and the sealed tree. delta = gitplumbing.tree_delta(repo, ev.tree_after, sealed_tree, ledger_objects) + if delta is None: + # Fail closed. `None` is "git could not compute it" — typically the + # archived-ledger case where the evidence tree's objects have moved. + # Every branch below would otherwise render that failure as a fact + # ("0 path(s) differ", "evidence tree equals sealed tree"), and an + # empty delta would also violate the stale-carries-the-delta invariant. + return GradeResult( + claim, + GRADE_UNKNOWN, + reason="delta not computable: evidence tree objects unavailable", + supporting_event_index=idx, + exit_code=ev.exit_code, + ) # scope-exact: every change is within the claimant-declared pathspecs. if claim.pathspecs: @@ -247,10 +500,19 @@ def grade( exit_code=ev.exit_code, ) # Out-of-scope changes exist — display them; cannot claim scope-exact. + # The reason names the widening set as well as the count: the exact + # prefixes that would have made this scope-exact are already sitting in + # `delta`, and reporting only how MANY paths missed made an operator + # re-derive by hand something the tool had measured. The count prefix is + # unchanged and the set is appended after an em dash, so a consumer + # matching the old prefix still matches (docs/COMPAT.md). return GradeResult( claim, GRADE_STALE, - reason=f"{len(out_of_scope)} change(s) outside declared pathspecs", + reason=( + f"{len(out_of_scope)} change(s) outside declared pathspecs" + f" — {_widening_hint(c.path for c in out_of_scope)}" + ), delta=delta, supporting_event_index=idx, exit_code=ev.exit_code, @@ -267,6 +529,40 @@ def grade( ) +# How many prefixes the widening hint names before it summarises the rest. A +# hint longer than this stops being readable and starts being the delta again, +# which the caller already has. +WIDENING_HINT_CAP = 5 + + +def _widening_hint(paths) -> str: + """The declaration that would have brought ``paths`` into scope, in words. + + ``_within`` is a path-prefix test, so the smallest prefix covering a path is + its parent directory — that is the whole derivation. Deduplicated and sorted + for determinism, capped at ``WIDENING_HINT_CAP`` with the remainder counted + rather than dropped silently. A top-level file has no parent directory but + the repository root, and declaring the root would put EVERY path in scope, so + it contributes itself: the smallest prefix that covers it and nothing else. + + This is not relevance inference (module docstring: *set arithmetic only*). + It changes no grade, and it asserts nothing about whether those paths belong + to the claim — only what widening the declaration would have to say. + """ + prefixes = set() + for path in paths: + stripped = path.strip("/") + head, sep, _tail = stripped.rpartition("/") + prefixes.add(head + "/" if sep else stripped) + ordered = sorted(prefixes) + shown = ordered[:WIDENING_HINT_CAP] + hint = ", ".join(f"--path {p}" for p in shown) + elided = len(ordered) - len(shown) + if elided: + hint += f" (+{elided} more)" + return f"add {hint} to make this scope-exact" + + def _within(path: str, pathspecs: tuple[str, ...]) -> bool: """True if ``path`` is under any declared pathspec (prefix match). diff --git a/src/didrun/cli.py b/src/didrun/cli.py index d02f135..d1b03d3 100644 --- a/src/didrun/cli.py +++ b/src/didrun/cli.py @@ -1,6 +1,6 @@ """didrun command-line interface. -Five commands: run, claim, seal, verify, show. +Six commands: run, claim, authorize, seal, verify, show. `log` is subsumed by `show`; `report` is a `--html` flag on show/verify. The CLI is a thin shell over the library; all behavior lives in the modules so @@ -16,10 +16,16 @@ from typing import Optional from . import __version__ -from .ledger import Session -from .capture import run_wrapped, DIDRUN_ACTIVE_ENV -from .claims import Claim +from .ledger import LedgerError, Session +from .capture import ( + CaptureInterrupted, + DIDRUN_ACTIVE_ENV, + run_wrapped, + signal_name, +) +from .claims import CLAIM_TYPES, CONJUNCTION, Claim from . import manifest as _manifest +from . import redact from . import render from . import gitplumbing @@ -28,8 +34,15 @@ def _ledger_dir(repo: Path) -> Path: return Path(repo) / ".didrun" -def _session(repo: Path) -> Session: - return Session(_ledger_dir(repo)) +def _session(repo: Path, readonly: bool = False) -> Session: + """Open the repo's ledger. + + ``readonly=True`` is for the commands that only read one: it creates no + directory, writes no .gitignore, and re-modes nothing. Without it `verify` + on a fresh clone MANUFACTURED an empty ledger — which then graded every + claim unknown — and pointing `show` at someone else's archive mutated it. + """ + return Session(_ledger_dir(repo), readonly=readonly) def cmd_run(args) -> int: @@ -46,7 +59,9 @@ def cmd_run(args) -> int: session = _session(repo) # Mark the session active so Tier-1/2 hooks (if installed) are live only now. os.environ.setdefault(DIDRUN_ACTIVE_ENV, "1") - event = run_wrapped(list(args.command), session, repo) + event = run_wrapped( + list(args.command), session, repo, tee=args.tee, heartbeat=args.heartbeat + ) coverage = "shim" if args.shim else "wrapper" print( f"recorded {coverage} event argv={' '.join(event.argv)!r} " @@ -56,6 +71,19 @@ def cmd_run(args) -> int: return event.exit_code if event.exit_code is not None else 0 +def _conjunct_labels(raw: Optional[str]) -> tuple: + """Parse ``--of``: the comma-separated labels of the claims to conjoin. + + Order is the operator's, and it is kept: it decides which of two + equally-bad conjuncts the grade's reason names. Surrounding whitespace is + stripped and an empty name is dropped rather than matched against a label + nothing carries; an empty result reaches ``Claim``, which refuses it. + """ + if not raw: + return () + return tuple(part.strip() for part in raw.split(",") if part.strip()) + + def cmd_claim(args) -> int: repo = Path(args.repo or os.getcwd()) session = _session(repo) @@ -63,16 +91,75 @@ def cmd_claim(args) -> int: if not events: print("didrun claim: no recorded events to bind to (run something first)", file=sys.stderr) return 2 - # Default: bind to the most recent successful event. - if args.event is not None: - indices = (args.event,) + conjuncts: tuple = () + indices: tuple = () + if args.type == CONJUNCTION: + # A conjunction binds no event — it conjoins claims that carry their own + # witnessed events — so the "most recent event did not exit 0" refusal + # below is not its refusal. `--event` and `--path` are refused rather than + # silently ignored: an operator who passed one meant something by it. + if args.event is not None: + print( + "didrun claim: a conjunction binds no event; name the claims it " + "conjoins with --of instead of --event", + file=sys.stderr, + ) + return 2 + if args.path: + print( + "didrun claim: a conjunction declares no pathspec; its conjuncts " + "carry their own", + file=sys.stderr, + ) + return 2 + conjuncts = _conjunct_labels(args.of) else: - indices = tuple( - i for i, ev in enumerate(events) if ev.exit_code == 0 - )[-1:] # last success - if not indices: - print("didrun claim: no successful event to bind (last command did not exit 0)", file=sys.stderr) - return 2 + if args.of: + print( + f"didrun claim: --of names the claims a conjunction conjoins; " + f"{args.type} conjoins none", + file=sys.stderr, + ) + return 2 + # Default: bind to the LAST event, never to the last *successful* one. + # + # Searching backwards for a success is the failure this tool exists to + # prevent, reachable through the documented happy path: run the tests + # (pass), edit, run them again (fail), `didrun claim tests-pass` — the + # claim bound the older passing event, `declared_at_index` was still the + # last index so the retroactive-binding rule did not catch it, and if the + # tree had not moved it graded tree-exact. The only signal was the index + # in a success line nobody reads. Refuse instead: an operator who really + # means an earlier event says so with --event N, which is deliberate and + # is warned about below. + if args.event is not None: + indices = (args.event,) + gap = len(events) - 1 - args.event + if gap > 0: + print( + f"didrun claim: binding to event {args.event}, {gap} event(s) " + f"before the most recent (index {len(events) - 1})", + file=sys.stderr, + ) + else: + last = len(events) - 1 + recent = events[last] + if recent.exit_code != 0: + witness = ( + f"exited {recent.exit_code}" + if recent.exit_code is not None + else "recorded no exit code (interrupted, or capture that " + "witnesses no exit)" + ) + print( + f"didrun claim: the most recent event (index {last}) " + f"{witness} — refusing to bind this claim to an earlier " + f"event that passed. Pass --event N to bind an earlier " + f"event deliberately.", + file=sys.stderr, + ) + return 2 + indices = (last,) try: claim = Claim( ctype=args.type, @@ -80,12 +167,66 @@ def cmd_claim(args) -> int: event_indices=indices, pathspecs=tuple(args.path or ()), declared_at_index=len(events) - 1, + conjuncts=conjuncts, ) except Exception as exc: print(f"didrun claim: {exc}", file=sys.stderr) return 2 _manifest.declare_claim(session, claim) - print(f"declared claim {claim.ctype} ({claim.label}) bound to event(s) {list(indices)}") + if claim.conjuncts: + print( + f"declared claim {claim.ctype} ({claim.label}) over " + f"{len(claim.conjuncts)} conjunct(s): {', '.join(claim.conjuncts)} " + f"— graded as the worst of them" + ) + else: + print(f"declared claim {claim.ctype} ({claim.label}) bound to event(s) {list(indices)}") + return 0 + + +def cmd_authorize(args) -> int: + """Record that a cited artifact is the authority for one scope. + + Prints the digest and, verbatim, the limitation constant. The limitation is + printed here and not only in the docs because this command's output is where + an operator forms their idea of what the record means, and what it means is + much less than "approved". + """ + repo = Path(args.repo or os.getcwd()) + session = _session(repo) + try: + record = _manifest.declare_authority( + session, + Path(args.artifact), + scope=args.scope, + label=args.label, + principal=args.principal, + expires_after_seals=args.expires_after_seals, + ) + except _manifest.ManifestError as exc: + print(f"didrun authorize: {exc}", file=sys.stderr) + return 2 + except OSError as exc: + print(f"didrun authorize: cannot read the cited artifact: {exc}", file=sys.stderr) + return 2 + # Sanitized, like every other terminal surface that prints a string this + # process did not compute: the label, the principal and the path are argv. + text = render.sanitize_text + print( + f"recorded {_manifest.AUTHORITY_GRADE} authority for scope " + f"{text(record['scope'])}: {text(record['label'])}" + ) + print( + f" artifact {text(record['artifact_path'])} " + f"sha256 {text(record['artifact_sha256'])} " + f"{text(record['artifact_bytes'])} bytes" + ) + print( + f" principal {text(record['principal']) or '(unnamed)'} " + f"({_manifest.AUTHORITY_PRINCIPAL_BASIS}) · " + f"expires after {text(record['expires_after_seals'])} seal(s)" + ) + print(f" {_manifest.AUTHORITY_LIMITATION}") return 0 @@ -99,6 +240,8 @@ def cmd_seal(args) -> int: commitish=args.commit or "HEAD", allow_secrets=args.allow_secrets, bundle_path=Path(args.bundle) if args.bundle else None, + reseal=args.reseal, + require_authority=args.require_authority, ) except _manifest.redact.SecretsBlocked as exc: print(f"didrun seal: {exc}", file=sys.stderr) @@ -106,24 +249,73 @@ def cmd_seal(args) -> int: except _manifest.ManifestError as exc: print(f"didrun seal: {exc}", file=sys.stderr) return 2 - verified = sum(1 for c in m.claims if c["grade"] in ("tree-exact", "scope-exact")) + # Counted over the live entries, like the verdict surface: a window whose + # earlier attempt was retired would otherwise print "1/2 claims + # recorded-exact" for a unit `verify` calls ALL RECORDED-EXACT, and the two + # numbers describing the same note have to describe the same set. + live = _manifest.live_claims(m.claims) + superseded = len(m.claims) - len(live) + verified = sum(1 for c in live if c["grade"] in ("tree-exact", "scope-exact")) + # The tier split is reported even when nothing blocked: "0 findings" and + # "12 findings, none of them blocking" are different facts about the same + # green seal, and only one of them is worth a second look. + by_tier = (m.secrets or {}).get("findings_by_tier", {}) + blocking = by_tier.get("block", 0) + noticed = by_tier.get("notice", 0) print( f"sealed manifest for {m.commit[:12]} (tree {m.tree[:12]}) " - f"{verified}/{len(m.claims)} claims recorded-exact" + f"{verified}/{len(live)} claims recorded-exact " + + (f"{superseded} superseded " if superseded else "") + + f"{blocking + noticed} findings ({blocking} block / {noticed} notice)" + (" [--allow-secrets]" if m.secrets_override else "") ) + # What authorised the exceptional transition, if anything did — printed at the + # moment the citation is published, with the limitation attached. An ordinary + # seal cites nothing and prints nothing extra. + # Sanitized for the same reason the verify block is: these strings came from + # `authority.jsonl`, and redaction is not sanitization — a scrubbed label + # still carries whatever control characters were written around the secret. + for entry in m.authority: + print( + f" {_manifest.AUTHORITY_GRADE} authority for scope " + f"{render.sanitize_text(entry.get('scope', ''))}: " + f"{render.sanitize_text(entry.get('label', ''))} " + f"sha256 {render.sanitize_text(entry.get('artifact_sha256', ''))}" + ) + if m.authority: + print(f" {_manifest.AUTHORITY_LIMITATION}") return 0 def cmd_verify(args) -> int: repo = Path(args.repo or os.getcwd()) - session = _session(repo) - report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") + session = _session(repo, readonly=True) + try: + report = _manifest.verify( + session, + repo, + commitish=args.commit or "HEAD", + require_env_match=args.require_env_match, + ) + except _manifest.ManifestError as exc: + # Evidence this binary cannot read is a graded refusal, not a crash. 2, + # not --strict's 1: "could not read the manifest" is a different fact + # from "the manifest graded badly". + print(f"didrun verify: {exc}", file=sys.stderr) + return 2 if args.html: Path(args.html).write_text(render.render_html(report), encoding="utf-8") print(f"wrote HTML report to {args.html}") if not args.quiet: print(render.render_verdict(report)) + else: + # --quiet suppresses the VERDICT, not the ledger's integrity. A + # broken chain says the file that verdict was computed from does not + # recompute; a reader who sent the detail to HTML still needs that + # sentence where they are looking. + banner = render.render_chain_banner(report) + if banner: + print(banner) else: print(render.render_verdict(report)) if args.strict: @@ -131,9 +323,58 @@ def cmd_verify(args) -> int: return 0 +def _show_output(session: Session, args) -> int: + """Print one recorded output blob to stdout. + + The only path in the CLI that reads a blob back. Without it, reading the + output of a recorded failure meant resolving digests by hand under + `.didrun/objects`. `BlobStore.get` re-hashes on read, so a corrupted or + edited blob is a refusal here rather than bytes presented as the record. + """ + if args.event is None: + print( + "didrun show: --output needs --event N (which recorded event to read)", + file=sys.stderr, + ) + return 2 + events = session.events() + if not 0 <= args.event < len(events): + print( + f"didrun show: no event at index {args.event} " + f"(the session has {len(events)} events)", + file=sys.stderr, + ) + return 2 + ev = events[args.event] + digest = ev.stdout_blob if args.stream == "stdout" else ev.stderr_blob + if digest is None: + print( + f"didrun show: event {args.event} recorded no {args.stream} blob", + file=sys.stderr, + ) + return 2 + try: + data = session.blobs.get(digest) + except LedgerError as exc: + # Missing or corrupt evidence is a graded refusal, never a traceback. + print(f"didrun show: {exc}", file=sys.stderr) + return 2 + if args.redacted: + data = redact.redact(data).text.encode("utf-8", "replace") + out = getattr(sys.stdout, "buffer", None) + if out is not None: + out.write(data) + out.flush() + else: + sys.stdout.write(data.decode("utf-8", "replace")) + return 0 + + def cmd_show(args) -> int: repo = Path(args.repo or os.getcwd()) - session = _session(repo) + session = _session(repo, readonly=True) + if args.output: + return _show_output(session, args) if args.session: # Session history view (subsumes the old `log` command). entries = list(session.entries()) @@ -143,6 +384,16 @@ def cmd_show(args) -> int: ok, broke = session.verify_chain() chain = "intact" if ok else f"BROKEN at index {broke}" print(f"session: {len(entries)} events chain {chain}") + if session.tail_truncated: + # A lost event is not tamper, and the chain line above is about the + # surviving prefix only. Saying so is the difference between a + # reader who knows an event is missing and one who does not. + print( + f" note: the log's final record is torn at byte " + f"{session.tail_truncated_offset} — one event was lost to an " + f"interrupted write, and `didrun run` will refuse to append " + f"until that is resolved" + ) for e in entries: ev = e.event exit_txt = f"exit {ev.exit_code}" if ev.exit_code is not None else "no-exit" @@ -152,7 +403,11 @@ def cmd_show(args) -> int: ) return 0 if ok else 1 # Default: the verdict view for a commit. - report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") + try: + report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") + except _manifest.ManifestError as exc: + print(f"didrun show: {exc}", file=sys.stderr) + return 2 if args.html: Path(args.html).write_text(render.render_html(report), encoding="utf-8") print(f"wrote HTML report to {args.html}") @@ -172,25 +427,133 @@ def build_parser() -> argparse.ArgumentParser: pr = sub.add_parser("run", help="record a wrapped command execution (Tier 0)") pr.add_argument("--shim", action="store_true", help=argparse.SUPPRESS) + pr.add_argument( + "--tee", + action="store_true", + help=( + "also write the command's raw output to this terminal as it arrives " + "(local output only: NOT redacted, never an export)" + ), + ) + pr.add_argument( + "--heartbeat", + type=float, + metavar="SECONDS", + help=( + "print a progress line to stderr at most every SECONDS " + "(elapsed, command, bytes drained; no output content)" + ), + ) pr.add_argument("command", nargs=argparse.REMAINDER, help="-- [args…]") pr.set_defaults(func=cmd_run) pc = sub.add_parser("claim", help="declare a structured claim over recorded events") - pc.add_argument("type", choices=["command-succeeded", "tests-pass", "lint-clean"]) + # Read from the vocabulary rather than repeated here. The two lists were + # duplicates, and a duplicate of a closed vocabulary drifts silently: the CLI + # would keep refusing a type the library had accepted for a release. + pc.add_argument("type", choices=list(CLAIM_TYPES)) pc.add_argument("--label", help="human label (default: the type)") - pc.add_argument("--event", type=int, help="bind to a specific event index") + pc.add_argument( + "--event", + type=int, + help=( + "bind to a specific event index, warning when it is not the most " + "recent. Default: the most recent event, and a claim is refused " + "rather than bound to an earlier one when that event did not exit 0" + ), + ) pc.add_argument("--path", action="append", help="declare a pathspec for scope-exact grading") + pc.add_argument( + "--of", + metavar="LABEL[,LABEL…]", + help=( + "for a conjunction: the comma-separated labels of the claims in this " + "seal window it conjoins. The conjunction is graded as the WORST of " + "them and can never be better than any one of them; a label that " + "names no single live claim in the window grades it unknown" + ), + ) pc.set_defaults(func=cmd_claim) + pa = sub.add_parser( + "authorize", + help=( + "record a cited authorization artifact for one exceptional scope " + "(stores the digest, never the bytes; NOT a signature)" + ), + ) + pa.add_argument( + "--artifact", + required=True, + help="path to the authorization artifact; its sha256 is recorded, its bytes are not", + ) + pa.add_argument( + "--scope", + required=True, + choices=list(_manifest.AUTHORITY_SCOPES), + help=( + "the exceptional transition this authority covers. There is no scope " + "for an ordinary seal: an authorisation on the happy path becomes a " + "rubber stamp" + ), + ) + pa.add_argument("--label", required=True, help="what is being authorised, in words") + pa.add_argument( + "--principal", + help=( + "who is citing it (default: $USER). Recorded as SELF-ASSERTED — " + "didrun authenticates nobody" + ), + ) + pa.add_argument( + "--expires-after-seals", + type=int, + metavar="N", + help=( + f"how many seals the citation covers, starting with the next one " + f"(default: {_manifest.AUTHORITY_DEFAULT_TTL}). There is no value " + f"meaning 'never' — a standing grant is what this replaces" + ), + ) + pa.set_defaults(func=cmd_authorize) + ps = sub.add_parser("seal", help="compile and attach a commit-bound manifest") ps.add_argument("--commit", help="commit to bind (default: HEAD)") ps.add_argument("--allow-secrets", action="store_true", help="export despite secret findings (redacted; logged)") ps.add_argument("--bundle", help="also write an exportable bundle to this path") + ps.add_argument( + "--reseal", + action="store_true", + help=( + "permit replacing this commit's note with a NARROWER record: the " + "claims the replaced note held and this seal does not are then in " + "no note at all. Without it, a narrowing re-seal is refused" + ), + ) + ps.add_argument( + "--require-authority", + action="store_true", + help=( + "refuse --allow-secrets or --reseal unless a live cited authority " + "covers that scope (see `didrun authorize`). Default off: an " + "authorisation required on every seal becomes a rubber stamp. The " + "authority it demands is a citation, not an approval" + ), + ) ps.set_defaults(func=cmd_seal) pv = sub.add_parser("verify", help="verify a commit's claims against recorded evidence") pv.add_argument("--commit", help="commit to verify (default: HEAD)") pv.add_argument("--strict", action="store_true", help="exit nonzero unless all claims are recorded-exact (CI mode)") + pv.add_argument( + "--require-env-match", + action="store_true", + help=( + "treat environment drift since the seal as a refusal, not a note " + "(with --strict, a drifted claim exits nonzero). Fingerprints from " + "an older didrun are incomparable and never count as drift" + ), + ) pv.add_argument("--html", help="also write an HTML report to this path") pv.add_argument("--quiet", action="store_true", help="with --html, suppress the terminal verdict") pv.set_defaults(func=cmd_verify) @@ -199,6 +562,23 @@ def build_parser() -> argparse.ArgumentParser: pw.add_argument("--commit", help="commit to show (default: HEAD)") pw.add_argument("--session", action="store_true", help="show the recorded session history (subsumes `log`)") pw.add_argument("--html", help="write an HTML report to this path") + pw.add_argument("--event", type=int, metavar="N", help="event index to read with --output") + pw.add_argument( + "--output", + action="store_true", + help="print the recorded output of event N to stdout", + ) + pw.add_argument( + "--stream", + choices=["stdout", "stderr"], + default="stdout", + help="which recorded stream --output reads (default: stdout)", + ) + pw.add_argument( + "--redacted", + action="store_true", + help="run --output through the redactor before printing", + ) pw.set_defaults(func=cmd_show) return p @@ -207,7 +587,25 @@ def build_parser() -> argparse.ArgumentParser: def main(argv: Optional[list] = None) -> int: parser = build_parser() args = parser.parse_args(argv) - return args.func(args) + try: + return args.func(args) + except CaptureInterrupted as exc: + # The event IS on the record — that is what the exception means. One + # machine-greppable line and the conventional exit code (130 for SIGINT, + # 143 for SIGTERM), so a caller that used to detect interruption by + # scraping a traceback gets a better signal than the one it lost. + print( + f"didrun: interrupted by {signal_name(exc.signum)} — recorded event " + f"{exc.event_index} as unobserved (no exit code was witnessed)", + file=sys.stderr, + ) + return exc.exit_code + except KeyboardInterrupt: + # Backstop for an interrupt outside a wrapped child's lifetime — during + # a seal, say, where there is no in-flight command to record. A + # traceback is still the wrong way to report Ctrl-C. + print("didrun: interrupted (SIGINT)", file=sys.stderr) + return 130 if __name__ == "__main__": diff --git a/src/didrun/gitplumbing.py b/src/didrun/gitplumbing.py index a087d16..1015832 100644 --- a/src/didrun/gitplumbing.py +++ b/src/didrun/gitplumbing.py @@ -142,11 +142,18 @@ def submodule_dirty(repo: Path) -> bool: # is stable across repos with different ignore rules. LEDGER_DIRNAME = ".didrun" +# Where rotated ledgers are kept. Same argument as the ledger dir, one step +# later: archiving a ledger is tooling housekeeping, so if the archive root +# counted, the act of archiving would move the digest and stale the very unit +# that just archived. This is a digest-SEMANTICS change from v0.1 and only for +# a repo that carries this directory WITHOUT gitignoring it — see docs/COMPAT.md. +ARCHIVE_DIRNAME = ".didrun-history" + def tree_digest( repo: Path, ledger_objects: Optional[Path] = None, - exclude: tuple[str, ...] = (LEDGER_DIRNAME,), + exclude: tuple[str, ...] = (LEDGER_DIRNAME, ARCHIVE_DIRNAME), ) -> Optional[str]: """Digest the current working tree (staged + unstaged + untracked). @@ -155,7 +162,7 @@ def tree_digest( are written into ``ledger_objects`` if given (zero pollution of the user's object DB) with the repo's real objects available as an alternate so the write-tree can reference existing blobs. ``exclude`` pathspecs are kept out - of the digest (the ledger dir by default). + of the digest (the ledger dir and the archive root by default). """ if not is_git_repo(repo): return None @@ -221,12 +228,19 @@ class PathChange: def tree_delta( repo: Path, tree_a: str, tree_b: str, ledger_objects: Optional[Path] = None -) -> list[PathChange]: +) -> Optional[list[PathChange]]: """Exact path-level delta between two tree objects. Requires that both trees' objects are reachable — the ledger's redirected objects plus the repo's own. This is what turns "stale" from a shrug into a measured statement ("stale: 1 line changed in src/foo.py"). + + Two return values that must never be conflated: ``[]`` means *computed, and + the trees are equal*; ``None`` means *not computable* — git failed, so + nothing at all is known about the difference. Returning ``[]`` on failure + (v0.1 did) lets the caller render "0 path(s) differ" and "evidence tree + equals sealed tree" as facts derived from a subprocess that failed, which + is exactly the archived-ledger case where the evidence objects are gone. """ env = {} if ledger_objects is not None: @@ -239,7 +253,7 @@ def tree_delta( check=False, ) if proc.returncode != 0: - return [] + return None changes: list[PathChange] = [] for line in proc.stdout.splitlines(): parts = line.split("\t") diff --git a/src/didrun/ledger.py b/src/didrun/ledger.py index 065d72a..6579e4b 100644 --- a/src/didrun/ledger.py +++ b/src/didrun/ledger.py @@ -18,17 +18,71 @@ from __future__ import annotations +import contextlib import hashlib import json import os -from dataclasses import dataclass, field, asdict +import sys +import tempfile +from dataclasses import dataclass, field, asdict, fields as dataclass_fields from pathlib import Path -from typing import Any, Iterator, Optional +from typing import IO, Any, Iterator, Optional + +try: + import fcntl +except ImportError: # pragma: no cover - exercised only off POSIX + fcntl = None # type: ignore[assignment] + +# True when this platform can serialize appends between processes. False means +# the degradation documented in docs/COMPAT.md is live: concurrent writers can +# fork the chain, exactly as v0.1 did everywhere. No portable fallback is +# invented here — a fallback that does not actually exclude would be worse than +# a declared gap, because callers would stop treating concurrency as a hazard. +APPEND_LOCK_AVAILABLE = fcntl is not None + +# The ledger is secret-bearing by the tool's own account (docs/TRUST_MODEL.md): +# it holds argv verbatim, full stdout/stderr, and a content-addressed snapshot of +# every non-ignored file in the tree. 0700 on the root is the durable control — +# see Session.harden for what it does and does not cover. +LEDGER_DIR_MODE = 0o700 +LEDGER_FILE_MODE = 0o600 + +# The append mutex. It lives INSIDE the ledger directory, which is excluded from +# every tree digest unconditionally and self-ignored by the "*" rule __init__ +# writes — so it moves no digest and can never be committed. +LOCK_FILENAME = ".lock" # The genesis link. The first entry chains from this fixed, well-known value so # an empty vs tampered-truncated log are distinguishable. GENESIS_HASH = "0" * 64 +# The chain preimage this binary writes. Every entry_hash ever recorded was +# computed over version 1, so version 1's field list is frozen forever: a name +# added to or removed from it stops every archived entry_hash reproducing. +# A future schema adds a *new* version here and leaves 1 untouched. +PREIMAGE_VERSION = 1 + +# canonical_json sorts keys, so the order within each tuple does not affect the +# digest — only membership does. +_PREIMAGE_FIELDS: dict[int, tuple[str, ...]] = { + 1: ( + "argv", + "cwd", + "env_fingerprint", + "observed_via", + "coverage", + "exit_code", + "started_at", + "ended_at", + "stdout_blob", + "stderr_blob", + "transcript_blob", + "tree_before", + "tree_after", + "submodule_dirty", + ), +} + # Observation provenance. Only wrapper/shim events witnessed an actual exit code; # a transcript only saw bytes on a terminal and must never claim an exit code. OBSERVED_VIA = ("wrapper", "shim", "trap", "transcript") @@ -64,6 +118,59 @@ class LedgerError(Exception): """Raised on a structural violation of a ledger invariant.""" +def _open_private_append(path: Path) -> IO[bytes]: + """Open ``path`` for binary append, creating it 0600 rather than 0666&~umask. + + The mode argument to ``os.open`` applies at CREATION only: a file an older + binary already created at 0644 keeps 0644. That bound is deliberate and is + documented on ``Session.harden`` — the root's 0700 is what makes it + acceptable, and it is not the same statement as "the files are 0600". + """ + fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_APPEND, LEDGER_FILE_MODE) + return os.fdopen(fd, "ab") + + +@contextlib.contextmanager +def _append_lock(root: Path) -> Iterator[None]: + """Serialize a read-tail-and-append pair against other didrun processes. + + The hazard is corruption, not throughput. ``Session.append`` reads the tail, + derives ``(index, prev_hash)`` from it, then writes; two processes that + interleave those steps both write the SAME index chained from the SAME + predecessor, and the fork is permanent — repairing it means rewriting the + log, the one thing an evidence store must not do. + + ``flock`` is the right primitive precisely because it is owned by the kernel + and released when the holder dies, however it dies. A PID-file or + ``O_EXCL`` sentinel would introduce a stale-lock class that a SIGKILLed + process leaves behind forever; this introduces none. + + Two bounds, stated rather than glossed: + + - Without ``fcntl`` this is a no-op and concurrent writers can fork the + chain exactly as they could in v0.1. See ``APPEND_LOCK_AVAILABLE``. + - ``flock`` over NFS and some network filesystems is emulated or ignored. + The exclusion is as good as the filesystem's, and no better. + + The critical section is microseconds of local I/O. It is never held across a + wrapped child process: the child has already exited by the time an event + exists to append. + """ + if fcntl is None: + yield + return + path = Path(root) / LOCK_FILENAME + fd = os.open(str(path), os.O_RDWR | os.O_CREAT, LEDGER_FILE_MODE) + try: + fcntl.flock(fd, fcntl.LOCK_EX) + try: + yield + finally: + fcntl.flock(fd, fcntl.LOCK_UN) + finally: + os.close(fd) + + @dataclass(frozen=True) class Event: """One observed execution. @@ -123,9 +230,21 @@ def to_dict(self) -> dict: @classmethod def from_dict(cls, d: dict) -> "Event": - d = dict(d) - d["argv"] = tuple(d.get("argv", ())) - return cls(**d) + # Unknown keys are dropped rather than fatal: a ledger written by a + # newer binary must stay *readable* by this one. What is dropped is not + # discarded silently — unknown_event_keys_of() reports it, and an entry + # carrying any is graded unverifiable rather than recomputed. + known = {k: v for k, v in d.items() if k in _EVENT_FIELD_NAMES} + known["argv"] = tuple(known.get("argv", ())) + return cls(**known) + + +_EVENT_FIELD_NAMES = frozenset(f.name for f in dataclass_fields(Event)) + + +def unknown_event_keys_of(d: dict) -> tuple[str, ...]: + """Keys in a stored event body that this binary's Event schema lacks.""" + return tuple(sorted(k for k in d if k not in _EVENT_FIELD_NAMES)) @dataclass @@ -136,16 +255,48 @@ class ChainEntry: prev_hash: str event: Event entry_hash: str = "" + # Neither of these is serialized into the event body or hashed; ChainEntry + # is built by hand, never via asdict, so they are free. + preimage_version: int = PREIMAGE_VERSION + unknown_event_keys: tuple[str, ...] = () def compute_hash(self) -> str: # The hash binds the previous link, the position, and the canonical - # event body. Reordering, truncation, or mutation all break it. + # event body, projected onto the field list declared for this preimage + # version. Reordering, truncation, or mutation all break it. At version + # 1 the projection is byte-identical to the whole to_dict(), which is + # what keeps every archived entry_hash reproducing. + field_names = _PREIMAGE_FIELDS.get(self.preimage_version) + if field_names is None: + raise LedgerError( + f"unknown chain preimage version: {self.preimage_version}" + ) + d = self.event.to_dict() body = canonical_json( - {"index": self.index, "prev": self.prev_hash, "event": self.event.to_dict()} + { + "index": self.index, + "prev": self.prev_hash, + "event": {k: d[k] for k in field_names}, + } ) return sha256_hex(self.prev_hash.encode("ascii") + body) +@dataclass(frozen=True) +class ChainVerdict: + """The honest three-way answer about a chain, not a boolean. + + ``unverifiable`` is the state the boolean could not express: the entry was + written by something this binary does not fully understand, so its hash + cannot be recomputed. That is never the same statement as tamper. + """ + + status: str # "intact" | "broken" | "unverifiable" | "empty" + first_broken_index: Optional[int] = None + unverifiable_index: Optional[int] = None + reason: str = "" + + class BlobStore: """Content-addressed byte store. @@ -154,18 +305,43 @@ class BlobStore: the seal step; here it is a plain directory of ```` files. """ - def __init__(self, root: Path) -> None: + def __init__(self, root: Path, readonly: bool = False) -> None: self.root = Path(root) - self.root.mkdir(parents=True, exist_ok=True) + self.readonly = readonly + # A read-only store creates nothing: it is opened by commands that only + # read a ledger, and a directory conjured by `verify` is the same defect + # as a world-readable one. + if not readonly: + self.root.mkdir(parents=True, exist_ok=True) def put(self, data: bytes) -> str: + if self.readonly: + raise LedgerError("read-only blob store: refusing to write a blob") digest = sha256_hex(data) path = self.root / digest if not path.exists(): # Write-once. Atomic via a temp file + rename so a crash mid-write # never leaves a partial blob under a valid-looking digest. - tmp = self.root / (digest + ".tmp") - tmp.write_bytes(data) + # + # The temp name is unique PER WRITER, not `.tmp`. Two didrun + # processes recording concurrently store the same blob constantly — + # an empty stderr is one digest shared by every event — and with a + # shared temp name the second writer's rename deletes the first + # writer's temp out from under it, so the first dies on chmod or + # replace with FileNotFoundError. Measured, not theorised: 2 of 6 + # eight-way concurrent `didrun run` attempts crashed that way. Both + # writers producing the same bytes is exactly what content + # addressing guarantees, so the last rename winning is correct. + tmp_fd, tmp_name = tempfile.mkstemp( + prefix=digest + ".", suffix=".tmp", dir=str(self.root) + ) + tmp = Path(tmp_name) + with os.fdopen(tmp_fd, "wb") as fh: + fh.write(data) + # os.replace preserves the SOURCE file's mode, so the blob inherits + # whatever the temp had. mkstemp creates 0600, but state it rather + # than depend on it: the blob is the one durable artifact here. + os.chmod(tmp, LEDGER_FILE_MODE) os.replace(tmp, path) return digest @@ -191,38 +367,179 @@ class Session: the first break. """ - def __init__(self, root: Path) -> None: + def __init__(self, root: Path, readonly: bool = False) -> None: self.root = Path(root) - self.root.mkdir(parents=True, exist_ok=True) - # The ledger is secret-bearing by construction (it records everything a - # session printed). Self-ignore the whole directory so a user's - # `git add -A` can never stage it — even if they forgot a root-level - # rule. A `*` here ignores every path in the ledger, including itself. - gitignore = self.root / ".gitignore" - if not gitignore.exists(): - gitignore.write_text("*\n", encoding="ascii") + self.readonly = readonly + # Set by every full read of the log (see _iter_records). A torn final + # record is a lost event, never tamper, so it is reported rather than + # raised — but it is never silent either. + self.tail_truncated = False + self.tail_truncated_offset: Optional[int] = None + # Recorded, not asserted: on a platform without fcntl the append is not + # serialized and two writers can fork the chain. A caller that wants to + # say so can read this; nothing in the trust path branches on it, because + # a lock is not evidence. + self.append_serialized = APPEND_LOCK_AVAILABLE + self.lock_path = Path(root) / LOCK_FILENAME + if not readonly: + self.root.mkdir(parents=True, exist_ok=True) + self.harden() + # The ledger is secret-bearing by construction (it records + # everything a session printed). Self-ignore the whole directory so + # a user's `git add -A` can never stage it — even if they forgot a + # root-level rule. A `*` here ignores every path in the ledger, + # including itself. + gitignore = self.root / ".gitignore" + if not gitignore.exists(): + with _open_private_append(gitignore) as fh: + fh.write(b"*\n") self.log_path = self.root / "session.log" - self.blobs = BlobStore(self.root / "objects") + self.blobs = BlobStore(self.root / "objects", readonly=readonly) + + def harden(self) -> None: + """Force the ledger ROOT to 0700, repairing a ledger already on disk. + + The chmod is the load-bearing half: ``mkdir(exist_ok=True)`` never + re-tightens an existing directory, so a ledger a v0.1 binary created at + 0755 would stay 0755 forever without this. It therefore runs on every + writable open, not only on creation. + + **The bound, stated because a rounded-up version of it would be a lie.** + This repairs the root and only the root. A ``session.log``, + ``claims.jsonl`` or ``.gitignore`` an older binary created at 0644 keeps + 0644 — ``__init__`` writes ``.gitignore`` only when it is absent, so it + is never re-created. 0700 on the root is what makes that acceptable: + another uid cannot reach those files through the directory. It is NOT + the same guarantee as "the files are 0600". + + ``objects/`` is deliberately not walked. git populates it directly via + ``GIT_OBJECT_DIRECTORY``, twice per ``didrun run``, at git's own modes, + so chmod-ing it would be undone by the next command. + + Never chmods a directory this process does not own: warns once on stderr + and continues. ``os.geteuid`` does not exist on Windows, where the check + is skipped (and where ``os.chmod`` is close to a no-op anyway — see + docs/COMPAT.md). + """ + geteuid = getattr(os, "geteuid", None) + if geteuid is not None: + try: + owner = self.root.stat().st_uid + except OSError as exc: + print(f"didrun: cannot stat {self.root}: {exc}", file=sys.stderr) + return + if owner != geteuid(): + print( + f"didrun: not tightening {self.root} to 0700 — it is owned " + f"by uid {owner}, not this process", + file=sys.stderr, + ) + return + try: + os.chmod(self.root, LEDGER_DIR_MODE) + except OSError as exc: + print( + f"didrun: could not set {self.root} to 0700: {exc}", + file=sys.stderr, + ) + + # --- reading the raw log -------------------------------------------------- + + def _iter_records(self) -> Iterator[dict]: + """Yield each stored record, tolerating a torn FINAL line. + + A process killed mid-append leaves a partial last line. ``json.loads`` + on it used to raise, and every reader called it, so one lost event + killed the entire ledger — reads AND writes. The tolerance here is + deliberately narrow: only the last line may fail to parse, the fact is + recorded on ``tail_truncated`` rather than swallowed, and an + unparseable line with data after it is corruption in the middle of the + log, which stays a hard error. + + Read in binary so a tear that split a multi-byte sequence is a parse + failure like any other rather than a decode error from the reader. + """ + self.tail_truncated = False + self.tail_truncated_offset = None + if not self.log_path.exists(): + return + torn_at: Optional[int] = None + with self.log_path.open("rb") as fh: + offset = 0 + for raw in fh: + start = offset + offset += len(raw) + if torn_at is not None: + raise LedgerError( + f"session.log has an unparseable record at byte " + f"{torn_at} with {offset - torn_at} bytes of data after " + f"it: that is corruption inside the log, not an " + f"interrupted final write" + ) + line = raw.strip() + if not line: + continue + try: + rec = json.loads(line) + except ValueError: + torn_at = start + continue + yield rec + if torn_at is not None: + self.tail_truncated = True + self.tail_truncated_offset = torn_at # --- writing ------------------------------------------------------------- def _last(self) -> tuple[int, str]: - """Return (last_index, last_hash) by reading the tail of the log.""" - if not self.log_path.exists(): - return -1, GENESIS_HASH - last_line = None - with self.log_path.open("r", encoding="ascii") as fh: - for line in fh: - line = line.strip() - if line: - last_line = line - if last_line is None: + """Return (last_index, last_hash) from the last PARSEABLE record. + + Shares ``_iter_records``' torn-tail tolerance deliberately. Giving it + only to ``entries()`` would leave every subsequent ``append`` raising + after an interrupted write: the reader would recover and the recorder + would stay dead, which is the failure this is here to prevent. Reading + past the tear is not the same as writing past it — ``append`` still + refuses that. + """ + last: Optional[dict] = None + for rec in self._iter_records(): + last = rec + if last is None: return -1, GENESIS_HASH - rec = json.loads(last_line) - return rec["index"], rec["entry_hash"] + return last["index"], last["entry_hash"] def append(self, event: Event) -> ChainEntry: + # Checked BEFORE the lock so a read-only session never creates or opens + # the lock file: `verify` and `show` must leave no trace in a ledger and + # must never contend with a writer. + if self.readonly: + raise LedgerError("read-only session: refusing to append to the ledger") + with _append_lock(self.root): + return self._append_locked(event) + + def _append_locked(self, event: Event) -> ChainEntry: + """The critical section: read the tail, derive the link, write the line. + + Split out only so the lock's extent is legible. Everything here must be + cheap and local — a wrapped child has long since exited by the time an + event exists to append, and the lock must never be widened to cover one. + """ last_index, last_hash = self._last() + if self.tail_truncated: + # Refuse; never truncate. Two reasons, and the second is the sharp + # one. The ledger is evidence, so a writer that edits it to make + # room is the one thing it must not do. And appending past the tear + # would leave those partial bytes in the MIDDLE of the log, where + # they are unparseable forever — a permanently broken chain, which + # turns one lost event into a dead ledger. The offset is named so + # an operator can decide. + raise LedgerError( + f"session.log has a torn final record at byte " + f"{self.tail_truncated_offset}; refusing to append. One event " + f"was lost to an interrupted write. Recover deliberately: " + f"truncate the file to {self.tail_truncated_offset} bytes, or " + f"archive this ledger and start a new one." + ) entry = ChainEntry(index=last_index + 1, prev_hash=last_hash, event=event) entry.entry_hash = entry.compute_hash() rec = { @@ -231,49 +548,133 @@ def append(self, event: Event) -> ChainEntry: "entry_hash": entry.entry_hash, "event": event.to_dict(), } + # Omitted at version 1 so a v1 record on disk stays byte-identical to + # what every previous release wrote, and an older binary keeps reading it. + if entry.preimage_version != 1: + rec["preimage_version"] = entry.preimage_version # Append one canonical line. Newline-delimited so the tail read is cheap. - with self.log_path.open("ab") as fh: + with _open_private_append(self.log_path) as fh: fh.write(canonical_json(rec) + b"\n") + # The line is evidence the moment append() returns: a crash between + # here and the kernel's own flush would lose an event this process + # already reported as recorded. fsync changes durability, never + # bytes. + fh.flush() + os.fsync(fh.fileno()) return entry # --- reading / verifying ------------------------------------------------- def entries(self) -> Iterator[ChainEntry]: - if not self.log_path.exists(): - return - with self.log_path.open("r", encoding="ascii") as fh: - for line in fh: - line = line.strip() - if not line: - continue - rec = json.loads(line) - entry = ChainEntry( - index=rec["index"], - prev_hash=rec["prev_hash"], - event=Event.from_dict(rec["event"]), - entry_hash=rec["entry_hash"], - ) - yield entry + for rec in self._iter_records(): + body = rec["event"] + entry = ChainEntry( + index=rec["index"], + prev_hash=rec["prev_hash"], + event=Event.from_dict(body), + entry_hash=rec["entry_hash"], + # Absent means version 1: that is what every archived + # record looks like and it must keep meaning v1. + preimage_version=rec.get("preimage_version", 1), + unknown_event_keys=unknown_event_keys_of(body), + ) + yield entry def events(self) -> list[Event]: return [e.event for e in self.entries()] - def verify_chain(self) -> tuple[bool, Optional[int]]: - """Recompute the chain. Returns (ok, first_broken_index). - - ``first_broken_index`` is the index whose stored hash or linkage does - not recompute — the earliest point tamper is detectable. ``None`` when - the chain is intact. + def verify_chain_detail(self) -> ChainVerdict: + """Recompute the chain and report which of the four states it is in. + + Stops at the first entry that is broken or that cannot be recomputed. + An entry this binary cannot recompute — an unknown preimage version, or + event keys outside the version's declared-exhaustive field list — is + ``unverifiable``: the projection would quietly hash a subset and + reproduce the stored digest, so reporting ``intact`` there would be a + lie and reporting ``broken`` would be a different one. + + A torn final record is none of those three. Every link that IS present + recomputes, so the status is ``intact`` over the surviving prefix and + the truncation travels in the reason. Calling it ``broken`` would read + as tamper for what is only a lost event, and that is the one + mistranslation this tool cannot afford. """ prev = GENESIS_HASH expected_index = 0 + seen = False for entry in self.entries(): + seen = True if entry.index != expected_index: - return False, entry.index + return ChainVerdict( + "broken", + first_broken_index=entry.index, + reason=( + f"entry index {entry.index} out of sequence " + f"(expected {expected_index})" + ), + ) if entry.prev_hash != prev: - return False, entry.index - if entry.compute_hash() != entry.entry_hash: - return False, entry.index + return ChainVerdict( + "broken", + first_broken_index=entry.index, + reason=f"entry {entry.index} does not link to the previous entry", + ) + if entry.unknown_event_keys: + return ChainVerdict( + "unverifiable", + unverifiable_index=entry.index, + reason=( + f"entry {entry.index} carries event keys this binary does " + f"not know ({', '.join(entry.unknown_event_keys)}); its " + f"preimage cannot be recomputed" + ), + ) + try: + recomputed = entry.compute_hash() + except LedgerError as exc: + return ChainVerdict( + "unverifiable", + unverifiable_index=entry.index, + reason=f"entry {entry.index}: {exc}", + ) + if recomputed != entry.entry_hash: + return ChainVerdict( + "broken", + first_broken_index=entry.index, + reason=f"entry {entry.index} does not recompute to its stored hash", + ) prev = entry.entry_hash expected_index += 1 - return True, None + # Only reachable once the iteration finished, which is exactly when the + # torn-tail flag is known: every early return above stopped at a fault + # that dominates this one. + tail = "" + if self.tail_truncated: + tail = ( + f"the log's final record is torn at byte " + f"{self.tail_truncated_offset} — one event was lost to an " + f"interrupted write; this verdict covers the surviving prefix" + ) + if not seen: + return ChainVerdict("empty", reason=tail) + return ChainVerdict("intact", reason=tail) + + def verify_chain(self) -> tuple[bool, Optional[int]]: + """Recompute the chain. Returns (ok, first_broken_index). + + ``first_broken_index`` is the index whose stored hash or linkage does + not recompute — the earliest point tamper is detectable. ``None`` when + the chain is intact. + + A thin wrapper over ``verify_chain_detail``; anything other than intact + or empty is ``ok=False``, including ``unverifiable``, which collapses to + the same boolean but is never the same fact. Callers that need to tell + "cannot recompute" from "tamper" must use ``verify_chain_detail``. This + signature is depended on and does not change. + """ + verdict = self.verify_chain_detail() + if verdict.status in ("intact", "empty"): + return True, None + if verdict.status == "broken": + return False, verdict.first_broken_index + return False, verdict.unverifiable_index diff --git a/src/didrun/manifest.py b/src/didrun/manifest.py index 451965d..11587a6 100644 --- a/src/didrun/manifest.py +++ b/src/didrun/manifest.py @@ -11,21 +11,59 @@ Verify resolves a manifest by commit id, falls back to tree id, recomputes the grades, and returns a structured report with CI-friendly exit semantics. It is a pre-merge / PR-head tool: squash-merged trees grade stale/unknown by design. + +The note is the only artifact that leaves the machine, so the redaction it +carries is declared rather than inferred: each claim entry gets a `redaction` +block naming the projection version, the detector-set version, and the exact +source spans that were replaced. A consumer reads that block. The marker string +alone is NOT authenticating — a recorded argv can contain the literal text +`«redacted:openai-key»` and nothing escapes it — so "is this a real redaction or +did the command print that?" is answered by `redaction.applied`, never by +pattern-matching the preview. Keyed markers would close the forgery gap; they +are out of scope (see docs/COMPAT.md). """ from __future__ import annotations +import hashlib import json +import os import subprocess +import sys from dataclasses import dataclass, field from pathlib import Path from typing import Optional -from .ledger import Session, canonical_json -from .claims import Claim, GradeResult, grade -from . import gitplumbing, redact +from .ledger import Session, canonical_json, _append_lock, _open_private_append +from .claims import ( + CONJUNCTION, + ENV_DRIFT, + ENV_INCOMPARABLE, + ENV_MATCH, + ENV_NOT_RECORDED, + GRADE_ORDER_WORST_FIRST, + GRADE_WITNESS_UNAVAILABLE, + Claim, + ClaimError, + GradeResult, + grade, +) +from . import capture, gitplumbing, redact -MANIFEST_VERSION = 1 +# 2: a claim may carry an `evidence` block naming the recorded entry that backed +# it, and verify CHECKS that binding instead of regrading by index alone. The +# bump is for the change of meaning, not for the new key: an older reader parses +# a v2 note fine (every key is additive) and regrades it index-only, exactly as +# it does today. See docs/COMPAT.md. +# +# 3: the claim-type vocabulary gained `conjunction`. This one is NOT additive in +# either direction — `Claim.from_dict` raises `unknown claim type` on a type it +# does not know, and in an older binary it raises from OUTSIDE verify's try, so a +# v0.1/v0.2 reader meeting a v3 note exits with an uncaught traceback rather than +# a graded refusal. There is no way to extend a closed vocabulary without that, +# and silently skipping claim types a reader does not know would be worse: the +# note would under-report and still print a verdict. See docs/COMPAT.md. +MANIFEST_VERSION = 3 NOTES_REF = "refs/notes/didrun" @@ -33,6 +71,26 @@ class ManifestError(Exception): pass +class ManifestFormatError(ManifestError): + """A note body this binary cannot parse at all. + + Split from ``ManifestError`` because the two get different treatment on the + tree-fallback scan: a body that is not a readable manifest is SKIPPED and + counted (a foreign note under the same ref must not hide every good one), + while a version this binary does not understand PROPAGATES. Both are a + graded refusal on the direct-commit path — it subclasses ``ManifestError``, + which is what the CLI already catches, so an unreadable note exits 2 rather + than raising out of the command. + + That distinction used to be made by exception type ACCIDENTALLY: the scan + caught bare ``Exception``, so a malformed body was skipped because + ``json`` happened to raise something that was not a ``ManifestError``. On + the direct-commit path the same shapes had nothing to catch them and left as + an uncaught ``TypeError``/``KeyError`` traceback, with exit 1 — which is + ``--strict``'s "this graded badly" code, not "I could not read this". + """ + + @dataclass class Manifest: version: int @@ -41,6 +99,36 @@ class Manifest: claims: list[dict] # graded results (serialized) coverage: dict secrets_override: bool = False + # What the secrets pass actually looked at. `secrets_override: false` used + # to be unfalsifiable — it is the same word for "nothing was found" and + # "almost nothing was scanned". This says which. Additive, defaults to {}, + # read with .get(): no MANIFEST_VERSION bump (docs/COMPAT.md). + secrets: dict = field(default_factory=dict) + # The half-open claim window this note covers: the seal watermark and the + # claim count at seal time. Without it the note's SCOPE was invisible from + # the note — a re-seal narrowed to `all_claims[watermark:]` and force-wrote + # over the wider record, and nothing in the artifact said so. Additive, + # read with .get() defaults of 0 / len(claims): no MANIFEST_VERSION bump + # (docs/COMPAT.md). + claims_from: int = 0 + claims_to: Optional[int] = None + # The authorisation artifacts this seal cited for its exceptional + # transitions, each graded `cited` and each carrying AUTHORITY_LIMITATION. + # A MANIFEST field and never an Event field: a reference to an external + # artifact inside the chain preimage would make every future rotation of + # that artifact a schema break (docs/COMPAT.md). Additive, defaults to [], + # read with .get(): no MANIFEST_VERSION bump. + authority: list = field(default_factory=list) + + def __post_init__(self) -> None: + # The reading default for `claims_to`, applied once so every consumer + # sees an int. `None` is the "key was absent" marker, and it must NOT + # survive into `to_json` — a note with a null window would be a third + # state nobody reads. What "absent" MEANS (the window is unknown, not + # [0, N)) is answered by `_recorded_window`, which reads key presence + # off the raw body instead of a parsed default. + if self.claims_to is None: + self.claims_to = len(self.claims) def to_json(self) -> bytes: return canonical_json( @@ -51,19 +139,93 @@ def to_json(self) -> bytes: "claims": self.claims, "coverage": self.coverage, "secrets_override": self.secrets_override, + "secrets": self.secrets, + "claims_from": self.claims_from, + "claims_to": self.claims_to, + "authority": self.authority, } ) @classmethod def from_json(cls, data: bytes) -> "Manifest": - d = json.loads(data) + """Parse a note body. Refuses a version it does not understand. + + Refuse, never coerce: a newer manifest may have changed what an + existing field MEANS, and grading it under this version's semantics + would produce a confident verdict about a format this binary has never + seen. Older versions are accepted — new fields are additive and are + read with a v1-reproducing default (see docs/COMPAT.md). + + Every shape that is not a readable manifest raises + ``ManifestFormatError`` rather than whatever ``json`` and ``dict`` + happened to raise on the way past. The version comparison is the reason + this matters: ``version`` arrives from a file, so it is not necessarily + an int, and ``"3" > 3`` raises while ``True > 3`` is quietly False. The + first left a traceback where a refusal belonged; the second ACCEPTED the + note and verified it green, because ``bool`` is an ``int`` subclass and + ``True`` compares as 1. A guard the whole forward-compat story rests on + (docs/COMPAT.md, "Version 3") cannot be one a malformed field walks past. + """ + try: + d = json.loads(data) + except ValueError as exc: + raise ManifestFormatError(f"note body is not valid JSON: {exc}") from exc + if not isinstance(d, dict): + raise ManifestFormatError( + f"note body is a JSON {type(d).__name__}, not a manifest object" + ) + if "version" not in d: + raise ManifestFormatError("note body declares no manifest version") + version = d["version"] + # `_is_index` and not `isinstance(..., int)`: bool is an int subclass, so + # a `version` of `true` would otherwise compare as 1 and be accepted as a + # v1 note. Measured before the fix: `didrun verify --strict` exited 0 on + # it. The same reading rule the supersession mark gets, for the same + # reason — a field that arrived from a file is not a field this process + # computed. + if not _is_index(version) or version < 1: + raise ManifestFormatError( + f"manifest version is {version!r}, which is not a version number " + f"(expected an integer >= 1)" + ) + if version > MANIFEST_VERSION: + raise ManifestError( + f"manifest version {version} is newer than this didrun understands " + f"(max {MANIFEST_VERSION}); upgrade didrun" + ) + # The keys with no reading default. A note missing one is not a note an + # older-or-newer default can stand in for, and indexing straight into + # `d` turned each one into a `KeyError` traceback out of the command. + missing = [k for k in ("commit", "tree", "claims", "coverage") if k not in d] + if missing: + raise ManifestFormatError( + f"manifest is missing required field(s): {', '.join(missing)}" + ) + # `claims` is indexed and iterated by every consumer, and `__post_init__` + # takes its len. A non-list here surfaced as `TypeError: string indices + # must be integers` from deep inside verify. + if not isinstance(d["claims"], list): + raise ManifestFormatError( + f"manifest `claims` is a {type(d['claims']).__name__}, not a list" + ) + # A note published before operator authority existed carries no key, and + # "no authority was cited" is exactly what [] says. A non-list is read as + # [] for the same reason: an authority block is record, never verdict, so + # a malformed one must not raise on a path that reads the rest fine. + authority = d.get("authority", []) + if not isinstance(authority, list): + authority = [] return cls( - version=d["version"], + version=version, commit=d["commit"], tree=d["tree"], claims=d["claims"], coverage=d["coverage"], secrets_override=d.get("secrets_override", False), + secrets=d.get("secrets", {}), + claims_from=d.get("claims_from", 0), + claims_to=d.get("claims_to"), + authority=authority, ) @@ -99,6 +261,70 @@ def _gc_durable_copy(session: Session, results: list[GradeResult]) -> None: raise ManifestError(f"claim-referenced blob missing at seal: {digest}") +def _supersession(claims: list[Claim]) -> dict: + """Which claims in this window a later one retired, as {index: live index}. + + A repair re-declares the same gate: same ``ctype``, same ``label``. That + pair is the identity, so a later claim carrying it supersedes every earlier + one — which is what lets a fix retire a `failed` or `stale` claim INSIDE the + window, with no intervening commit. Before this, the only two escapes were + seal-early (needs a commit) and re-seal (which truncated the record), and + discarding the ledger was the rational third. + + **Claim-preserving by construction.** Nothing here relabels, deletes or + re-grades anything: every claim in the window is graded and every graded + result is published. The earlier grade and reason stay in the note verbatim, + marked with the index of the entry that retired it. Supersession subtracts + from the VERDICT, never from the RECORD. + + The value is the index of the LIVE entry for that key — the last one in the + window — not the immediate successor. A reader asking "which claim stands in + place of this one" gets that answer directly instead of walking a chain, and + three attempts at the same gate all point at the one that stands. + + Window-local by construction: the caller passes only this seal's claims, so + a claim in an earlier window (already sealed into an earlier commit's note) + can never be retired from here. History is not rewritten. + """ + live: dict = {} + for index, claim in enumerate(claims): + live[(claim.ctype, claim.label)] = index + return { + index: live[(claim.ctype, claim.label)] + for index, claim in enumerate(claims) + if live[(claim.ctype, claim.label)] != index + } + + +def _warn_missing_watermark(session: Session, claim_count: int) -> None: + """Say, loudly, when the watermark file is absent and claims exist. + + The watermark is a local, gitignored, unhashed file, and its ABSENCE is not + visible from the note: delete it and every later seal silently re-covers the + whole claim history. Measured in the archives this matters most for — only + 62 of 167 archived ledgers carried a seals.jsonl at all. + + This does not block. A first seal legitimately has no watermark file, and so + does a rotated ledger, and nothing available here distinguishes the two — + saying which is impossible, so the warning says BOTH and leaves the operator + to know which one they are in. Blocking would break the first seal of every + session; silence is what produced a note whose scope was a function of a + deleted file. + """ + if _seals_path(session).exists() or claim_count == 0: + return + print( + f"didrun seal: WARNING: no seal watermark file, so the watermark is 0 " + f"and this seal covers the WHOLE claim history " + f"({claim_count} declared claim(s)). That is correct for a session's " + f"first seal. If this session has sealed before, the watermark file was " + f"removed or rotated and this seal re-covers claims an earlier note " + f"already recorded — nothing in the note or the ledger distinguishes the " + f"two from here.", + file=sys.stderr, + ) + + def seal( session: Session, repo: Path, @@ -106,12 +332,28 @@ def seal( allow_secrets: bool = False, write_notes: bool = True, bundle_path: Optional[Path] = None, + reseal: bool = False, + require_authority: bool = False, ) -> Manifest: """Compile, redact, and attach a manifest for ``commitish``. - Raises ``redact.SecretsBlocked`` if secrets are found and ``allow_secrets`` - is False. On override, the exported artifact is still redacted and the + Raises ``redact.SecretsBlocked`` if a BLOCK-tier detector fires on the + EXPORT domain and ``allow_secrets`` is False. Notice-tier findings, and + block-tier findings confined to the local output blobs, warn loudly on + stderr and do not stop the seal — the blobs stay in a gitignored ledger and + are not published, so refusing over them stopped work without protecting + anything. On override, the exported artifact is still redacted and the override is recorded in the manifest. + + Raises ``ManifestError`` when the commit already carries a note whose claim + window this seal would NARROW, unless ``reseal`` is True. That overwrite was + silent and lossy: `git notes add -f` force-replaces, and a seal scoped to + `all_claims[watermark:]` replaced a wide record with a narrow one. + + ``require_authority`` refuses an exceptional transition — ``allow_secrets`` + or ``reseal`` — that no live `cited` authority covers. Default off: an + authorisation on the happy path becomes a rubber stamp, so an ordinary seal + demands nothing and binds nothing. """ repo = Path(repo) commit = gitplumbing.head_commit(repo) if commitish == "HEAD" else _rev(repo, commitish) @@ -121,7 +363,27 @@ def seal( if tree is None: raise ManifestError(f"cannot resolve tree for commit {commit}") - events = session.events() + # Authorities first, and the refusal with them: it is a statement about the + # INVOCATION, so it must cost nothing and write nothing when it fires. The + # scopes are the exceptional transitions this seal actually performs — an + # ordinary seal performs none, binds none, and gains no line anywhere. + authority_scopes = tuple( + scope + for scope, requested in ( + (AUTHORITY_ALLOW_SECRETS, allow_secrets), + (AUTHORITY_RESEAL, reseal), + ) + if requested + ) + bound_authority = _bind_authorities(session, authority_scopes, _seal_count(session)) + if require_authority: + _require_authority(authority_scopes, bound_authority) + + # Entries, not just events: the seal records WHICH recorded entry backed + # each claim, and an entry's chain hash is the only handle on that which + # survives the ledger being archived, rebuilt, or replaced. + entries = list(session.entries()) + events = [e.event for e in entries] ledger_objects = session.blobs.root # Seal scoping: a manifest carries the claims declared SINCE THE LAST SEAL — @@ -132,40 +394,147 @@ def seal( # The claims file itself remains append-only; only the seal's view is scoped. all_claims = _load_claims(session) watermark = _last_seal_watermark(session) + _warn_missing_watermark(session, len(all_claims)) new_claims = all_claims[watermark:] - # Grade the claims for THIS seal against the sealed tree. - results: list[GradeResult] = [] - for claim in new_claims: - results.append(grade(claim, tree, events, repo, ledger_objects)) + # Grade the claims for THIS seal against the sealed tree. EVERY claim in the + # window is graded, including the ones a later claim supersedes: the mark + # below excludes a retired claim from the verdict, and grading it anyway is + # what keeps the record honest about what was declared and what it earned. + # + # Two passes, because a conjunction is graded against its SIBLINGS and pass + # one is what produces them. `verify` is restructured the same way, over the + # note's own results — the two must agree or a note that sealed green would + # regrade `unknown`. + results: list = [None] * len(new_claims) + conjunctions: list[int] = [] + for i, claim in enumerate(new_claims): + if claim.ctype == CONJUNCTION: + conjunctions.append(i) + continue + results[i] = grade(claim, tree, events, repo, ledger_objects) + superseded = _supersession(new_claims) + # The retired attempts are dropped from the pool here rather than marked on + # the result: `superseded_by` on a GradeResult is verify-side, and the seal + # holds the authoritative mapping already. + siblings = [r for i, r in enumerate(results) if r is not None and i not in superseded] + for i in conjunctions: + results[i] = grade( + new_claims[i], tree, events, repo, ledger_objects, siblings=siblings + ) _gc_durable_copy(session, results) - # Secrets pass over argv + backing output blobs before anything is exported. - findings = _scan_for_secrets(session, results) - if findings and not allow_secrets: - raise redact.SecretsBlocked(findings) + # Secrets pass. The export domain is compiled FIRST so the scan runs over + # the bytes that would actually be published — claim labels and delta paths + # included, which the argv+blobs scan never covered. Nothing is written + # until this clears: a blocked seal must leave the watermark and the note + # exactly where they were, which is what makes retrying one free. + # + # The two fields absent here are the two this pass GENERATES — + # `secrets_override` and `secrets` — and they are the scan's own counters, + # versions and booleans, never operator- or command-authored text. Every + # field that carries content from outside is in these bytes, and a field + # added later is in them too, without anyone remembering to add it. That is + # why the claim window is here even though it is two integers this function + # computed: the rule is "the published bytes are the scanned bytes", and a + # field exempted because it looked harmless is how that rule stops holding. + # + # Compiling the export domain also REDACTS it, and the findings that come + # back are what the gate blocks on. A field cannot be blocked-but-published + # (the label bug) or published-but-unscanned, because one pass does both. + redacted = [ + _redact_result( + r, session, entries, claim_index=i, superseded_by=superseded.get(i) + ) + for i, r in enumerate(results) + ] + claims_payload = [rc.payload for rc in redacted] + field_findings = [f for rc in redacted for f in rc.findings] + field_bytes = sum(rc.bytes_scanned for rc in redacted) + # The authority block is published, so it is scanned and redacted by the same + # pass, on the same rule as the claim fields above. + authority_payload, authority_findings, authority_bytes = _redact_authorities( + bound_authority + ) + field_findings.extend(authority_findings) + field_bytes += authority_bytes + coverage = _coverage_statement(session) + export_bytes = canonical_json( + { + "version": MANIFEST_VERSION, + "commit": commit, + "tree": tree, + "claims": claims_payload, + "coverage": coverage, + "claims_from": watermark, + "claims_to": len(all_claims), + "authority": authority_payload, + } + ) + scan = _scan_for_secrets( + session, results, export_bytes, field_findings, field_bytes + ) + if scan.blocking and not allow_secrets: + raise redact.SecretsBlocked(scan.blocking) + _warn_unblocked_secrets(scan) + # `secrets_override` keeps its exact v0.1 meaning: "this seal would have + # been refused and the operator overrode it". The set that would refuse is + # the blocking set, which is what `findings` denoted before tiering existed. + overridden = bool(scan.blocking) and allow_secrets manifest = Manifest( version=MANIFEST_VERSION, commit=commit, tree=tree, - claims=[_redact_result(r, session) for r in results], - coverage=_coverage_statement(session), - secrets_override=bool(findings) and allow_secrets, + claims=claims_payload, + coverage=coverage, + secrets_override=overridden, + secrets=scan.to_dict(overridden), + claims_from=watermark, + claims_to=len(all_claims), + authority=authority_payload, ) + # Publication and the watermark are one atomic pair. A published note with + # no watermark re-seals every earlier claim forever (the watermark is what + # scopes a seal); a watermark with no published note reports evidence that + # was never written. Nothing is recorded until the note is on the commit, + # and if recording fails the note goes back to what it was. + prior_note = _read_note(repo, commit) if write_notes else None if write_notes: + _refuse_narrowing_reseal(commit, prior_note, manifest, reseal) _attach_note(repo, commit, manifest) - if bundle_path is not None: - Path(bundle_path).write_bytes(manifest.to_json()) - # Advance the seal watermark (append-only, like everything in the ledger). - _record_seal(session, commit, tree, len(all_claims)) + # Everything after publication runs inside the rollback. A caller-supplied + # bundle path is the likeliest failure in this function, and an exception + # escaping here would leave the note published with no watermark -- exactly + # the state the pair above exists to prevent. + try: + if bundle_path is not None: + Path(bundle_path).write_bytes(manifest.to_json()) + # Advance the seal watermark (append-only, like everything in the ledger). + _record_seal(session, commit, tree, len(all_claims)) + except Exception as exc: + raise ManifestError( + f"seal aborted: no watermark was recorded; " + f"{_rollback_note(repo, commit, prior_note, write_notes)}; cause: {exc}" + ) return manifest +# A chain fault is not a grade any claim can carry; it is a statement about the +# ledger every grade was read out of. It dominates the verdict for that reason. +CHAIN_BROKEN = "chain-broken" + +# The two chain states that are faults. `empty` and `absent` are not: a missing +# or empty ledger is a witness-availability fact, already reported per claim as +# `witness-unavailable`, and calling it tamper would make every verify against +# an archived ledger look like an attack. +_CHAIN_FAULTS = ("broken", "unverifiable") + + @dataclass class VerifyReport: commit: str @@ -174,14 +543,73 @@ class VerifyReport: results: list[GradeResult] coverage: dict secrets_override: bool = False + notes_skipped: int = 0 # notes the tree fallback could not parse at all + # The live ledger's chain state. Defaults to "absent" rather than "intact": + # a report built without consulting a ledger has not checked the chain, and + # saying "intact" there would be the same silent lie an unregistered grade + # is in render.py. + chain_status: str = "absent" # intact | broken | unverifiable | empty | absent + chain_broken_index: Optional[int] = None # set only when status is "broken" + chain_reason: str = "" + # Whether environment drift REFUSES rather than reports. Off by default: + # drift is a statement about this machine now, not about whether the + # recorded command ran, so making it fail --strict silently would flip green + # units red on routine noise. The flag is recorded on the report so a reader + # can tell which of the two verdicts they are looking at. + require_env_match: bool = False + # The authorities the note says this seal cited, as published. Record, never + # verdict: an authority does not enter `worst_status` or `all_verified`, and + # it cannot make a red note green. `cited` attests that a file with that + # digest was cited — not that anyone approved anything — which is why it + # grades nothing and only renders. + authority: list = field(default_factory=list) + + @property + def chain_faulted(self) -> bool: + return self.chain_status in _CHAIN_FAULTS + + @property + def live_results(self) -> list: + """The results the verdict is computed over: everything not superseded. + + A superseded result is record, not verdict — a claim a later claim in the + same seal window retired. It stays in ``results``, keeps its grade and + reason verbatim, and renders; it just does not vote. + """ + return [r for r in self.results if not r.is_superseded] + + @property + def superseded_count(self) -> int: + """How many results are retained as record and excluded from the verdict. + + Printed on both human surfaces. Every retry at the same gate is kept and + counted, so "claim until it goes green" is VISIBLE rather than prevented + — which is the honest bound, and strictly better than the invisible + truncation it replaces. + """ + return len(self.results) - len(self.live_results) @property def worst_status(self) -> str: """Worst grade present — drives the verdict and the CI exit code.""" - order = ["failed", "unknown", "stale", "scope-exact", "tree-exact"] - present = {r.grade for r in self.results} + # A broken chain dominates every grade, including "empty": the grades + # below it were read out of a ledger that does not recompute, so none of + # them is a statement anyone should act on. + if self.chain_faulted: + return CHAIN_BROKEN + # Read from claims.py rather than repeated here: a conjunction grades as + # the worst of its conjuncts under this same order, and two copies of it + # could disagree about which grade is worse. + order = GRADE_ORDER_WORST_FIRST + present = {r.grade for r in self.live_results} if not present: - return "empty" + # "no results at all" and "results, every one of them superseded" + # are different facts and must not share a word. The first is an + # unsealed commit. The second is a window that declared claims and + # retired all of them, which has nothing left to verify — so it + # grades `unknown`, never `empty` (which renders as NO CLAIMS) and + # never anything the strict set accepts. + return "empty" if not self.results else "unknown" for status in order: if status in present: return status @@ -189,22 +617,122 @@ def worst_status(self) -> str: @property def all_verified(self) -> bool: - return bool(self.results) and all( - r.grade in ("tree-exact", "scope-exact") for r in self.results - ) + # Membership is deliberately NOT widened. `witness-unavailable` reports + # a grade this run did not check, so admitting it here would make + # --strict pass on evidence nobody has seen. + # + # Environment drift subtracts from this set ONLY under + # --require-env-match. `incomparable` never does, under either mode: it + # is the state every archived v1 note lands in, and refusing on it would + # turn "this note predates the current fingerprint" into a failure. + # + # Superseded results are excluded — they are record, not verdict — and + # the NON-EMPTINESS check is over the surviving set for that reason. + # `all([])` is True, so filtering without re-checking would make a window + # whose every claim was retired verify vacuously green: a green verdict + # over zero live claims, which is the exact shape of overclaiming this + # tool exists to prevent. + if self.chain_faulted: + return False + live = self.live_results + if not live: + return False + for r in live: + if r.grade not in ("tree-exact", "scope-exact"): + return False + if self.require_env_match and r.env_status == ENV_DRIFT: + return False + return True + + @property + def env_counts(self) -> dict: + """How the sealed environment fingerprints compared, by state. + + Always all four keys, including zeros: "0 drifted" and "not checked" + are different facts, and a counter that disappears when it is zero + cannot say the first one. + Over the live set, like every other header counter: these gloss the + verdict, and a denominator that silently included retired claims would + not add up against the one beside it. + """ + counts = { + ENV_MATCH: 0, + ENV_DRIFT: 0, + ENV_INCOMPARABLE: 0, + ENV_NOT_RECORDED: 0, + } + for r in self.live_results: + counts[r.env_status] = counts.get(r.env_status, 0) + 1 + return counts -def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyReport: + @property + def env_drift_count(self) -> int: + return self.env_counts[ENV_DRIFT] + + @property + def evidence_bound_count(self) -> int: + """Claims whose verdict was checked against the entry the seal named. + + The complement is not a failure: a v1 note carries no binding to check, + so its claims are regraded by index exactly as they always were. The + counter exists so a reader can tell the two apart at a glance. + + Over the live set, for the same reason as ``env_counts``. + """ + return sum(1 for r in self.live_results if r.evidence_bound) + + @property + def total(self) -> int: + """Entries in the RECORD, superseded ones included. + + The human surfaces count the live set instead — the verdict's + denominator has to be the set the verdict was computed over — and print + ``superseded_count`` beside it. This stays the record's size because + that is what a caller counting what the note holds is asking for. + """ + return len(self.results) + + +def verify( + session: Session, + repo: Path, + commitish: str = "HEAD", + require_env_match: bool = False, +) -> VerifyReport: """Resolve and re-grade a manifest for ``commitish``. Resolves by commit id, then by tree id (survives amend/rebase). Degrades a claim to unknown ("referenced object gc'd") rather than erroring. + + The sealed environment fingerprint is compared against this process's, once + per verify. That comparison is ACROSS TIME, which is the only axis on which + it says anything: comparing the fingerprints within a claim's own events is + vacuous, because every claim didrun's CLI can declare binds exactly one + event and one event trivially agrees with itself. Drift is reported and does + not change the verdict unless ``require_env_match`` is set. + + A claim the note bound to a specific recorded entry is only regraded when + the live ledger still holds that entry; otherwise the sealed grade is + reported as ``witness-unavailable`` and --strict refuses it. This detects + ACCIDENT AND DRIFT on a published note — an archived, rebuilt or replaced + ledger. It is not a forger barrier: whoever can substitute a ledger can + regenerate its chain and re-run claim+seal to mint fresh hash-bound claims + (ledger.py's declared non-guarantee, and docs/TRUST_MODEL.md). + + The chain is checked BEFORE any grading and a fault dominates the verdict: + a ledger whose links do not recompute cannot support a statement about + anything it holds. Until now ``verify_chain`` had exactly one caller and it + was ``show --session``, so a forked or mid-file-rewritten ledger certified + green under --strict. ``seal`` still does not consult the chain; that is a + separate policy decision, deliberately out of scope here. """ repo = Path(repo) commit = _rev(repo, commitish) tree = gitplumbing.commit_tree(repo, commitish) if commit else None + chain_status, chain_index, chain_reason = _chain_state(session) - manifest, resolved_by = _resolve_manifest(repo, commit, tree) + manifest, resolved_by, notes_skipped = _resolve_manifest(repo, commit, tree) if manifest is None: return VerifyReport( commit=commit or "", @@ -212,23 +740,65 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor resolved_by="none", results=[], coverage={}, + notes_skipped=notes_skipped, + chain_status=chain_status, + chain_broken_index=chain_index, + chain_reason=chain_reason, + require_env_match=require_env_match, ) # Re-grade the MANIFEST'S OWN claims against its sealed tree (deterministic # replay). The manifest is the only thing verify reads besides the repo — # the source-of-truth rule. Reading the live session claims file # here would let later, unsealed claims leak into an old commit's verdict. - events = session.events() + entries = list(session.entries()) + events = [e.event for e in entries] ledger_objects = session.blobs.root - results: list[GradeResult] = [] - for c in manifest.claims: - claim = Claim.from_dict(c["claim"]) - try: - results.append(grade(claim, manifest.tree, events, repo, ledger_objects)) - except Exception as exc: # object gc'd or unreadable → unknown, not crash - results.append( - GradeResult(claim, "unknown", reason=f"regrade failed: {exc}") - ) + # Once per verify, not once per claim: it is a property of this process, and + # reading the umask inside it is a read-modify-write (capture.process_umask). + current_fingerprint = capture.env_fingerprint() + + def _graded(position: int, stored: dict, claim: Claim, siblings=None) -> GradeResult: + result = _verify_claim( + stored, claim, manifest, entries, events, repo, ledger_objects, siblings + ) + # Assigned around the ladder for the same reason as the two below: which + # claim retired this one is a fact about the seal window, not something + # the evidence earns. + result.superseded_by = _superseded_by(stored, position, manifest.claims) + # Assigned around the ladder, like witness-unavailable: the environment + # comparison is a fact about this run, not a grade the evidence earns. + # It is set on EVERY path, including witness-unavailable — the sealed + # fingerprint is in the note whether or not the ledger still holds the + # entry it names. + result.env_status, result.env_reason = _env_comparison( + stored.get("evidence"), current_fingerprint + ) + return result + + # Two passes, the same shape `seal` grades in. A conjunction is graded + # against its siblings' results FROM THIS NOTE, so pass one grades every + # ordinary claim and pass two grades the conjunctions over what pass one + # produced. Regrading a conjunction in one pass with no siblings resolves + # every conjunct to "no match" and verifies `unknown` — so a note that + # sealed green would fail --strict, which is this restructure's whole point. + # Resolving them against the live claims.jsonl instead would be worse still: + # it breaks the source-of-truth rule above and lets a later, unsealed claim + # change an old commit's verdict. + results: list = [None] * len(manifest.claims) + deferred: list = [] + for position, c in enumerate(manifest.claims): + claim = _stored_claim(c, position) + if claim.ctype == CONJUNCTION: + deferred.append((position, c, claim)) + continue + results[position] = _graded(position, c, claim) + # Superseded pass-one results are already marked, and grade_conjunction drops + # them from the pool: a retired attempt shares its label with the claim that + # replaced it, so counting it would make that name ambiguous. + siblings = [r for r in results if r is not None] + for position, c, claim in deferred: + results[position] = _graded(position, c, claim, siblings) return VerifyReport( commit=manifest.commit, tree=manifest.tree, @@ -236,7 +806,321 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor results=results, coverage=manifest.coverage, secrets_override=manifest.secrets_override, + notes_skipped=notes_skipped, + chain_status=chain_status, + chain_broken_index=chain_index, + chain_reason=chain_reason, + require_env_match=require_env_match, + # Entries that are not objects are dropped: the block is rendered, and a + # renderer walking a string as if it were a record is how a malformed + # note becomes a traceback out of `verify`. + authority=[a for a in manifest.authority if isinstance(a, dict)], + ) + + +def _env_comparison(evidence, current: str) -> tuple[str, str]: + """Compare the fingerprint the seal recorded against this process's. + + Four outcomes, and three of them are refusals to compare. The refusals are + the reason this is safe to add to a tool with sealed evidence already in the + world: a note that bound no fingerprint, a bare v0.1 digest, and a + fingerprint written under a different FINGERPRINT_VERSION are all cases + where the two strings are not measurements of the same thing. Reporting any + of them as drift would make every note published before this unit look like + the environment moved. + + Only the GATE portion is compared. A `path=` difference is appended as an + advisory to whatever the gate said — never promoted to drift, and never able + to turn a match into one. + """ + if not isinstance(evidence, dict): + return ( + ENV_NOT_RECORDED, + "this note bound no environment fingerprint to the claim", + ) + sealed = evidence.get("env_fingerprint") + if not isinstance(sealed, str) or not sealed: + return ( + ENV_NOT_RECORDED, + "the sealed evidence block carries no environment fingerprint", + ) + + sealed_version = capture.fingerprint_version(sealed) + current_version = capture.fingerprint_version(current) + if sealed_version is None: + return ( + ENV_INCOMPARABLE, + "the sealed fingerprint is an unversioned v0.1 digest, which is " + "incomparable with this one — not evidence that anything drifted", + ) + if current_version is None or sealed_version != current_version: + return ( + ENV_INCOMPARABLE, + f"the sealed fingerprint is version {sealed_version} and this didrun " + f"computes version {current_version}; different key sets are " + "incomparable, not drifted", + ) + + sealed_gate = capture.fingerprint_gate(sealed) + current_gate = capture.fingerprint_gate(current) + if sealed_gate is None or current_gate is None: + return ( + ENV_INCOMPARABLE, + "a fingerprint carries no readable gate portion to compare", + ) + + advisory = _path_advisory(sealed, current) + if sealed_gate == current_gate: + return ENV_MATCH, advisory + drifted = ( + f"environment changed since seal (sealed gate {sealed_gate}, " + f"now {current_gate})" ) + return ENV_DRIFT, f"{drifted}; {advisory}" if advisory else drifted + + +def _path_advisory(sealed: str, current: str) -> str: + """PATH drift, as text. Never a gate, never drift, never a verdict. + + PATH varies with terminal tab, direnv, `nix develop`, tmux and homebrew + shellenv ordering, so gating on it produces a refusal that fires in a new + shell and is ignored within a week. It is recorded as its own digest so the + variation stays visible to a reader who is looking for it. + + Digests, not values: what changed is not said, because saying it would put a + developer machine's PATH in a published note. + """ + sealed_path = capture.fingerprint_path_digest(sealed) + current_path = capture.fingerprint_path_digest(current) + if not sealed_path or not current_path or sealed_path == current_path: + return "" + return ( + f"PATH differs (sealed path={sealed_path}, now path={current_path}) — " + "recorded, not gated" + ) + + +def _chain_state(session: Session) -> tuple[str, Optional[int], str]: + """The live ledger's chain state, as (status, broken_index, reason). + + ``verify_chain_detail`` reports a log with no records as ``empty`` whether + the file is missing or present-and-zero-length. Those are different facts + here: a missing log is an ARCHIVED (or never-created) ledger, which P1.2 + already reports honestly per claim as ``witness-unavailable``, and reporting + it as a chain state at all would put a ledger-integrity sentence in front of + a reader whose ledger is simply somewhere else. + + ``broken_index`` is populated only for ``broken``. An unverifiable entry has + no broken index — nothing was shown not to recompute; this binary could not + recompute it — so its index travels in the reason instead, which is where + ``verify_chain_detail`` already put it. + + A log this binary cannot even READ — a torn line, a record whose event body + carries keys the schema rejects — is ``unverifiable`` rather than a + traceback. Two reasons: the package's stated posture is that evidence it + cannot read is a graded refusal, not a crash (``cmd_verify``'s exit 2, + ``_regrade``'s fallback); and this runs BEFORE the manifest is resolved, so + without it a torn ledger would start crashing ``verify`` on commits with no + sealed note at all — a path that never read the ledger before. + """ + try: + verdict = session.verify_chain_detail() + except Exception as exc: + # The message is structural (a parse position, a rejected key name), + # never a ledger line: nothing here echoes recorded content. + return "unverifiable", None, f"the ledger could not be read: {exc}" + status = verdict.status + if status == "empty" and not session.log_path.exists(): + status = "absent" + return ( + status, + verdict.first_broken_index if status == "broken" else None, + verdict.reason, + ) + + +def _verify_claim( + stored: dict, + claim: Claim, + manifest: Manifest, + entries: list, + events: list, + repo: Path, + ledger_objects: Optional[Path], + siblings: Optional[list] = None, +) -> GradeResult: + """Grade one sealed claim, checking the note's binding to its evidence first. + + Two paths, and the split is what protects notes published before v0.2: + + - **No evidence block** — a v1 note. Regrade exactly as v0.1 regraded it, + unbound. There is nothing to check: the note never recorded which entry + backed the claim, and inventing a binding for it after the fact would be a + verdict about evidence this run cannot identify. + - **An evidence block** — check it before grading anything. If the live + ledger cannot supply that entry, report the SEALED grade under + ``witness-unavailable`` rather than recomputing a fresh verdict from + whatever the ledger holds now. That silent recomputation is the defect + this whole unit exists to close. + + A conjunction takes the first path: it binds no event, so the seal wrote it + no evidence block and there is no binding to check. ``siblings`` is what it + is graded against, and it is forwarded unread on every other path. + """ + evidence = stored.get("evidence") + if not isinstance(evidence, dict): + return _regrade(claim, manifest, events, repo, ledger_objects, siblings) + + sealed_grade = stored.get("grade") + idx = evidence.get("event_index") + problem = _witness_problem(entries, idx, evidence.get("entry_hash")) + if problem is not None: + return GradeResult( + claim, + GRADE_WITNESS_UNAVAILABLE, + reason=( + f"sealed as {sealed_grade} at commit {manifest.commit[:12]}; " + f"witness ledger unavailable ({problem})" + ), + supporting_event_index=idx if _is_index(idx) else None, + sealed_grade=sealed_grade, + ) + + result = _regrade(claim, manifest, events, repo, ledger_objects, siblings) + result.evidence_bound = True + result.sealed_grade = sealed_grade + return result + + +def _is_index(value) -> bool: + """A real integer index. ``bool`` is an ``int`` subclass and is not one.""" + return isinstance(value, int) and not isinstance(value, bool) + + +def _stored_claim(entry, position: int) -> Claim: + """Parse one stored claim entry, refusing rather than raising out of verify. + + ``Claim.__post_init__`` is the claim vocabulary's gate, and it raises + ``ClaimError`` — which is not a ``ManifestError``, so it escaped ``verify`` + and the CLI's handler as an uncaught traceback. docs/COMPAT.md's "Version 3" + section names this exact shape as the vocabulary break an OLD reader hits on + a v3 note, and the ``MANIFEST_VERSION`` bump is what steers such a reader + into a version refusal first. This closes the same hole from the other end: + THIS reader, meeting a claim type it cannot parse in a note whose version it + accepts, now also refuses instead of crashing. + + A whole-manifest refusal, not a per-claim ``unknown``: grading around the gap + would print a verdict over the claims this binary happened to understand, + which docs/COMPAT.md rejects in as many words as the worse of the two + options. The position is named because a note has many claims and "one of + them is unreadable" is not an actionable sentence. + """ + if not isinstance(entry, dict): + raise ManifestFormatError( + f"claim entry {position} is a {type(entry).__name__}, not an object" + ) + stored = entry.get("claim") + if not isinstance(stored, dict): + raise ManifestFormatError( + f"claim entry {position} carries no claim object" + ) + try: + return Claim.from_dict(stored) + except (ClaimError, TypeError, ValueError) as exc: + # ClaimError is the vocabulary gate. TypeError/ValueError cover the field + # shapes it never gets to look at — `tuple(d.get("event_indices"))` on an + # int raises before __post_init__ runs at all. + raise ManifestFormatError(f"claim entry {position} is unreadable: {exc}") from exc + + +def _claim_key(entry) -> Optional[tuple]: + """The (ctype, label) identity of a stored claim entry, or None.""" + if not isinstance(entry, dict): + return None + claim = entry.get("claim") + if not isinstance(claim, dict): + return None + return claim.get("ctype"), claim.get("label") + + +def live_claims(claims: list) -> list: + """A manifest's claim entries that no later claim in the window retired. + + The one definition of "live", shared by the verdict (``verify``) and by the + seal's own summary line, so the two cannot report different denominators for + the same note. + """ + return [c for i, c in enumerate(claims) if _superseded_by(c, i, claims) is None] + + +def _superseded_by(stored: dict, position: int, claims: list) -> Optional[int]: + """The entry that retired this one, or None — validated, not trusted. + + A supersession mark SUBTRACTS a claim from the verdict, so it is the one + published field that can turn a red note green. It is therefore checked + against the definition that produced it rather than taken at face value: the + target must be a real forward index into this note's own claim list, and it + must carry the same ``(ctype, label)`` the seal supersedes by. + + A mark that fails any of those is dropped, which counts the claim toward the + verdict — the direction that refuses rather than the one that excuses. This + is not a forgery barrier and does not pretend to be one: whoever can rewrite + a note can rewrite a grade directly (docs/TRUST_MODEL.md). It is the same + fail-closed reading didrun applies to every other field it did not compute + in this process. + """ + mark = stored.get("superseded_by") + if not _is_index(mark): + return None + if not (position < mark < len(claims)): + return None + key = _claim_key(stored) + if key is None or key != _claim_key(claims[mark]): + return None + return mark + + +def _witness_problem(entries: list, idx, recorded_hash) -> Optional[str]: + """Why the live ledger cannot supply the sealed evidence, or None. + + Two mechanisms, and they are not the same fact. The ledger may be too SHORT + to hold that index — the archived-ledger case, which an index-keyed regrade + already noticed, if only by grading everything unknown. Or it may be long + enough and hold a DIFFERENT entry there, which an index-keyed regrade + reports as a confident verdict about evidence it never saw. The second is + the dangerous one, so the reason names which happened. + + The comparison is against the entry's STORED hash. Whether that hash is the + one the entry's own body recomputes to is the chain check's question, not + this one; conflating them would report a ledger this binary merely cannot + recompute as evidence that went missing. + """ + if not _is_index(idx): + return "the sealed evidence block names no usable event index" + if not (0 <= idx < len(entries)): + return ( + f"recorded event index {idx} is out of range " + f"(the live ledger holds {len(entries)} event(s))" + ) + if not isinstance(recorded_hash, str) or not recorded_hash: + return f"the sealed evidence block carries no entry hash for index {idx}" + if entries[idx].entry_hash != recorded_hash: + return f"entry hash mismatch at index {idx}" + return None + + +def _regrade( + claim: Claim, + manifest: Manifest, + events: list, + repo: Path, + ledger_objects: Optional[Path], + siblings: Optional[list] = None, +) -> GradeResult: + try: + return grade(claim, manifest.tree, events, repo, ledger_objects, siblings) + except Exception as exc: # object gc'd or unreadable → unknown, not crash + return GradeResult(claim, "unknown", reason=f"regrade failed: {exc}") # --- claim persistence (declared claims live in the ledger dir) -------------- @@ -262,19 +1146,51 @@ def _last_seal_watermark(session: Session) -> int: return watermark +def _seal_count(session: Session) -> int: + """How many seals this ledger has recorded — the authority expiry clock. + + Seals, not wall time. A time-based TTL would put a clock in the trust path, + which is a hard invariant this package does not break, and it would make an + authority's liveness depend on how long a build took. A seal is the + transition an authority is cited for, so counting them is the measure that + means something. + """ + path = _seals_path(session) + if not path.exists(): + return 0 + return sum( + 1 for line in path.read_text(encoding="ascii").splitlines() if line.strip() + ) + + def _record_seal(session: Session, commit: str, tree: str, claims_watermark: int) -> None: - with _seals_path(session).open("a", encoding="ascii") as fh: - fh.write( - canonical_json( - {"commit": commit, "tree": tree, "claims_watermark": claims_watermark} - ).decode("ascii") - + "\n" - ) + # Private-append: these carry operator-authored labels and commit ids into + # the secret-bearing ledger directory, and a plain open() would create them + # 0666 & ~umask. canonical_json is already ascii bytes, so the byte written + # is unchanged from the text-mode open this replaced. + # + # Under the ledger's append lock for the same reason as declare_claim below: + # a seal line carries a commit id and a watermark other seals read back, so a + # torn one loses the watermark and re-seals every claim. + with _append_lock(session.root): + with _open_private_append(_seals_path(session)) as fh: + fh.write( + canonical_json( + {"commit": commit, "tree": tree, "claims_watermark": claims_watermark} + ) + + b"\n" + ) def declare_claim(session: Session, claim: Claim) -> None: - with _claims_path(session).open("a", encoding="ascii") as fh: - fh.write(canonical_json(claim.to_dict()).decode("ascii") + "\n") + # O_APPEND is atomic only below PIPE_BUF, and a claim line is not bounded by + # that: every declared pathspec lengthens it. Two concurrent `didrun claim` + # invocations could therefore interleave into one unparseable record, so this + # takes the same ledger-wide lock the event append takes. Cheap by + # construction — one short local write, never a child process. + with _append_lock(session.root): + with _open_private_append(_claims_path(session)) as fh: + fh.write(canonical_json(claim.to_dict()) + b"\n") def _load_claims(session: Session) -> list[Claim]: @@ -289,6 +1205,377 @@ def _load_claims(session: Session) -> list[Claim]: return claims +# --- operator authority ------------------------------------------------------ +# +# didrun's thesis is that a claim with no witnessed event grades `unknown`. An +# out-of-band human authorisation is structurally a claim with no witnessed +# event, and `--allow-secrets` is the shipped instance: a bare argv flag stored +# as a bare boolean, with no principal, no basis, and no way for a reader to +# tell an authorised override from an agent that simply passed the flag. +# +# What this converts that into is narrow and is stated on every surface: "someone +# passed a flag citing an artifact with this digest, at this point in an +# append-only record". It is NOT evidence that a human approved anything — an +# agent can author its own authorisation artifact in one line. Signing is +# deferred (docs/TRUST_MODEL.md) and this does not close that gap; the honest +# anchor for a stronger tier is a key the agent cannot read. + +# The scopes an authority may be cited for: the exceptional transitions, and only +# those. There is deliberately no scope for an ordinary seal — an authorisation +# required on the happy path becomes a rubber stamp within a week, which is +# exactly how `--allow-secrets`'s information content went to zero. +AUTHORITY_ALLOW_SECRETS = "allow-secrets" +AUTHORITY_RESEAL = "reseal" +AUTHORITY_SCOPES = (AUTHORITY_ALLOW_SECRETS, AUTHORITY_RESEAL) + +# The grade a bound authority carries, in the same vocabulary as `tree-exact` and +# under the same rule: reported verbatim, never upgraded. `cited` never renders as +# "signed", "approved", "verified" or "authorized by". +AUTHORITY_GRADE = "cited" + +# How the principal was established. One value, because there is exactly one +# mechanism: didrun read $USER, or the operator typed a name. Recorded as a field +# rather than left implicit so a reader cannot mistake it for an authenticated +# identity. +AUTHORITY_PRINCIPAL_BASIS = "self-asserted" + +# Default TTL in seals when the operator names none. One, not unbounded: a +# standing grant is the shape `--allow-secrets` already had, and re-earning the +# citation per transition is the only thing that keeps it from routine-ising. +AUTHORITY_DEFAULT_TTL = 1 + +# The feature's honesty, as a constant that ships in the source and is printed on +# every surface that shows an authority — the terminal verdict, the HTML report, +# `authorize`'s own output, and the note itself. It lives here rather than only in +# the docs because a reader of a note or a verdict is not reading the docs, and +# `"grade": "cited"` with nothing beside it would be read as approval. +AUTHORITY_LIMITATION = ( + "cited authority: recorded that a file with this digest was cited for this " + "scope. NOT a signature and NOT approval — didrun cannot tell who wrote it." +) + +_DIGEST_CHUNK = 1 << 20 + + +def _authority_path(session: Session) -> Path: + return session.root / "authority.jsonl" + + +def _artifact_digest(path: Path) -> tuple: + """(sha256 hex, byte count) for the cited artifact. Never its bytes. + + Chunked because an operator may cite a large file and there is no reason to + hold it in memory. The digest is the only thing that leaves this function: + an authorisation artifact is exactly the kind of file that carries a + credential in passing, and the note it would ride into is the one artifact + that leaves the machine. + """ + digest = hashlib.sha256() + total = 0 + with open(path, "rb") as fh: + while True: + chunk = fh.read(_DIGEST_CHUNK) + if not chunk: + break + total += len(chunk) + digest.update(chunk) + return digest.hexdigest(), total + + +def declare_authority( + session: Session, + artifact: Path, + scope: str, + label: str, + principal: Optional[str] = None, + expires_after_seals: Optional[int] = None, +) -> dict: + """Record that a file with this digest was cited as the authority for a scope. + + Refuses a scope outside ``AUTHORITY_SCOPES``, an empty label, a TTL below + one, and an artifact that does not exist or is empty. An empty file is + refused rather than digested because its digest is a constant: every empty + authorisation in the world shares it, so it distinguishes nothing and would + be the one shape a fabricated citation could produce without a file. + + The path is recorded as the operator wrote it and is not resolved. A relative + citation therefore stays relative, which keeps a home directory out of a + published note when the operator cites one relatively; an absolute path is + published as typed. + """ + if scope not in AUTHORITY_SCOPES: + raise ManifestError( + f"unknown authority scope {scope!r}: didrun authorises only the " + f"exceptional transitions ({', '.join(AUTHORITY_SCOPES)}). There is " + f"no scope for an ordinary seal, and none will be added: an " + f"authorisation required on the happy path becomes a rubber stamp." + ) + if not label or not label.strip(): + raise ManifestError( + "an authority needs a --label saying what is being authorised; " + "an unlabelled citation records nothing a reader can act on" + ) + ttl = AUTHORITY_DEFAULT_TTL if expires_after_seals is None else expires_after_seals + if not _is_index(ttl) or ttl < 1: + raise ManifestError( + f"--expires-after-seals must be an integer >= 1 (got {ttl!r}); an " + f"authority that expires after zero seals authorises nothing, and " + f"there is no value meaning 'never'" + ) + artifact = Path(artifact) + if not artifact.is_file(): + raise ManifestError( + f"cannot cite {artifact}: no such file (an authority cites an " + f"artifact that exists, so the digest is of something a reviewer " + f"can open)" + ) + sha256, size = _artifact_digest(artifact) + if size == 0: + raise ManifestError( + f"cannot cite {artifact}: the file is empty, and every empty file " + f"has the same digest — it would distinguish nothing" + ) + record = { + "principal": ( + principal if principal is not None else os.environ.get("USER", "") + ), + "principal_basis": AUTHORITY_PRINCIPAL_BASIS, + "artifact_path": str(artifact), + "artifact_sha256": sha256, + "artifact_bytes": size, + "scope": scope, + "label": label, + "declared_at_index": _seal_count(session), + "expires_after_seals": ttl, + } + # Same append discipline as claims and seals: 0600 at creation, under the + # ledger-wide lock. A record line carries an operator-authored label and is + # not bounded by PIPE_BUF, so two concurrent `didrun authorize` invocations + # could otherwise interleave into one unparseable record. + with _append_lock(session.root): + with _open_private_append(_authority_path(session)) as fh: + fh.write(canonical_json(record) + b"\n") + return record + + +def _valid_authority(record) -> bool: + """Whether a stored authority record is one this seal may read. + + ``authority.jsonl`` is a local, gitignored, unhashed file, exactly like + ``claims.jsonl`` — so every field here arrived from a file rather than from + this process, and gets the same validate-rather-than-trust reading that + ``_superseded_by`` gets. Not a forgery barrier and not presented as one: + whoever can write this file can write a well-formed record. + """ + if not isinstance(record, dict): + return False + if record.get("scope") not in AUTHORITY_SCOPES: + return False + if not _is_index(record.get("declared_at_index")) or record["declared_at_index"] < 0: + return False + if not _is_index(record.get("expires_after_seals")) or record["expires_after_seals"] < 1: + return False + digest = record.get("artifact_sha256") + if not isinstance(digest, str) or len(digest) != 64: + return False + if any(c not in "0123456789abcdef" for c in digest): + return False + if not _is_index(record.get("artifact_bytes")) or record["artifact_bytes"] < 1: + return False + for key in ("principal", "artifact_path", "label"): + if not isinstance(record.get(key), str): + return False + return True + + +def _load_authorities(session: Session) -> tuple: + """Every readable authority record, plus how many were dropped. + + A record that does not validate is DROPPED. Dropping is the refusing + direction: an authority grants permission, so ignoring a malformed one + produces a refusal the operator can see and fix, while trusting it would + produce a permission nobody granted. A torn tail therefore costs a + citation, never a false one. + """ + path = _authority_path(session) + if not path.exists(): + return [], 0 + records = [] + dropped = 0 + for line in path.read_text(encoding="ascii", errors="replace").splitlines(): + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except ValueError: + dropped += 1 + continue + if _valid_authority(record): + records.append(record) + else: + dropped += 1 + return records, dropped + + +def _bind_authorities(session: Session, scopes: tuple, seal_index: int) -> list: + """The authorities this seal cites, graded ``cited``. + + Three conditions, every one of them fail-closed: + + * **Scope** — the record's scope is one this seal actually exercises. An + ordinary seal exercises none and therefore binds none, which is what keeps + an authority attached to exceptional transitions instead of becoming a line + on every note. + * **Not expired** — ``seal_index - declared_at_index < expires_after_seals``. + The clock starts at the seal that follows the authorisation, so + ``--expires-after-seals 1`` covers the next seal and not the one after. + * **Not from the future** — a record may not name a seal generation that has + not happened (``declared_at_index <= seal_index``). That is the guard + against a record written or backdated by hand to cover seals it never saw. + + ``limitation``, ``grade`` and ``principal_basis`` are written into every + bound entry from their SOURCE CONSTANTS so the note itself carries them, + not just the surfaces that render it — and so a hand-edited + ``authority.jsonl`` cannot dictate the three fields that state how much the + citation is worth. ``principal_basis`` belongs in that set because it + qualifies the principal: "self-asserted" is the only basis this version can + establish, so it is the only basis it will publish. + """ + records, dropped = _load_authorities(session) + if dropped: + print( + f"didrun seal: WARNING: dropped {dropped} unreadable authority " + f"record(s) from the ledger; they authorise nothing. A dropped " + f"record is the refusing direction, so a scope that needed one is " + f"refused rather than granted.", + file=sys.stderr, + ) + bound = [] + for record in records: + if record["scope"] not in scopes: + continue + declared = record["declared_at_index"] + if declared > seal_index: + continue + if seal_index - declared >= record["expires_after_seals"]: + continue + entry = dict(record) + entry["grade"] = AUTHORITY_GRADE + entry["limitation"] = AUTHORITY_LIMITATION + entry["principal_basis"] = AUTHORITY_PRINCIPAL_BASIS + bound.append(entry) + return bound + + +def _require_authority(scopes: tuple, bound: list) -> None: + """Refuse a transition whose scope no live cited authority covers. + + Default off, and it stays that way: making it default-on would put an + authorisation on the happy path, which is the failure mode this whole design + is shaped around. It exists for a CI gate that wants "a bare + `--allow-secrets` is not enough", and it says nothing stronger than that — + the authority it demands is still only a citation. + """ + covered = {entry["scope"] for entry in bound} + missing = [scope for scope in scopes if scope not in covered] + if not missing: + return + raise ManifestError( + "--require-authority: no live cited authority with scope " + + ", ".join(repr(s) for s in missing) + + f". Declare one with `didrun authorize --artifact --scope " + f"{missing[0]} --label \"\"` and re-run. {AUTHORITY_LIMITATION}" + ) + + +# How `_redact_authorities` must treat every key a bound authority carries. The +# split is exhaustive BY CONSTRUCTION: the pass publishes a key only if it is +# named here, so a field added later is dropped rather than published unscanned, +# and `test_every_authority_field_is_classified` fails the moment +# `declare_authority` grows a field nobody classified. +# +# Enumerating the text fields by hand, with no check that the enumeration was +# complete, is exactly what published `principal` verbatim into a git note while +# the refusal message promised a redacted export. The enumeration is no longer +# allowed to be silently incomplete. +_AUTHORITY_TEXT_FIELDS = ( + # (field name, the FindingSource kind its content is scanned as) + ("label", "label"), + ("artifact_path", "pathspec"), + # `--principal`, or `$USER` when that is absent. Outside text, exactly as + # much as the label is, and published in the same JSON object. + ("principal", "label"), +) + +# Computed here, validated by `_valid_authority`, or pinned from a source +# constant by `_bind_authorities`. None of these can carry outside text, which +# is why they are published as-is — the reason is recorded per field because +# "this one is safe" is the claim that has to stay true. +_AUTHORITY_PASSTHROUGH_FIELDS = frozenset( + { + "artifact_sha256", # computed here; validated as 64 lowercase hex + "artifact_bytes", # computed here; validated as an int >= 1 + "scope", # closed vocabulary (AUTHORITY_SCOPES) + "declared_at_index", # computed here; validated as an int >= 0 + "expires_after_seals", # validated as an int >= 1 + "principal_basis", # source constant, pinned at bind + "grade", # source constant, pinned at bind + "limitation", # source constant, pinned at bind + } +) + + +def _redact_authorities(bound: list) -> tuple: + """Scrub the operator-authored strings in the bound authorities. + + Returns ``(payload, findings, bytes_scanned)`` — the same three things + ``_redact_result`` returns for a claim, and for the same reason: the + findings ARE the export gate's input for these fields, so an authority + label cannot be blocked-but-published. That pair coming apart is the exact + defect the claim-field pass exists to have closed, and a field added later + reintroduces it unless it is redacted by the pass that reports it. + + THREE fields carry outside text: the label, the cited path and the principal. + They are listed in ``_AUTHORITY_TEXT_FIELDS`` and every one of them goes + through ``redact_field``. Everything else is published only if + ``_AUTHORITY_PASSTHROUGH_FIELDS`` vouches for it; an unclassified key is + dropped, because the failing direction for a field nobody has thought about + is to leave it out of the artifact that leaves the machine. + + Findings are located by field NAME rather than by an authority index — + ``FindingSource`` addresses claims and streams, and widening it is a change + to the projection contract, not to this unit — so the precise address + travels in ``redaction.fields`` instead. + """ + payload = [] + findings: list = [] + scanned = 0 + for index, entry in enumerate(bound): + fields: list = [] + + def _field(name: str, kind: str, value) -> str: + nonlocal scanned + text = value if isinstance(value, str) else str(value) + scanned += len(text.encode("utf-8", "replace")) + redaction = redact.redact_field(name, text, redact.FindingSource(kind)) + fields.extend(redaction.applied) + findings.extend(redaction.findings) + return redaction.text + + out = {k: v for k, v in entry.items() if k in _AUTHORITY_PASSTHROUGH_FIELDS} + for name, kind in _AUTHORITY_TEXT_FIELDS: + if name in entry: + out[name] = _field(f"authority[{index}].{name}", kind, entry[name]) + out["redaction"] = { + "projection_version": redact.PROJECTION_VERSION, + "detector_set_version": redact.DETECTOR_SET_VERSION, + "fields": fields, + } + payload.append(out) + return payload, findings, scanned + + # --- helpers ----------------------------------------------------------------- def _rev(repo: Path, commitish: str) -> Optional[str]: @@ -298,18 +1585,179 @@ def _rev(repo: Path, commitish: str) -> Optional[str]: return proc.stdout.strip() if proc.returncode == 0 else None +def _git_stderr(proc) -> str: + err = proc.stderr or b"" + if isinstance(err, bytes): + err = err.decode("utf-8", "replace") + return err.strip()[:500] + + +def _read_note(repo: Path, commit: str) -> Optional[bytes]: + """The note body currently attached to ``commit``, or None if there is none. + + Read before publishing so a failed seal can put back what it overwrote. + """ + proc = subprocess.run( + ["git", "notes", f"--ref={NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +def _recorded_window(body: Optional[bytes]) -> Optional[tuple]: + """The claim window an existing note records, or None when it records none. + + None is NOT ``(0, 0)`` and it is not the reading default either. A note + published before v0.2 carries no window keys at all, so what it covered is + genuinely unknown — and ``Manifest.from_json``'s defaults (0 / len(claims)) + would turn that unknown into a confident ``[0, N)`` the note never claimed, + which is exactly the kind of invented fact that would let the guard below + wave a lossy overwrite through. So this reads key PRESENCE off the raw body. + + A body this function cannot parse also yields None: a foreign or corrupt + note under the same ref is the case where an overwrite destroys the most and + explains the least. + """ + if not body: + return None + try: + d = json.loads(body) + except ValueError: + return None + if not isinstance(d, dict): + return None + lo, hi = d.get("claims_from"), d.get("claims_to") + if not _is_index(lo) or not _is_index(hi): + return None + return lo, hi + + +def _refuse_narrowing_reseal( + commit: str, prior_note: Optional[bytes], manifest: Manifest, reseal: bool +) -> None: + """Refuse to replace a commit's note with a narrower record. + + ``git notes add -f`` force-replaces, and a seal is scoped to + ``all_claims[watermark:]``. Together those two made the documented repair + loop lossy: re-sealing the same commit published a note holding only the + post-watermark claims and silently destroyed the record of everything the + earlier note held. Nothing in the artifact said so, which is what made it + dangerous rather than merely surprising. + + Three ways past this, and they are all explicit: the commit carries no note; + the new window CONTAINS the recorded one (a superset re-seal loses nothing); + or the operator passed ``reseal``. An unknown recorded window is not treated + as containable — see ``_recorded_window``. + """ + if prior_note is None or reseal: + return + new_window = (manifest.claims_from, manifest.claims_to) + old_window = _recorded_window(prior_note) + if old_window is not None and ( + new_window[0] <= old_window[0] and new_window[1] >= old_window[1] + ): + return + recorded = ( + "the note records no claim window (it was published before v0.2, or it is " + "not a didrun manifest), so what replacing it would destroy cannot be " + "determined from here" + if old_window is None + else ( + f"the note records claims [{old_window[0]},{old_window[1]}), " + f"which that does not contain" + ) + ) + raise ManifestError( + f"refusing to replace the note on {commit[:12]}: this seal covers claims " + f"[{new_window[0]},{new_window[1]}) and {recorded} — replacing the note " + f"would drop the earlier record, which exists nowhere else. Commit first " + f"so this unit gets its own note, or re-run with --reseal to replace it " + f"anyway (the narrower window is then visible in claims_from/claims_to)." + ) + + def _attach_note(repo: Path, commit: str, manifest: Manifest) -> None: - subprocess.run( - ["git", "notes", f"--ref={NOTES_REF}", "add", "-f", "-m", manifest.to_json().decode("ascii"), commit], + """Publish the manifest as ``commit``'s note. Fails closed. + + The body travels on stdin (``-F -``), not as an argv element: notes grow + with claim count and the command line does not. ``-F -`` and ``-m`` produce + the identical note object, so this moves no stored byte. + """ + proc = subprocess.run( + ["git", "notes", f"--ref={NOTES_REF}", "add", "-f", "-F", "-", commit], cwd=str(repo), + input=manifest.to_json(), capture_output=True, - text=True, + ) + if proc.returncode != 0: + raise ManifestError( + f"could not publish note to {NOTES_REF} for {commit}: " + f"git exited {proc.returncode}: {_git_stderr(proc)}" + ) + + +def _restore_note(repo: Path, commit: str, prior: Optional[bytes]) -> bool: + """Put ``commit``'s note back the way it was. Best-effort; never raises. + + ``_attach_note`` forces (-f), so a seal can overwrite an existing note. + Rolling back by REMOVING the note would turn "the watermark write failed" + into "the commit's earlier evidence is gone", so the prior body is + re-attached; removal happens only when there was nothing there before. + Returns whether the restore is known to have succeeded. + """ + try: + if prior is None: + proc = subprocess.run( + ["git", "notes", f"--ref={NOTES_REF}", "remove", "--ignore-missing", commit], + cwd=str(repo), + capture_output=True, + ) + else: + proc = subprocess.run( + ["git", "notes", f"--ref={NOTES_REF}", "add", "-f", "-F", "-", commit], + cwd=str(repo), + input=prior, + capture_output=True, + ) + return proc.returncode == 0 + except Exception: + return False + + +def _rollback_note( + repo: Path, commit: str, prior: Optional[bytes], published: bool +) -> str: + """Undo this seal's publication and say, exactly, what state the commit is in.""" + if not published: + return "no note was published, so none was rolled back" + restored = _restore_note(repo, commit, prior) + if prior is None: + return ( + "the note was rolled back and the commit was left noteless (it carried none before)" + if restored + else "the note could NOT be rolled back and this seal's note is STILL ATTACHED" + ) + return ( + "the note was rolled back and the commit's prior note was restored" + if restored + else "the note was NOT rolled back and the commit's prior note was NOT restored" ) def _resolve_manifest( repo: Path, commit: Optional[str], tree: Optional[str] -) -> tuple[Optional[Manifest], str]: +) -> tuple[Optional[Manifest], str, int]: + """Find the manifest for a commit. Returns (manifest, resolved_by, skipped). + + ``skipped`` counts notes on the tree-fallback path that could not be + parsed at all — a foreign note under the same ref, or a corrupt body. Those + are skipped so one bad note cannot hide every good one, but they are + counted so the caller can say so rather than reporting a silent "none". + A *version* refusal is not a skip: it means didrun met evidence it cannot + read, and it propagates. + """ + skipped = 0 # Try the note on the exact commit. if commit: proc = subprocess.run( @@ -319,7 +1767,7 @@ def _resolve_manifest( text=True, ) if proc.returncode == 0 and proc.stdout.strip(): - return Manifest.from_json(proc.stdout.encode("ascii")), "commit" + return Manifest.from_json(proc.stdout.encode("ascii")), "commit", skipped # Tree fallback: scan notes for one whose sealed tree matches (survives amend). if tree: listing = subprocess.run( @@ -343,35 +1791,332 @@ def _resolve_manifest( if show.returncode == 0 and show.stdout.strip(): try: m = Manifest.from_json(show.stdout.encode("ascii")) + except ManifestFormatError: + # Not a readable manifest: a foreign note under the same + # ref, or a corrupt body. Skipped and counted, so one bad + # note cannot hide every good one. Ordered BEFORE the + # clause below because it is a subclass of it — and it + # states by type what used to happen by accident, when a + # bare `except Exception` skipped these only because + # `json` raised something that was not a ManifestError. + skipped += 1 + continue + except ManifestError: + # A version this binary cannot read. Fail closed: the + # alternative is scanning past it and reporting the + # verdict of some older note as if it were current. + raise except Exception: + skipped += 1 continue + # First match wins (unchanged). Ambiguity policy is not + # this unit's; see docs/COMPAT.md. if m.tree == tree: - return m, "tree-fallback" - return None, "none" + return m, "tree-fallback", skipped + return None, "none", skipped + + +# What the secrets pass looked at, published so `secrets_override: false` is a +# checkable statement rather than the same word for "nothing found" and +# "almost nothing looked at". `claim-fields` is the label, the pathspecs and +# the changed paths: they used to ride into the scan only inside the +# serialized manifest, where they blocked a seal and were published anyway. +SCAN_DOMAIN = "claim-bound-events+claim-fields+manifest" -def _scan_for_secrets(session: Session, results: list[GradeResult]) -> list[redact.Finding]: - findings: list[redact.Finding] = [] +@dataclass +class SecretsScan: + """What the secrets pass looked at, and what it found where. + + Two domains, and the distinction is the whole point. ``export`` is what is + about to leave the machine: the argv of each claim-bound event and the + serialized manifest bytes themselves. ``local`` is what stays: the stdout + and stderr blobs, which live in a gitignored ledger and are never published. + Blocking a seal over bytes that never leave, while publishing claim labels + and delta paths entirely unscanned, was the gate pointed at the wrong half. + """ + + export: list # findings in bytes about to be written + local: list # findings in blobs that stay on this machine + events_scanned: int + events_total: int + bytes_scanned: int + + @property + def blocking(self) -> list: + """The findings that stop a seal: block-tier, in the export domain.""" + return [f for f in self.export if f.tier == redact.TIER_BLOCK] + + @property + def non_blocking(self) -> list: + """Everything found that did not stop the seal — reported, not silent.""" + return [f for f in self.export if f.tier != redact.TIER_BLOCK] + list(self.local) + + @property + def all_findings(self) -> list: + return self.export + self.local + + def to_dict(self, overridden: bool) -> dict: + by_tier = {tier: 0 for tier in redact.TIERS} + for f in self.all_findings: + by_tier[f.tier] = by_tier.get(f.tier, 0) + 1 + return { + "scan_domain": SCAN_DOMAIN, + "events_scanned": self.events_scanned, + "events_total": self.events_total, + "bytes_scanned": self.bytes_scanned, + "detector_set_version": redact.DETECTOR_SET_VERSION, + "findings_by_tier": by_tier, + "blocked": bool(self.blocking), + "overridden": overridden, + } + + +def _scan_for_secrets( + session: Session, + results: list[GradeResult], + export_bytes: bytes, + field_findings: Optional[list] = None, + field_bytes: int = 0, +) -> SecretsScan: + """Scan both domains, tagging every finding with where it is. + + Three contributions make up the export domain, and they are deliberately + not the same mechanism: + + * the argv of each claim-bound event, scanned RAW here and redacted into + the published `argv_preview` separately; + * ``field_findings`` — the claim fields (label, pathspecs, changed paths), + which arrive already located from the pass that redacted them, so what + blocks and what is scrubbed cannot drift apart; + * ``export_bytes``, the serialized manifest as it would be published, + scanned WHOLE rather than field by field. It is the backstop: a field a + later version adds and forgets to redact is still in these bytes, so it + still blocks. The redacted fields are already markers by the time they + get here, so nothing is counted twice. + + The scan covers each claim's supporting event once, not once per claim, so + ``events_scanned`` counts events rather than claim-event pairs. + """ + export: list[redact.Finding] = list(field_findings or ()) + local: list[redact.Finding] = [] events = session.events() + seen: set = set() + # Starts at the claim-field bytes rather than 0: those bytes were scanned, + # by the redaction pass, and a published counter that omitted them would be + # the same "checkable statement that is not true" this block exists to fix. + bytes_scanned = field_bytes + for r in results: idx = r.supporting_event_index - if idx is None or not (0 <= idx < len(events)): + if idx is None or not (0 <= idx < len(events)) or idx in seen: continue + seen.add(idx) ev = events[idx] - findings.extend(redact.scan(" ".join(ev.argv))) - for blob in (ev.stdout_blob, ev.stderr_blob): + for argv_index, arg in enumerate(ev.argv): + bytes_scanned += len(arg.encode("utf-8", "replace")) + export.extend( + redact.scan( + arg, + redact.FindingSource( + "argv", event_index=idx, argv_index=argv_index + ), + ) + ) + for stream, blob in (("stdout", ev.stdout_blob), ("stderr", ev.stderr_blob)): if blob and session.blobs.has(blob): - findings.extend(redact.scan(session.blobs.get(blob))) - return findings + data = session.blobs.get(blob) + bytes_scanned += len(data) + local.extend( + redact.scan(data, redact.FindingSource(stream, event_index=idx)) + ) + + bytes_scanned += len(export_bytes) + export.extend(redact.scan(export_bytes, redact.FindingSource("manifest"))) + + return SecretsScan( + export=export, + local=local, + events_scanned=len(seen), + events_total=len(events), + bytes_scanned=bytes_scanned, + ) + + +def _warn_unblocked_secrets(scan: SecretsScan) -> None: + """Say, loudly, what was found and not blocked on. + + Silence here would make the tiering a suppression list with extra steps. + Notice-tier findings and block-tier findings confined to local blobs do not + stop a seal, but the operator is told they exist and where they are. + """ + findings = scan.non_blocking + if not findings: + return + notice = sum(1 for f in findings if f.tier == redact.TIER_NOTICE) + local_block = len(findings) - notice + print( + f"didrun seal: WARNING: {len(findings)} secret finding(s) did not block " + f"this seal ({notice} notice-tier, {local_block} block-tier in local " + f"output blobs that are not exported):", + file=sys.stderr, + ) + for line in redact.enumerate_findings(findings): + print(line, file=sys.stderr) + + +@dataclass(frozen=True) +class RedactedClaim: + """One exported claim entry, plus what redacting it found. + + ``findings`` is not diagnostic output: it is the export gate's input for + every field of this claim that is not argv. Producing the two together is + the only reason the gate cannot disagree with the publication again. + ``bytes_scanned`` keeps the manifest's published byte count honest about + the fields this pass added to the domain. + """ + + payload: dict + findings: list + bytes_scanned: int = 0 -def _redact_result(r: GradeResult, session: Session) -> dict: +def _redact_result( + r: GradeResult, + session: Session, + entries: list, + claim_index: int = 0, + superseded_by: Optional[int] = None, +) -> RedactedClaim: d = r.to_dict() - # Redact argv in the exported claim view. - d["claim"]["argv_preview"] = redact.scrub_argv( + # Redact argv in the exported claim view, and DECLARE the projection that + # produced it. The marker string alone is not authenticating — a recorded + # argv may contain the literal marker text — so a consumer distinguishes a + # real redaction from literal text by reading `redaction.applied`, which + # names the source spans that were replaced. Additive, read with .get(): + # no MANIFEST_VERSION bump (docs/COMPAT.md). + argv_redaction = redact.redact_argv( list(_event_argv(session, r.supporting_event_index)) ) - return d + d["claim"]["argv_preview"] = argv_redaction.argv + + # The rest of the published claim. argv used to be the only field scrubbed + # here, which made the seal's own refusal message ("re-run with + # --allow-secrets to export a redacted artifact anyway") false for every + # other field: a token in a claim label BLOCKED the seal and was then + # published verbatim into refs/notes/didrun on the override. These are the + # exported strings that carry operator- or repository-authored text, and + # each one is redacted by the same pass that reports it. + fields: list[dict] = [] + findings: list = [] + scanned = 0 + + def _field(name: str, kind: str, value: str) -> str: + nonlocal scanned + scanned += len(value.encode("utf-8", "replace")) + redaction = redact.redact_field( + name, + value, + redact.FindingSource(kind, claim_index=claim_index), + ) + fields.extend(redaction.applied) + findings.extend(redaction.findings) + return redaction.text + + d["claim"]["label"] = _field("claim.label", "label", d["claim"]["label"]) + d["claim"]["pathspecs"] = [ + _field(f"claim.pathspecs[{i}]", "pathspec", spec) + for i, spec in enumerate(d["claim"]["pathspecs"]) + ] + # A conjunct name is the LABEL of another claim, so it is scanned as one. + # Every conjunct is redacted by the same deterministic projection that + # redacted the label it refers to, which is what keeps a conjunction + # resolvable at verify time: both sides of the match carry the same text. + d["claim"]["conjuncts"] = [ + _field(f"claim.conjuncts[{i}]", "label", name) + for i, name in enumerate(d["claim"]["conjuncts"]) + ] + # `reason` is generated text for every other claim type, and it was exported + # unredacted for that reason (docs/COMPAT.md). A conjunction's reason NAMES a + # conjunct, which is operator-authored, so it stops being generated-only text + # here — and "blocked by the gate, published verbatim on the override" is the + # exact defect the claim-field pass exists to have closed. Redacted for every + # claim, not only for a conjunction: a reason a later version interpolates + # something into is then covered without anyone remembering to add it. + d["reason"] = _field("reason", "reason", d["reason"]) + d["delta"] = [ + { + "status": change["status"], + "path": _field(f"delta[{i}].path", "delta-path", change["path"]), + } + for i, change in enumerate(d["delta"]) + ] + + d["redaction"] = { + "projection_version": redact.PROJECTION_VERSION, + "detector_set_version": redact.DETECTOR_SET_VERSION, + "applied": argv_redaction.applied, + # Additive and separate from `applied`, which stays argv-addressed by + # `argv_index` exactly as v0.2 consumers already read it. Entries here + # are addressed by field name instead, because a label has no argv + # index. A reader that does not know this key sees the same argv + # projection it always did. + "fields": fields, + } + pairs = d.pop("conjunct_grades", None) + if pairs is not None: + # The pairs' labels ARE `claim.conjuncts`, one per declared conjunct in + # declaration order (claims.grade_conjunction builds them that way), so + # they take the redacted text from above instead of being scanned — and + # reported to the export gate — a second time. The length check is not + # decoration: a silent zip would drop a graded conjunct out of the + # published record, so it fails closed instead. + names = d["claim"]["conjuncts"] + if len(pairs) != len(names): + raise ManifestError( + f"refusing to publish a conjunction claim carrying " + f"{len(pairs)} graded conjunct(s) for {len(names)} declared " + f"conjunct(s): the note would not say what each member earned" + ) + d["conjunct_grades"] = [ + {"label": name, "grade": pair["grade"]} + for name, pair in zip(names, pairs) + ] + block = _evidence_block(entries, r.supporting_event_index) + if block is not None: + d["evidence"] = block + # Written only on the entries that carry one. A permanently-null key on + # every claim in every note would say nothing, and the absence of the key is + # already the complete statement "this claim was not retired". + if superseded_by is not None: + d["superseded_by"] = superseded_by + return RedactedClaim(payload=d, findings=findings, bytes_scanned=scanned) + + +def _evidence_block(entries: list, idx: Optional[int]) -> Optional[dict]: + """Which recorded entry backed this claim, named by its chain hash. + + This is the handle the note never had. A sealed grade used to be reachable + only through an integer index into whatever ledger happens to be on disk at + verify time, so an archived, rebuilt or substituted ledger changed the + verdict with no signal. The entry hash makes "index 4 of the ledger you have + now" checkable against "index 4 of the ledger this was sealed from". + + Recorded in the MANIFEST, never in the ledger: the v1 chain preimage is + frozen (docs/COMPAT.md), and putting a field into Event would invalidate + every entry hash ever written. Omitted entirely when no event backs the + claim — an absent block means "nothing to bind", which is not the same fact + as "bound to nothing". + """ + if idx is None or not (0 <= idx < len(entries)): + return None + entry = entries[idx] + return { + "event_index": idx, + "entry_hash": entry.entry_hash, + "env_fingerprint": entry.event.env_fingerprint, + "tree_after": entry.event.tree_after, + } def _event_argv(session: Session, idx: Optional[int]): diff --git a/src/didrun/redact.py b/src/didrun/redact.py index e4d34cc..12948f4 100644 --- a/src/didrun/redact.py +++ b/src/didrun/redact.py @@ -5,27 +5,211 @@ turn a trust tool into a leak ("didrun leaked our staging token" is category-killing; see docs/TRUST_MODEL.md). -This pass runs at seal/export: it scans argv and manifest-bound blobs, BLOCKS -export by default when a likely secret is found, and always redacts what reaches -an exported/rendered artifact. The raw *local* ledger is left intact (it is -gitignored and local); only what leaves the machine is scrubbed. +This pass runs at seal/export: it scans argv, the bytes about to be published, +and the manifest-bound output blobs. The raw *local* ledger is left intact (it +is gitignored and local); only what leaves the machine is scrubbed. What BLOCKS +is scoped to the export domain for the same reason: stopping a seal over bytes +that never leave the machine, while publishing claim labels unscanned, is the +gate pointed at the wrong half of the tool (see manifest.py's `_scan_for_secrets`). + +Blocking and redacting cover the SAME fields, and saying so is not a tautology +— they did not, and the gap was a leak. Every string in the published claim +entry that carries operator- or repository-authored text (the argv preview, the +label, the pathspecs, the changed paths) is replaced by a marker before it is +serialized. A field that only blocked was published raw the moment anyone +passed --allow-secrets, under a refusal message that had already promised +"export a redacted artifact anyway". So ``redact_field`` returns findings along +with text: for those fields, being scrubbed and being reported to the gate are +one act, not two passes that can drift apart. The heuristic is deliberately tuned to bound false positives — a benign 40-char git hash must NOT trip it, or the tool blocks everything and gets disabled. + +Redaction is a SPAN PARTITION, not a splice. ``scan`` produces overlapping +findings routinely (``OPENAI_API_KEY=sk-…`` matches two patterns at once), and +replacing overlapping spans one at a time silently deletes whatever followed the +outermost one — the exported record then shows a different command than the one +that ran, which is an evidence-integrity bug, not a cosmetic one. ``redact`` +therefore merges findings into disjoint maximal spans and rebuilds the string by +walking that partition, so the output is exactly reconstructible: substituting +each marker back with its recorded source span reproduces the input byte for +byte. + +The marker string alone is NOT authenticating: a recorded argv can contain the +literal text ``«redacted:openai-key»``, and nothing here escapes it. What a +consumer reads instead is the applied partition, which ``seal`` publishes into +the manifest as ``redaction.applied``. Keyed markers would close the forgery +gap and are deliberately not built (they need a per-ledger key store, and a +per-ledger key destroys the cross-ledger stability that motivated them). + +Detectors are TIERED, and the two tiers say different things. The six +structured patterns fire on shape and name what they found: ``block``. The +entropy sweep fires on "this looked random", which over a 24+ character run is +a statement about length far more than about secrecy — per-character entropy is +capped at log2(n), so roughly 18 distinct characters clears the threshold. It +is ``notice``: reported, never blocking, and never deleted, because it is the +only detector that catches an unstructured credential. + +Every finding carries its own locality (``source``, ``line``, and the offset +inside that source), because "346 likely secret(s) found" with no address is +not something an operator can act on. """ from __future__ import annotations +import hashlib import math import re from dataclasses import dataclass +from typing import Optional + +# What a finding does to an export. `block` is a structured detector that named +# what it found; `notice` is reported and warned about, never blocking. +TIER_BLOCK = "block" +TIER_NOTICE = "notice" +TIERS = (TIER_BLOCK, TIER_NOTICE) + +# Where a finding is. `argv`/`stdout`/`stderr` address a recorded event; +# `label`, `pathspec`, `delta-path` and `reason` address individual exported +# claim fields; `manifest` addresses the serialized bytes about to be published. +SOURCE_KINDS = ( + "argv", + "stdout", + "stderr", + "label", + "pathspec", + "delta-path", + "reason", + "manifest", +) + +# Source kinds that address ONE exported field rather than a stream of bytes. +# A line number inside them would always be 1, which is noise, not locality. +_LINELESS_KINDS = ("argv", "label", "pathspec", "delta-path", "reason") + + +@dataclass(frozen=True) +class FindingSource: + """Where a finding is, in the coordinates of the text that was scanned. + + ``event_index`` is the ledger index of the event whose bytes were scanned + (None when the scanned text is not an event's). ``claim_index`` is the + seal-relative claim whose field was scanned (None for anything that is not + a claim field): a label belongs to a claim, not to an event, and without it + two claims carrying a secret at the same offset are one location. + ``argv_index`` is the argv element (None for anything that is not argv). + ``byte_offset`` is the offset of the finding inside that text, so + ``text[byte_offset:]`` begins at it. The detectors only match ASCII, so + across a match it is a byte count; in non-ASCII surroundings it counts + decoded characters. + """ + + kind: str + event_index: Optional[int] = None + argv_index: Optional[int] = None + byte_offset: int = 0 + claim_index: Optional[int] = None + + def __post_init__(self) -> None: + if self.kind not in SOURCE_KINDS: + raise ValueError(f"unknown finding source kind: {self.kind!r}") @dataclass(frozen=True) class Finding: + """One detector hit. + + The first three fields are the original positional triple, in their + original order, so every existing construction still works unchanged. + Everything tier- and locality-related is additive with a default. + """ + kind: str span: tuple[int, int] preview: str # already-masked preview, never the raw secret + detector_id: str = "" + detector_version: int = 1 + tier: str = TIER_NOTICE + source: Optional[FindingSource] = None + line: Optional[int] = None # 1-based within the scanned text; None for argv + fingerprint: str = "" + + +@dataclass(frozen=True) +class AppliedSpan: + """One disjoint span that was actually replaced, in SOURCE coordinates. + + This is the exported projection's unit. ``kind`` is the marker text that + took its place; ``detector_id`` names the detector whose extent won the + merge. ``withheld`` marks the honest-absence path: the element was dropped + whole rather than rewritten, so the span is the element, not a token. + """ + + start: int + end: int + kind: str + detector_id: str + withheld: bool = False + + +@dataclass(frozen=True) +class RedactionResult: + text: str + findings: list[Finding] + applied: list[AppliedSpan] + + +@dataclass(frozen=True) +class ArgvRedaction: + """Scrubbed argv plus the projection that produced it. + + ``applied`` is the wire shape ``seal`` writes into the manifest, one entry + per replaced span, addressed by ``argv_index``. + """ + + argv: list[str] + applied: list[dict] + + +@dataclass(frozen=True) +class FieldRedaction: + """One scrubbed exported field, its projection, and what it found. + + ``findings`` is the part that is not a convenience: it is what makes + "redacted" and "reported to the export gate" the same act for a field, + rather than two passes that can disagree. + """ + + text: str + findings: list[Finding] + applied: list[dict] + + +# The exported projection's contract version. Bump PROJECTION_VERSION when the +# RULE that turns findings into replaced spans changes (merge policy, marker +# placement, the withheld path). Bump DETECTOR_SET_VERSION when the set of +# detectors or what they match changes. A consumer reads these instead of +# maintaining a matrix of didrun's historical behaviour. +# +# 2: the entropy sweep decides per run whether `/` is a path separator or a +# base64 alphabet member, instead of scoring every run the same way, and every +# detector now declares a tier. That is a change to what the detectors match, +# which is exactly what this number is for — leaving it at 1 would re-create +# the "read the changelog and infer it" problem the version exists to remove. +# +# It stays at 2 rather than going to 3: version 2 has never been released, so +# there is no consumer anywhere that saw the intermediate "tokenizer simply +# dropped `/`" behaviour. Bumping to 3 would publish a version number for a +# detector set that never shipped, which is the opposite of what these numbers +# are for. +# +# The span-partition ALGORITHM did not change, so PROJECTION_VERSION stays 1. +# What the projection COVERS did widen (claim fields, not just argv), and that +# is readable directly off the note: argv spans are in `redaction.applied` and +# field spans are in `redaction.fields`. A consumer checks for the key rather +# than consulting a version matrix. +PROJECTION_VERSION = 1 +DETECTOR_SET_VERSION = 2 # High-confidence structured patterns. These fire on shape, not entropy. @@ -33,7 +217,11 @@ class Finding: ("aws-access-key", re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b")), ("github-token", re.compile(r"\bgh[pousr]_[A-Za-z0-9]{36,}\b")), ("slack-token", re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{10,}\b")), - ("openai-key", re.compile(r"\bsk-[A-Za-z0-9]{20,}\b")), + # `[-_]` inside the body, or `sk-proj-…` (and every other prefixed key + # OpenAI has shipped since) matches nothing: the internal hyphen ends the + # body under `[A-Za-z0-9]`. Leading \b stays, so a `sk-` inside a word + # (`risk-...`) is still not a key. + ("openai-key", re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b")), ("pem-private-key", re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----")), # KEY=value / TOKEN=value / SECRET=value / PASSWORD=value assignments in argv. ( @@ -45,14 +233,71 @@ class Finding: ] _REDACTION_MARKER = "«redacted:{kind}»" +# Redact-or-withhold, never mangle: when a finding cannot be expressed as a +# whole-span replacement, the element is dropped whole rather than rewritten +# into something that reads authoritative and is wrong. +_WITHHELD_MARKER = "«withheld:{kind}»" + +# Detector specificity for merge tie-breaks: a structured pattern names what it +# found, the entropy sweep only names that something looked random. +_ENTROPY_KIND = "high-entropy" # Entropy heuristic bounds. A long, high-entropy, non-hex token that isn't a git # hash is suspicious. Git SHAs are hex (entropy ~4.0 bits/char over [0-9a-f]), # so requiring >4.2 bits/char over a larger alphabet excludes them. +# +# The length floor and the threshold are deliberately UNCHANGED from v0.1: one +# variable at a time, or the measurement stops being interpretable. What moved +# is the CANDIDATE — what gets scored — and it moved twice. +# +# `/` was in the token class in v0.1, and that single character was the +# detector's dominant false-positive source: with `/` included, an entire +# absolute filesystem path is one token, and a path accumulates distinct +# characters faster than any single segment does (179 of 2,642 real paths +# cleared the threshold whole; none clears it per segment). +# +# But `/` is also 1 of the 64 characters of base64, so dropping it outright +# stopped scoring credentials as well as paths: a base64 secret whose +# slash-free runs are all shorter than the floor became invisible — not +# demoted, not re-scored, INVISIBLE, and therefore also unredacted, because +# replacement is driven off the findings. Measured, that cost 15.7% of +# AWS-secret-access-key-shaped strings and 25.3% of base64(24 bytes). +# +# So the separator is not decided by the tokenizer at all. A run is taken over +# the class INCLUDING `/`, and `_looks_like_path` decides whether the `/` in it +# is a separator (score the segments) or an alphabet member (score the run). +# See docs/MEASUREMENTS.md for both directions of that decision. _ENTROPY_MIN_LEN = 24 _ENTROPY_MIN_BITS_PER_CHAR = 4.2 _HEXISH = re.compile(r"^[0-9a-fA-F]+$") -_TOKENISH = re.compile(r"[A-Za-z0-9_\-+/=]{%d,}" % _ENTROPY_MIN_LEN) +# One SEGMENT: the credential alphabet with the path separator removed. This is +# the unit a path is scored in. +_TOKENISH = re.compile(r"[A-Za-z0-9_\-+=]{%d,}" % _ENTROPY_MIN_LEN) +# One RUN: the same class plus `/`, which is the unit a base64 credential is +# scored in. A path and a base64 blob are the SAME run shape — they are told +# apart by what the segments look like, below, never by the character class. +_RUNISH = re.compile(r"[A-Za-z0-9_\-+/=]{%d,}" % _ENTROPY_MIN_LEN) + +# What tells a filesystem path from a base64 credential, given that both are +# written in [A-Za-z0-9+/]: a path is a sequence of NAMES, and base64 is not. +# A name is mostly letters and those letters are mostly lower case — `usr`, +# `homebrew`, `site-packages`, and CamelCase like `CoreServices` (10/12) or +# `Frameworks` (9/10) all clear 0.8, while base64 is ~50% lower case by +# construction and clears it only by accident. +# +# ONE name is enough to call the whole run a path. That direction is the +# conservative one: it can only move a run from whole-scoring back to the +# segment-scoring that ships today, so it cannot introduce a false positive +# that today's code does not already have — measured at exactly 0 new ones over +# 46,403 real paths. It costs recall instead (~1-3%), and that is the honest +# side to lose on for a notice-tier heuristic. +_NAME_MIN_LEN = 3 +_NAME_LETTER_FRACTION = 0.6 +_NAME_LOWERCASE_FRACTION = 0.8 + +# Per-detector revisions. Distinct from DETECTOR_SET_VERSION: this says "this +# detector's own matcher changed", the other says "the set changed". +_DETECTOR_VERSIONS = {_ENTROPY_KIND: 2} def _shannon_bits_per_char(s: str) -> float: @@ -69,11 +314,118 @@ def _mask(kind: str) -> str: return _REDACTION_MARKER.format(kind=kind) -def scan(data) -> list[Finding]: +def _withheld_mask(kind: str) -> str: + return _WITHHELD_MARKER.format(kind=kind) + + +def _tier(kind: str) -> str: + """Structured patterns block; the entropy sweep only notices.""" + return TIER_NOTICE if kind == _ENTROPY_KIND else TIER_BLOCK + + +def _is_name(segment: str) -> bool: + """Is this `/`-delimited segment a NAME (a directory or file name)? + + Mostly letters, and those letters mostly lower case. This is the whole + path-vs-base64 discriminator; see the constants above for why it is stated + over case rather than over length or slash density (both of those + distributions overlap — measured). + """ + if len(segment) < _NAME_MIN_LEN: + return False + letters = [c for c in segment if c.isalpha()] + if len(letters) < _NAME_LETTER_FRACTION * len(segment): + return False + lowercase = sum(1 for c in letters if c.islower()) + return lowercase >= _NAME_LOWERCASE_FRACTION * len(letters) + + +def _looks_like_path(run: str) -> bool: + """Is the `/` in this run a separator (True) or an alphabet member (False)?""" + return any(_is_name(segment) for segment in run.split("/")) + + +def _entropy_qualifies(token: str) -> bool: + """The unchanged v0.1 test, applied to whatever candidate is handed to it.""" + if len(token) < _ENTROPY_MIN_LEN: + return False + if _HEXISH.match(token): + return False # git hashes and hex digests are not secrets + return _shannon_bits_per_char(token) >= _ENTROPY_MIN_BITS_PER_CHAR + + +def _entropy_candidates(run: str, base: int): + """Yield ``(offset, token)`` for every unit of ``run`` that gets scored. + + Three cases, and the third is the one that carries the fix: + + * no `/` at all — the run is the token, exactly as it has always been. + * `/` as a separator — score the segments, which is what keeps a filesystem + path from being scored as one 90-character token. + * `/` as an alphabet member — score the run whole, which is what catches a + base64 credential that its own slashes chop into sub-floor pieces. If the + whole run does NOT clear the bar, fall through to the segments anyway, so + a long low-entropy prefix cannot dilute a high-entropy tail below the + threshold and hide it. + """ + if "/" not in run: + yield base, run + return + if not _looks_like_path(run) and _entropy_qualifies(run): + yield base, run + return + for m in _TOKENISH.finditer(run): + yield base + m.start(), m.group(0) + + +def locality_fingerprint(detector_id: str, source: FindingSource) -> str: + """A stable handle on WHERE a finding is. Never on WHAT it matched. + + The token is deliberately not digested: an unkeyed digest of a short + high-entropy token is a brute-force oracle for the very thing that was just + redacted, and publishing one in a manifest would undo the redaction for + anyone willing to spend a GPU-hour. The consequence is that this moves when + the content around it moves — it identifies a location, not a secret, and + it is for reporting, never for suppression (there is no suppression). + """ + material = ( + f"{detector_id}\0{source.kind}\0{source.event_index}\0{source.byte_offset}" + f"\0{source.claim_index}" + ) + return hashlib.sha256(material.encode("utf-8")).hexdigest()[:16] + + +def _locate(kind: str, span: tuple[int, int], text: str, source: Optional[FindingSource]): + """The (source, line, fingerprint) triple for one hit.""" + if source is None: + return None, None, "" + here = FindingSource( + kind=source.kind, + event_index=source.event_index, + argv_index=source.argv_index, + byte_offset=source.byte_offset + span[0], + claim_index=source.claim_index, + ) + # A single exported field (an argv element, a label, a pathspec, a changed + # path) has no lines; a blob does, and the line is what makes a finding in + # 4 MB of test output reachable. + line = ( + None + if here.kind in _LINELESS_KINDS + else text.count("\n", 0, span[0]) + 1 + ) + return here, line, locality_fingerprint(kind, here) + + +def scan(data, source: Optional[FindingSource] = None) -> list[Finding]: """Return findings for likely secrets in ``data`` (str or bytes). Structured patterns first, then a bounded entropy sweep that explicitly skips hex-only tokens (git hashes) to keep false positives low. + + ``source`` is optional and additive: pass one and every finding comes back + located (source kind, event/argv index, line, offset, fingerprint); omit it + and the findings are exactly the shape they have always been. """ if isinstance(data, bytes): text = data.decode("utf-8", "replace") @@ -82,57 +434,308 @@ def scan(data) -> list[Finding]: findings: list[Finding] = [] matched_spans: list[tuple[int, int]] = [] + def _make(kind: str, span: tuple[int, int]) -> Finding: + here, line, fingerprint = _locate(kind, span, text, source) + return Finding( + kind=kind, + span=span, + preview=_mask(kind), + detector_id=kind, + detector_version=_DETECTOR_VERSIONS.get(kind, 1), + tier=_tier(kind), + source=here, + line=line, + fingerprint=fingerprint, + ) + for kind, pat in _PATTERNS: for m in pat.finditer(text): - findings.append(Finding(kind=kind, span=m.span(), preview=_mask(kind))) + findings.append(_make(kind, m.span())) matched_spans.append(m.span()) - for m in _TOKENISH.finditer(text): - tok = m.group(0) - if _HEXISH.match(tok): - continue # git hashes and hex digests are not secrets - if any(s <= m.start() < e for s, e in matched_spans): - continue # already covered by a structured pattern - if _shannon_bits_per_char(tok) >= _ENTROPY_MIN_BITS_PER_CHAR: - findings.append( - Finding(kind="high-entropy", span=m.span(), preview=_mask("high-entropy")) - ) + for m in _RUNISH.finditer(text): + for offset, tok in _entropy_candidates(m.group(0), m.start()): + if any(s <= offset < e for s, e in matched_spans): + continue # already covered by a structured pattern + if _entropy_qualifies(tok): + findings.append(_make(_ENTROPY_KIND, (offset, offset + len(tok)))) return findings -def redact(data) -> tuple[str, list[Finding]]: - """Return (redacted_text, findings). Matches are replaced by stable markers.""" +def _specificity(kind: str) -> int: + """Lower is more specific. A structured detector names what it found.""" + return 1 if kind == _ENTROPY_KIND else 0 + + +def _preferred(a: Finding, b: Finding) -> Finding: + """Which of two overlapping findings labels the merged span. + + Outermost extent first — the wider finding is the one whose replacement + covers the other's bytes. On an exact tie, the more specific detector, so a + structured match is never relabelled as generic entropy. Ties beyond that + keep the earlier finding, which makes the choice deterministic rather than + dependent on pattern iteration order. + """ + a_width = a.span[1] - a.span[0] + b_width = b.span[1] - b.span[0] + if b_width > a_width: + return b + if b_width == a_width and _specificity(b.kind) < _specificity(a.kind): + return b + return a + + +def merge_findings(findings) -> list[AppliedSpan]: + """Collapse findings into DISJOINT MAXIMAL spans, left to right. + + ``scan`` emits overlapping findings by construction: the entropy sweep's + dedup tests start containment only, and the structured patterns are never + deduped against each other, so `MY_TOKEN=ghp_…` yields both an + `env-secret-assignment` span and a `github-token` span ending at the same + offset. Replacing those one at a time deletes the text after the outer span. + Merging first is what makes replacement a partition. + """ + ordered = sorted(findings, key=lambda f: (f.span[0], -f.span[1])) + merged: list[AppliedSpan] = [] + winners: list[Finding] = [] # the finding whose kind currently labels merged[i] + for f in ordered: + start, end = f.span + if merged and start < merged[-1].end: + prev = merged[-1] + winner = _preferred(winners[-1], f) + winners[-1] = winner + merged[-1] = AppliedSpan( + start=prev.start, + end=max(prev.end, end), + kind=winner.kind, + detector_id=winner.detector_id or winner.kind, + ) + else: + merged.append( + AppliedSpan( + start=start, + end=end, + kind=f.kind, + detector_id=f.detector_id or f.kind, + ) + ) + winners.append(f) + return merged + + +def redact(data, source: Optional[FindingSource] = None) -> RedactionResult: + """Replace every finding with a marker, as a partition of the input. + + The output is built by walking the merged spans left to right and + concatenating alternating kept slices and markers — never by index + arithmetic on a string that is changing length underneath the offsets. + + ``source`` is optional and additive, and it matters for one reason beyond + reporting: a caller that both redacts a field and needs the findings for + the export gate gets a LOCATED set here, so the same pass that removes the + secret is the one that reports it. Redacting and scanning in two places is + what let a claim label block a seal and be published anyway. + """ if isinstance(data, bytes): text = data.decode("utf-8", "replace") else: text = data - findings = scan(text) - # Replace from the end so earlier spans keep their offsets. - ordered = sorted(findings, key=lambda f: f.span[0], reverse=True) - out = text - for f in ordered: - s, e = f.span - out = out[:s] + _mask(f.kind) + out[e:] - return out, findings + findings = scan(text, source) + applied = merge_findings(findings) + parts: list[str] = [] + cursor = 0 + for span in applied: + parts.append(text[cursor:span.start]) + parts.append(_mask(span.kind)) + cursor = span.end + parts.append(text[cursor:]) + return RedactionResult(text="".join(parts), findings=findings, applied=applied) + + +def reconstruct(source: str, result: RedactionResult) -> str: + """Rebuild ``source`` from the redacted text plus the applied partition. + + Kept slices come from the OUTPUT and replaced slices from the source, so a + deletion, a stale offset or a misplaced marker all fail to reproduce the + input. Markers are located arithmetically, never searched for: a recorded + argv is allowed to contain the literal marker text. + """ + rebuilt: list[str] = [] + out_cursor = 0 + src_cursor = 0 + for span in result.applied: + kept = span.start - src_cursor + if kept < 0: + raise ValueError("applied spans are not disjoint and ascending") + rebuilt.append(result.text[out_cursor:out_cursor + kept]) + out_cursor += kept + marker = _mask(span.kind) + if result.text[out_cursor:out_cursor + len(marker)] != marker: + raise ValueError("no marker at the offset the projection names") + out_cursor += len(marker) + rebuilt.append(source[span.start:span.end]) + src_cursor = span.end + rebuilt.append(result.text[out_cursor:]) + return "".join(rebuilt) + + +def _projection_entry(argv_index: int, span: AppliedSpan) -> dict: + return { + "argv_index": argv_index, + "start": span.start, + "end": span.end, + "detector_id": span.detector_id, + "withheld": span.withheld, + } + + +def redact_argv(argv) -> ArgvRedaction: + """Redact each argv element and report the projection that was applied. + + Post-condition per element: the redaction must reconstruct the element + exactly. It cannot fail once ``merge_findings`` has run — but if it ever + does, the element is WITHHELD whole rather than exported in a mangled form. + An honest absence beats a corrupted string that reads authoritative. + """ + scrubbed: list[str] = [] + applied: list[dict] = [] + for index, arg in enumerate(argv): + result = redact(arg) + source = arg.decode("utf-8", "replace") if isinstance(arg, bytes) else arg + try: + faithful = reconstruct(source, result) == source + except ValueError: + faithful = False + if faithful: + scrubbed.append(result.text) + applied.extend(_projection_entry(index, s) for s in result.applied) + continue + kind = result.applied[0].kind if result.applied else "unknown" + scrubbed.append(_withheld_mask(kind)) + applied.append( + _projection_entry( + index, + AppliedSpan( + start=0, + end=len(source), + kind=kind, + detector_id=kind, + withheld=True, + ), + ) + ) + return ArgvRedaction(argv=scrubbed, applied=applied) def scrub_argv(argv) -> list[str]: """Redact secret-shaped tokens inside argv elements (e.g. FOO=secret).""" - scrubbed = [] - for arg in argv: - red, _ = redact(arg) - scrubbed.append(red) - return scrubbed + return redact_argv(argv).argv + + +def _field_entry(field: str, span: AppliedSpan) -> dict: + return { + "field": field, + "start": span.start, + "end": span.end, + "detector_id": span.detector_id, + "withheld": span.withheld, + } + + +def redact_field(field: str, value: str, source: Optional[FindingSource] = None): + """Redact ONE named exported field, and return what was found doing it. + + This is ``redact_argv``'s discipline (redact-or-withhold, never mangle; + declare the projection) for the exported claim fields that are not argv — + the label, the pathspecs, the changed paths. It returns the findings as + well as the text on purpose: those findings ARE the export-gate input for + this field, so the field cannot be blocked-but-published or + published-but-unscanned. Both of those were real. + """ + result = redact(value, source) + try: + faithful = reconstruct(value, result) == value + except ValueError: + faithful = False + if faithful: + return FieldRedaction( + text=result.text, + findings=result.findings, + applied=[_field_entry(field, s) for s in result.applied], + ) + kind = result.applied[0].kind if result.applied else "unknown" + return FieldRedaction( + text=_withheld_mask(kind), + findings=result.findings, + applied=[ + _field_entry( + field, + AppliedSpan( + start=0, + end=len(value), + kind=kind, + detector_id=kind, + withheld=True, + ), + ) + ], + ) + + +# How many findings a refusal enumerates before it collapses into a count. A +# refusal that prints 19,900 lines is as unusable as one that prints none. +ENUMERATION_CAP = 20 + + +def describe_finding(f: Finding) -> str: + """One located line for a finding. Carries the masked preview, never the token.""" + src = f.source + if src is None: + where = "unlocated" + offset = "-" + else: + where = src.kind + if src.claim_index is not None: + where += f" of claim {src.claim_index}" + if src.event_index is not None: + where += f" of event {src.event_index}" + if src.argv_index is not None: + where += f" arg {src.argv_index}" + offset = str(src.byte_offset) + return ( + f"{f.detector_id or f.kind} [{f.tier}] at {where} " + f"line {'-' if f.line is None else f.line} offset {offset}: " + f"{f.preview} fingerprint {f.fingerprint or '-'}" + ) + + +def enumerate_findings(findings, cap: int = ENUMERATION_CAP) -> list[str]: + """Located lines for ``findings``, capped, with the remainder as a count.""" + lines = [f" - {describe_finding(f)}" for f in findings[:cap]] + remainder = len(findings) - cap + if remainder > 0: + lines.append(f" ... and {remainder} more finding(s) not listed") + return lines class SecretsBlocked(Exception): - """Raised when export is blocked because secrets were found.""" + """Raised when export is blocked because secrets were found. + + The message ENUMERATES: a finding an operator cannot locate is a finding + they can only respond to with --allow-secrets, which is the failure mode + that gets a blocking gate switched off. + """ - def __init__(self, findings: list[Finding]) -> None: + def __init__(self, findings: list[Finding], cap: int = ENUMERATION_CAP) -> None: self.findings = findings kinds = ", ".join(sorted({f.kind for f in findings})) - super().__init__( - f"export blocked: {len(findings)} likely secret(s) found ({kinds}). " + lines = [ + f"export blocked: {len(findings)} likely secret(s) found in the " + f"exported artifact ({kinds}).", + ] + lines.extend(enumerate_findings(findings, cap)) + lines.append( "Re-run with --allow-secrets to export a redacted artifact anyway " "(the override is logged into the manifest)." ) + super().__init__("\n".join(lines)) diff --git a/src/didrun/render.py b/src/didrun/render.py index 564b980..38d84c6 100644 --- a/src/didrun/render.py +++ b/src/didrun/render.py @@ -20,20 +20,56 @@ import os from typing import Optional +from .claims import ENV_DRIFT, ENV_INCOMPARABLE, ENV_MATCH, ENV_NOT_RECORDED +from .manifest import ( + AUTHORITY_GRADE, + AUTHORITY_LIMITATION, + AUTHORITY_PRINCIPAL_BASIS, +) + # Grade -> (text token, ANSI color, severity marker for NO_COLOR, honest gloss). # The token NEVER overclaims: the strongest positive says TREE-EXACT ("the # evidence tree equals the sealed tree"), never "VERIFIED"/"PROVEN" — didrun # records, it does not prove. The marker carries severity when color is absent. +# +# EVERY grade must be registered in BOTH tables. Both are read through +# `.get(…, UNKNOWN)`, so an unregistered grade does not raise — it renders as +# UNKNOWN, which is a silent lie about what the verdict actually said. +# +# `chain-broken` is the one row here that is not a claim grade: no GradeResult +# ever carries it, so no table row ever displays it. It is a whole-report +# verdict — `VerifyReport.worst_status` returns it when the live ledger's chain +# does not recompute — and `render_verdict` looks the VERDICT up in this same +# table, so without a row the worst thing didrun can say about a ledger would +# print as UNKNOWN. _GRADE_DISPLAY = { + "chain-broken": ("CHAIN-BROKEN", "31", "x", "the ledger chain does not recompute"), "failed": ("FAILED", "31", "x", "recorded command exited non-zero"), "unknown": ("UNKNOWN", "33", "?", "no honest binding"), + "witness-unavailable": ( + "WITNESS-UNAVAIL", + "33", + "?", + "sealed grade shown; live evidence not found", + ), "stale": ("STALE", "33", "!", "tree moved since evidence"), "scope-exact": ("SCOPE-EXACT", "36", "~", "change within declared scope"), "tree-exact": ("TREE-EXACT", "32", "=", "recorded against the sealed tree"), } # Sort order: worst first. A reviewer sees problems before recorded-exact rows. -_SORT_RANK = {"failed": 0, "unknown": 1, "stale": 2, "scope-exact": 3, "tree-exact": 4} +# `chain-broken` sorts above `failed` for completeness only — it is a report-level +# verdict, never a per-claim grade, so nothing is ever sorted by this rank. Do not +# go looking for the rows; there are none. +_SORT_RANK = { + "chain-broken": -1, + "failed": 0, + "unknown": 1, + "witness-unavailable": 2, + "stale": 3, + "scope-exact": 4, + "tree-exact": 5, +} def _use_color() -> bool: @@ -66,32 +102,260 @@ def _sanitize(s: str) -> str: return "".join(out) +def sanitize_text(value) -> str: + """`_sanitize` for the command surfaces outside this module. + + `didrun seal` and `didrun authorize` print an authority's label and principal + straight to the terminal, and neither value is computed by this process: + `authority.jsonl` is a local, unhashed, gitignored file that gets the same + validate-rather-than-trust reading as `claims.jsonl`, so whoever can write it + chooses those strings. A newline in a label fabricates a verdict row in the + seal's own output, the same way it did in the verify block. + """ + return _sanitize(value if isinstance(value, str) else str(value)) + + +def _skipped_notes(n: int) -> str: + """Notes under the didrun ref that could not be parsed at all. + + Reported rather than swallowed: "no manifest" and "there were notes here + and none of them could be read" are different facts. + """ + return f"{n} note{'' if n == 1 else 's'} skipped (unparseable)" + + +def env_summary_text(report) -> str: + """One line for how the sealed environment fingerprints compared. + + The three counters are always printed, zeros included: "0 drifted" is a + measurement and a missing counter is not. `not-recorded` is appended only + when it is nonzero — it is the shape of every note published before the + fingerprint was bound, and printing "0 not-recorded" on every modern verify + would be noise about a case that no longer occurs. + + Whether drift REFUSES is said here rather than left implied. The same three + numbers mean different things under --require-env-match, and a reader who + cannot tell which mode produced them cannot act on either. + """ + counts = report.env_counts + line = ( + f"env: {counts[ENV_MATCH]} match / {counts[ENV_DRIFT]} drifted / " + f"{counts[ENV_INCOMPARABLE]} incomparable" + ) + if counts[ENV_NOT_RECORDED]: + line += f" / {counts[ENV_NOT_RECORDED]} not-recorded" + if report.require_env_match: + line += " (drift refuses: --require-env-match)" + elif counts[ENV_DRIFT]: + line += " (advisory; --require-env-match makes it refuse)" + return line + + +def chain_banner_text(report) -> str: + """The chain sentence a reviewer must not miss, or "" when there is none. + + Printed for `broken` and `unverifiable` only. `absent` and `empty` are NOT + chain faults — a ledger that is elsewhere, or has recorded nothing yet, is a + witness-availability fact that the per-claim rows already state as + `witness-unavailable`. Rendering a missing ledger as tamper would make every + verify against an archived ledger read like an attack. + + `broken` and `unverifiable` get different words on purpose: broken means an + entry did not recompute to its stored hash, unverifiable means this binary + could not recompute it at all. Only the first is evidence of mutation. + """ + status = report.chain_status + if status == "broken": + where = "" if report.chain_broken_index is None else f" at index {report.chain_broken_index}" + return f"ledger chain BROKEN{where} — every grade below is unreliable" + if status == "unverifiable": + reason = _sanitize(report.chain_reason) or "this binary cannot recompute it" + return f"ledger chain unverifiable — {reason}; every grade below is unreliable" + return "" + + +def render_chain_banner(report) -> str: + """The terminal form of the chain sentence: marker, colour, "" when clean.""" + text = chain_banner_text(report) + return _c(f"x {text}", "31") if text else "" + + +def superseded_text(report) -> str: + """One line for the claims a later claim in the same window retired, or "". + + Appended only when it is nonzero, on the `not-recorded` precedent: zero + superseded claims is the shape of nearly every window, and a counter on + every verify would be boilerplate about a case that did not occur. + + What it says is the honest bound of the mechanism. Retrying a gate until it + passes is not prevented — it is RETAINED and counted here, which is strictly + better than the invisible truncation it replaces. + """ + count = report.superseded_count + if not count: + return "" + return ( + f"{count} claim{'' if count == 1 else 's'} superseded within this seal " + f"window (retained in the record, excluded from the verdict)" + ) + + +# The token for a bound authority. Deliberately NOT a row in `_GRADE_DISPLAY` or +# `_SORT_RANK`: `cited` is an AUTHORITY's grade and never a claim's. No +# GradeResult carries it, `worst_status` never returns it, and a `_SORT_RANK` row +# would imply it sorts among the claim grades, which it never does. It is held to +# the same rule as every token in that table — reported verbatim, never upgraded: +# `CITED` must never be rendered as "signed", "approved" or "authorized by". +_AUTHORITY_TOKEN = AUTHORITY_GRADE.upper() + + +def authority_lines(report, indent: str = " ") -> list: + """The block for each authority this note cited, or [] when it cited none. + + The limitation sentence comes from the SOURCE CONSTANT, never from the note. + A note is data this process did not compute, so a limitation a note could + edit would be no limitation at all — and this is the one string standing + between `CITED` and being read as approval. + + An ordinary seal cites nothing, so this returns [] and the verdict gains no + line. That is not a rendering choice: an authority attaches only to an + exceptional transition (docs/TRUST_MODEL.md). + """ + lines: list = [] + for entry in report.authority: + + def field(name: str, _entry=entry) -> str: + """Every value this block reads out of the note, sanitized at the + single point of access. + + Not `_sanitize` spelled out at each use site: that is how `size` and + `ttl` ended up as the two of six fields nobody wrapped. An + unsanitized value interpolated into these lines can carry newlines, + and a newline here fabricates whole verdict rows ABOVE the real + table — rows the renderer never computed, in the vocabulary this + module exists to keep honest. A note travels over `git fetch`, so it + is data from another machine, and `_sanitize` is not optional for + any of it. The HTML path already funnels every field through one + `esc`; this is the terminal's equivalent. + """ + value = _entry.get(name, "") + return _sanitize(value if isinstance(value, str) else str(value)) + + scope = field("scope") + label = field("label") + path = field("artifact_path") + # The full digest, not a 12-character prefix like the commit and tree + # above it. Comparing this value against a file is the one thing the + # feature buys, and printing a prefix would invite a reviewer to compare + # a prefix. + digest = field("artifact_sha256") + size = field("artifact_bytes") + principal = field("principal") or "(unnamed)" + # From the SOURCE CONSTANT, like the token and the limitation below, and + # for the same reason: this field says how the principal's name was + # established, so a note that supplied it could print "principal alice + # (verified-ssh-signature)" directly above a sentence saying didrun + # cannot tell who wrote it. "self-asserted" is the only basis this + # version can establish, so it is the only one it will print. + basis = _sanitize(AUTHORITY_PRINCIPAL_BASIS) + ttl = field("expires_after_seals") + lines.append( + _c(f"{indent}! {_AUTHORITY_TOKEN} authority for scope {scope}", "33") + + f": {label}" + ) + lines.append(f"{indent} artifact {path} sha256 {digest} {size} bytes") + lines.append( + f"{indent} principal {principal} ({basis}) · " + f"expires after {ttl} seal(s)" + ) + lines.append(f"{indent} {AUTHORITY_LIMITATION}") + return lines + + +def _conjunct_token(grade) -> str: + """The token for one conjunct of a conjunction — never a grade for a name + that resolved to nothing. + + ``None`` means the label matched no single live claim in the seal window. + That is not a grade, and rendering it through ``_GRADE_DISPLAY`` would print + it as UNKNOWN — a real grade, and a claim the note never made. The + conjunction itself already grades unknown in that case; this row says which + member is the reason. + """ + if grade is None: + return "UNRESOLVED" + return _GRADE_DISPLAY.get(grade, ("UNKNOWN", "33", "?", ""))[0] + + +def _row_detail(r, reason: str) -> str: + """The detail cell text, prefixed when the row is record rather than verdict. + + Without this a retired FAILED row still reads FAILED under an ALL + RECORDED-EXACT headline, which looks like the tool contradicting itself. The + grade and reason are NOT rewritten — that is the whole point of keeping the + entry — so the marker goes in front of them. + """ + if not r.is_superseded: + return reason + return f"superseded by claim #{r.superseded_by} (record, not verdict) — {reason}" + + def render_verdict(report, width: int = 80) -> str: """Render a VerifyReport as the CLI instrument panel.""" lines: list[str] = [] - results = sorted(report.results, key=lambda r: _SORT_RANK.get(r.grade, 9)) + # Retired rows sort BELOW every live one, whatever they graded: they are the + # record of how the unit got here, and the live claims are what a reviewer + # is being asked about. + results = sorted( + report.results, key=lambda r: (r.is_superseded, _SORT_RANK.get(r.grade, 9)) + ) + banner = render_chain_banner(report) worst = report.worst_status if report.results else "empty" if worst == "empty": lines.append(_c("○ NO CLAIMS", "33") + " nothing sealed for this commit") + # A broken chain is worth saying even with nothing sealed here: the + # ledger is still the thing every other command reads. + if banner: + lines.append(banner) lines.append(f" commit {report.commit[:12] or '(none)'} · resolved-by {report.resolved_by}") + if report.notes_skipped: + lines.append(_c(f" ! {_skipped_notes(report.notes_skipped)}", "33")) return "\n".join(lines) token, color, marker, _gloss = _GRADE_DISPLAY.get(worst, ("UNKNOWN", "33", "?", "")) - verified = sum(1 for r in report.results if r.grade in ("tree-exact", "scope-exact")) - total = len(report.results) + # Counted over the LIVE set, which is the set the verdict was computed over. + # Counting retired claims here would print "1/2 recorded-exact" under an ALL + # RECORDED-EXACT headline; the retained count gets its own line instead. + live = report.live_results + verified = sum(1 for r in live if r.grade in ("tree-exact", "scope-exact")) + total = len(live) # Verdict header — the one line a reviewer reads first. The marker prefix # carries severity even with color stripped. headline = "ALL RECORDED-EXACT" if report.all_verified else f"{token} — review needed" lines.append(_c(f"{marker} {headline}", color)) + if banner: + lines.append(banner) lines.append( f" {verified}/{total} claims recorded-exact · " f"commit {report.commit[:12]} · tree {report.tree[:12]} · " f"resolved-by {report.resolved_by}" ) + # How many of those verdicts were checked against the recorded entry the + # seal named, rather than regraded against whatever ledger is on disk now. + lines.append(f" {report.evidence_bound_count}/{total} claims evidence-bound") + superseded = superseded_text(report) + if superseded: + lines.append(f" {superseded}") + lines.append(f" {env_summary_text(report)}") if report.secrets_override: lines.append(_c(" ! sealed with --allow-secrets (redacted export)", "33")) + # Directly under the override line, because the override is what an authority + # is most often cited for and a reader must see the two together. + lines.extend(authority_lines(report)) + if report.notes_skipped: + lines.append(_c(f" ! {_skipped_notes(report.notes_skipped)}", "33")) lines.append("") # Per-claim table. Failing/stale first (already sorted). @@ -100,10 +364,26 @@ def render_verdict(report, width: int = 80) -> str: for r in results: token, color, marker, gloss = _GRADE_DISPLAY.get(r.grade, ("UNKNOWN", "33", "?", "")) label = _sanitize(r.claim.label)[:26] - detail = _sanitize(r.reason) + detail = _row_detail(r, _sanitize(r.reason)) if r.exit_code is not None: detail = f"exit {r.exit_code} {detail}" lines.append(f" {marker} {_c(f'{token:<12}', color)} {label:<26} {detail}") + # The drifted rows say so individually. `incomparable` does not get a + # row of its own: it is the state of every note published before the + # fingerprint was bound, so a per-claim line would be a paragraph of + # boilerplate under an archive — the header counter already reports it. + if r.env_status == ENV_DRIFT: + lines.append( + f" {'':<12} {'':<26} {ENV_DRIFT}: {_sanitize(r.env_reason)}" + ) + # A conjunction's members, each with what it earned. Uncapped on + # purpose: the row's grade IS one of these, and hiding the tail would + # hide the conjunct the verdict came from. + for label, cgrade in r.conjunct_grades: + lines.append( + f" {'':<12} {'':<26} of {_sanitize(label)}: " + f"{_conjunct_token(cgrade)}" + ) # Drill-down: show the delta for stale/scope, capped. for change in r.delta[:5]: lines.append(f" {'':<12} {'':<26} {change.status} {_sanitize(change.path)}") @@ -124,9 +404,12 @@ def render_html(report, title: str = "didrun evidence report") -> str: CSP; 100% of dynamic content HTML-escaped; failing/stale first; responsive; print-clean. This is the shareable evidence surface. """ - results = sorted(report.results, key=lambda r: _SORT_RANK.get(r.grade, 9)) - verified = sum(1 for r in report.results if r.grade in ("tree-exact", "scope-exact")) - total = len(report.results) + results = sorted( + report.results, key=lambda r: (r.is_superseded, _SORT_RANK.get(r.grade, 9)) + ) + live = report.live_results + verified = sum(1 for r in live if r.grade in ("tree-exact", "scope-exact")) + total = len(live) worst = report.worst_status if report.results else "empty" all_ok = report.all_verified @@ -137,9 +420,24 @@ def esc(s: str) -> str: for r in results: token, _color, _marker, _gloss = _GRADE_DISPLAY.get(r.grade, ("UNKNOWN", "", "?", "")) grade_class = r.grade.replace("-", "_") - detail = esc(r.reason) + if r.is_superseded: + # A second class, not a replaced one: the row keeps the grade it + # earned (that is the record) and the tone says it does not vote. + grade_class += " superseded" + detail = _row_detail(r, esc(r.reason)) exit_txt = f"exit {r.exit_code}" if r.exit_code is not None else "—" argv = " ".join(r.claim.get("argv_preview", [])) if isinstance(r.claim, dict) else "" + # The conjunct list rides in the detail column, above the delta: a + # conjunction's grade is one of these, so it is the first thing a reader + # of that row needs. Same list markup as the delta — a conjunction + # carries an existing grade, so no new badge or token is introduced. + conjunct_html = "" + if r.conjunct_grades: + items = "".join( + f"
  • {esc(_conjunct_token(g))} {esc(label)}
    • " + for label, g in r.conjunct_grades + ) + conjunct_html = f"
      {items}
    " delta_html = "" if r.delta: items = "".join( @@ -149,17 +447,60 @@ def esc(s: str) -> str: more = f"
  • … {len(r.delta) - 20} more
    • " if len(r.delta) > 20 else "" delta_html = f"
      {items}{more}
    " claim_label = r.claim.label if not isinstance(r.claim, dict) else r.claim.get("label", "") + # Same rule as the CLI table: drifted rows say so, incomparable ones are + # left to the header counter. + env_html = ( + f"{esc(ENV_DRIFT)}: {esc(r.env_reason)}" + if r.env_status == ENV_DRIFT + else "" + ) rows.append( f"" f"{esc(token)}" f"{esc(claim_label)}" f"{detail}" - f"{esc(exit_txt)}{delta_html}" + f"{esc(exit_txt)}{env_html}" + f"{conjunct_html}{delta_html}" f"" ) + # The chain sentence gets its own block above the table, not a table row: + # it is a statement about the ledger every row was read out of, so a row + # would file it alongside the very grades it invalidates. + chain_text = chain_banner_text(report) + chain_html = ( + f'
    {esc(chain_text)}
    ' if chain_text else "" + ) + superseded = superseded_text(report) + superseded_html = f"{esc(superseded)} ·" if superseded else "" + + # One block per cited authority, above the table for the same reason as the + # chain sentence: it is a statement about the seal, not about a claim, so a + # table row would file it alongside the grades. Every field is escaped, and + # the limitation, the token and the principal's basis come from source + # constants, never from the note. + authority_html = "".join( + "
    " + f"{esc(_AUTHORITY_TOKEN)} " + f"authority for scope {esc(a.get('scope', ''))}: {esc(a.get('label', ''))}" + f"artifact {esc(a.get('artifact_path', ''))} · " + f"sha256 {esc(a.get('artifact_sha256', ''))} · " + f"{esc(a.get('artifact_bytes', ''))} bytes · " + f"principal {esc(a.get('principal', '') or '(unnamed)')} " + f"({esc(AUTHORITY_PRINCIPAL_BASIS)}) · " + f"expires after {esc(a.get('expires_after_seals', ''))} seal(s)" + f"{esc(AUTHORITY_LIMITATION)}" + "
    " + for a in report.authority + ) + if all_ok: verdict_class, verdict_text = "ok", "ALL RECORDED-EXACT" + elif worst == "chain-broken": + verdict_class = "failed" + verdict_text = ( + f"{esc(_GRADE_DISPLAY['chain-broken'][0])} — the ledger chain does not recompute" + ) elif worst == "failed": verdict_class = "failed" verdict_text = f"{esc(_GRADE_DISPLAY['failed'][0])} — a claimed command was recorded failing" @@ -212,7 +553,20 @@ def esc(s: str) -> str: .verdict.ok {{ color:var(--ok); }} .verdict.attention {{ color:var(--warn); }} .verdict.failed {{ color:var(--bad); }} + /* Filled, not tinted: a chain fault invalidates every row under it, so it is + the one block that must survive a fast scan and a monochrome printout. */ + .chainbanner {{ margin:8px 0; padding:8px 10px; border-radius:3px; + background:var(--bad-fill); color:var(--on-fill); font-weight:700; }} .meta {{ color:var(--dim); font-size:12px; word-break:break-all; }} + /* An authority is record, not verdict, so it is toned like the advisory + states — never like a pass. The limitation is not small print: it is the + same size as the citation it qualifies. */ + .authority {{ margin:8px 0; padding:8px 10px; border-radius:3px; + border:1px solid var(--warn); color:var(--ink); font-size:12px; }} + .badge.cited {{ color:var(--warn); }} + .authority .ameta {{ display:block; margin-top:4px; color:var(--dim); + word-break:break-all; }} + .authority .alimit {{ display:block; margin-top:4px; color:var(--warn); }} table {{ width:100%; border-collapse:collapse; margin-top:8px; table-layout:fixed; }} th {{ text-align:left; color:var(--dim); font-weight:500; font-size:11px; text-transform:uppercase; letter-spacing:.05em; padding:8px; border-bottom:1px solid var(--line); }} @@ -226,19 +580,29 @@ def esc(s: str) -> str: font-weight:700; letter-spacing:.03em; border:1px solid currentColor; white-space:nowrap; }} .badge.tree_exact {{ color:var(--ok); }} .badge.scope_exact {{ color:var(--scope); }} - .badge.stale, .badge.unknown {{ color:var(--warn); }} + .badge.stale, .badge.unknown, .badge.witness_unavailable {{ color:var(--warn); }} /* Failure and stale are the MOST-designed states: solid-filled badges + a left accent bar so a problem row is unmistakable on a fast scan. */ .badge.failed {{ color:var(--on-fill); background:var(--bad-fill); border-color:var(--bad-fill); }} .badge.stale {{ color:var(--on-fill); background:var(--warn-fill); border-color:var(--warn-fill); }} - tr.stale td, tr.unknown td {{ background:color-mix(in srgb, var(--warn) 10%, transparent); }} + tr.stale td, tr.unknown td, tr.witness_unavailable td {{ + background:color-mix(in srgb, var(--warn) 10%, transparent); }} tr.failed td {{ background:color-mix(in srgb, var(--bad) 12%, transparent); }} + /* A retired row is record, not verdict: toned down and stripped of its accent + bar so it cannot be read as a live problem. Dimmed, never hidden — the + whole point of supersession is that the earlier attempt stays visible. */ + tr.superseded td {{ opacity:.62; }} + tr.superseded td.status {{ box-shadow:none; }} tr.failed td.status {{ box-shadow:inset 3px 0 0 var(--bad); }} tr.stale td.status {{ box-shadow:inset 3px 0 0 var(--warn); }} tr.unknown td.status {{ box-shadow:inset 3px 0 0 var(--warn); }} + tr.witness_unavailable td.status {{ box-shadow:inset 3px 0 0 var(--warn); }} .detail {{ color:var(--dim); }} .detail .reason {{ display:block; }} .exit {{ display:inline-block; margin-top:2px; color:var(--mono); font-size:12px; }} + /* Advisory by default, so it is toned like the warn states rather than the + failure ones — drift does not make a recorded command un-run. */ + .envdrift {{ display:block; margin-top:4px; color:var(--warn); font-size:12px; }} .delta {{ margin:6px 0 0; padding-left:16px; color:var(--dim); font-size:12px; }} .delta li {{ overflow-wrap:anywhere; }} .delta .ch {{ display:inline-block; width:14px; color:var(--mono); font-weight:700; }} @@ -250,7 +614,11 @@ def esc(s: str) -> str: .badge {{ padding:2px 5px; font-size:10px; }} }} @media print {{ body {{ background:#fff; color:#000; }} .wrap {{ max-width:none; }} - tr.failed td, tr.stale td, tr.unknown td {{ background:transparent; }} }} + tr.failed td, tr.stale td, tr.unknown td, tr.witness_unavailable td {{ + background:transparent; }} + /* Browsers drop backgrounds when printing, which would leave the chain + sentence white-on-white — the one line that must never disappear. */ + .chainbanner {{ background:transparent; color:#000; border:2px solid #000; }} }} @@ -258,7 +626,12 @@ def esc(s: str) -> str:
    didrun · flight record
    {verdict_text}
    + {chain_html} + {authority_html}
    {esc(verified)}/{esc(total)} claims recorded-exact · + {esc(report.evidence_bound_count)}/{esc(total)} claims evidence-bound · + {superseded_html} + {esc(env_summary_text(report))} · commit {esc(report.commit[:12])} · tree {esc(report.tree[:12])} · resolved-by {esc(report.resolved_by)} · coverage {coverage_html}
    diff --git a/tests/compat/__init__.py b/tests/compat/__init__.py new file mode 100644 index 0000000..5ed974b --- /dev/null +++ b/tests/compat/__init__.py @@ -0,0 +1,15 @@ +"""Compat corpus replay harness — CONTENT-BLIND by construction. + +THE CONTENT RULE (non-negotiable, and the reason this package exists at all): +this harness may READ ledger bytes; it may never print, log, echo, or assert on +them. Every assertion in here compares digests, booleans, counts, or path +shapes. ``assert computed_hash == stored_hash`` is fine — a hash is not +content. ``assert line == ...`` is forbidden, because pytest prints both sides +of a failed comparison, and a ledger line is a verbatim record of everything a +session ran and printed: argv, stdout, stderr, and any credential that passed +through them. + +The tree this harness is aimed at is 174 MB of secret-bearing evidence for a +build that is still running. A harness that echoes one line of it into a CI log +or a chat transcript has done more damage than any bug it could have found. +""" diff --git a/tests/compat/conftest.py b/tests/compat/conftest.py new file mode 100644 index 0000000..fe4e7bc --- /dev/null +++ b/tests/compat/conftest.py @@ -0,0 +1,154 @@ +"""Fixtures and guardrails for the compat replay harness. + +The corpus leg is OFF unless ``DIDRUN_COMPAT_CORPUS`` points at a copy of an +archive tree, so CI is green by construction and no unattended run can ever +touch a live ledger. +""" + +from __future__ import annotations + +import hashlib +import os +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +import pytest + +from . import synthetic + +ENV_VAR = "DIDRUN_COMPAT_CORPUS" + +SKIP_UNSET = ( + "set DIDRUN_COMPAT_CORPUS to a COPY of an archive tree to run the corpus replay" +) +SKIP_MISSING = ( + "DIDRUN_COMPAT_CORPUS does not point at a directory; " + "point it at a COPY of an archive tree to run the corpus replay" +) +SKIP_LIVE = ( + "DIDRUN_COMPAT_CORPUS points at a live repo (it has both a .didrun ledger " + "and a .git dir); this harness must never read a ledger something else is " + "writing. Copy the tree WITHOUT its live .didrun directory (the archives " + "under .didrun-history and the git dir are what the legs need) and point " + "the variable at the copy" +) +SKIP_REFUSED = ( + "DIDRUN_COMPAT_CORPUS points at (or inside) a tree this harness refuses by " + "name: it is live, secret-bearing evidence for a running build. Make a copy " + "on a separate volume and point the variable at the copy" +) + +# The refused trees, stored as sha256 digests of their absolute paths rather +# than as literals. Two rules meet here and a digest is the only thing that +# satisfies both: the pack requires these exact paths refused BY NAME, and +# RULES 17 forbids any private absolute path, repo name, or home directory in a +# shipped file. A one-way digest names nothing and still matches exactly. +# A heuristic is not a guardrail when the thing it guards is a running build's +# evidence, so this sits in front of the live-repo heuristic, not instead of it. +_REFUSED_PATH_DIGESTS = frozenset( + { + "44b1bd2169fdc4af5ac880330a6917f49456f19ad63efeb79326655e5cd3b4d2", + "08a945e5d11024bbc32f34f1f5dbf0b57c3590c08177d0391c0e0172a777acf1", + } +) + + +def path_digest(path) -> str: + """sha256 of an absolute path string, trailing separator normalized away.""" + s = str(path).rstrip("/") + return hashlib.sha256(s.encode("utf-8")).hexdigest() + + +def is_refused(path, digests=_REFUSED_PATH_DIGESTS) -> bool: + """True if ``path``, its realpath, or any ancestor of either is refused.""" + for candidate in {str(path), os.path.realpath(str(path))}: + p = Path(candidate) + for ancestor in (p, *p.parents): + if path_digest(ancestor) in digests: + return True + return False + + +def resolve_corpus_root(environ, digests=_REFUSED_PATH_DIGESTS) -> tuple: + """Return ``(root, skip_reason)``. Exactly one of the two is None. + + Pure over its ``environ`` and ``digests`` arguments so both the skip + behaviour and the refusal branch are testable without depending on what the + ambient environment holds or on knowing a refused path. + """ + raw = environ.get(ENV_VAR, "").strip() + if not raw: + return None, SKIP_UNSET + root = Path(raw) + if is_refused(root, digests): + return None, SKIP_REFUSED + if not root.is_dir(): + return None, SKIP_MISSING + if (root / ".didrun").exists() and (root / ".git").exists(): + return None, SKIP_LIVE + return root, None + + +@dataclass(frozen=True) +class ReplaySource: + """One tree the legs run over. ``expected`` is set only for synthetic.""" + + name: str + root: Path + expected: Optional[synthetic.Expectations] = None + + +@pytest.fixture(scope="session") +def synthetic_corpus(tmp_path_factory) -> synthetic.SyntheticCorpus: + return synthetic.build_corpus(tmp_path_factory.mktemp("synthetic-corpus")) + + +@pytest.fixture(scope="session") +def known_bad_corpus(tmp_path_factory) -> Path: + return synthetic.build_known_bad(tmp_path_factory.mktemp("known-bad-corpus")) + + +@pytest.fixture(scope="session") +def corpus_root() -> Path: + root, reason = resolve_corpus_root(os.environ) + if root is None: + pytest.skip(reason) + return root + + +@pytest.fixture(params=["synthetic", "corpus"]) +def replay_source(request) -> ReplaySource: + if request.param == "synthetic": + fixture = request.getfixturevalue("synthetic_corpus") + return ReplaySource("synthetic", fixture.root, fixture.expected) + return ReplaySource("corpus", request.getfixturevalue("corpus_root")) + + +def has_git_repo(root: Path) -> bool: + proc = subprocess.run( + ["git", "-C", str(root), "rev-parse", "--git-dir"], capture_output=True + ) + return proc.returncode == 0 + + +# --- the run summary --------------------------------------------------------- +# Counts only. Every line put in here has been through _assert_no_content first. + +_SUMMARY: list = [] + + +def record_summary(line: str) -> None: + _SUMMARY.append(line) + + +def pytest_terminal_summary(terminalreporter, exitstatus, config) -> None: + tr = terminalreporter + tr.write_sep("-", "compat replay") + root, _reason = resolve_corpus_root(os.environ) + if root is None: + # In these exact words: a green synthetic run is not corpus validation. + tr.write_line("corpus: SKIPPED (unverified)") + for line in _SUMMARY: + tr.write_line(line) diff --git a/tests/compat/synthetic.py b/tests/compat/synthetic.py new file mode 100644 index 0000000..a1dca91 --- /dev/null +++ b/tests/compat/synthetic.py @@ -0,0 +1,483 @@ +"""A deterministic, archive-shaped synthetic corpus. + +The real corpus cannot ship: it is secret-bearing, it is 174 MB, and it belongs +to a running build. So CI gets this instead — a tree with the same *shapes* the +real archives have, generated from literals with no randomness and no wall +clock, carrying one instance of every outcome the replay legs are supposed to +detect. + +Nothing in here is real. Paths, digests, argv and labels are invented; the only +product code used is the code under test (``Event``, ``ChainEntry``, +``Manifest``), so the fixture's shape follows the schema instead of drifting +from it. +""" + +from __future__ import annotations + +import json +import os +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +from didrun.claims import Claim, GradeResult +from didrun.ledger import ( + GENESIS_HASH, + ChainEntry, + Event, + canonical_json, + sha256_hex, +) +from didrun.manifest import MANIFEST_VERSION, Manifest + +# Fixed identities and timestamps: the fixture must be byte-identical on every +# machine and every run, so nothing here may read the clock or the environment. +_GIT_ENV = { + "GIT_AUTHOR_NAME": "compat fixture", + "GIT_AUTHOR_EMAIL": "fixture@invalid", + "GIT_COMMITTER_NAME": "compat fixture", + "GIT_COMMITTER_EMAIL": "fixture@invalid", + "GIT_AUTHOR_DATE": "2020-01-01T00:00:00+00:00", + "GIT_COMMITTER_DATE": "2020-01-01T00:00:00+00:00", +} + +_SYNTHETIC_CWD = "/w/repo" +_SYNTHETIC_FINGERPRINT = "v1:0123456789abcdef" +_SYNTHETIC_TREE = "3" * 40 + + +@dataclass(frozen=True) +class Expectations: + """What the legs must report for this fixture. Declared, never derived.""" + + # Ordered by sorted ledger-directory path, which is the order the harness + # walks. See build_corpus for which unit is which. + log_statuses: tuple + log_entries: tuple + objects: dict + notes: int + claim_shape: dict + + +@dataclass(frozen=True) +class SyntheticCorpus: + root: Path + expected: Expectations + + +def _git(root: Path, *args: str, stdin: Optional[bytes] = None) -> bytes: + """Run git in ``root``. Never echoes git's output — it can carry paths.""" + proc = subprocess.run( + ["git", "-C", str(root), *args], + input=stdin, + capture_output=True, + env={**os.environ, **_GIT_ENV}, + ) + if proc.returncode != 0: + raise RuntimeError(f"synthetic fixture: git {args[0]} failed") + return proc.stdout + + +def _init_repo(root: Path) -> None: + root.mkdir(parents=True, exist_ok=True) + _git(root, "init", "-q") + _git(root, "config", "user.email", "fixture@invalid") + _git(root, "config", "user.name", "compat fixture") + _git(root, "config", "commit.gpgsign", "false") + # Git's background maintenance can repack while a test walks objects. + _git(root, "config", "gc.auto", "0") + _git(root, "config", "maintenance.auto", "false") + + +def _commit(root: Path, name: str, body: str) -> str: + (root / name).write_text(body, encoding="ascii") + _git(root, "add", name) + _git(root, "commit", "-qm", f"fixture {name}") + return _git(root, "rev-parse", "HEAD").decode("ascii").strip() + + +def _attach_note(root: Path, commit: str, body: bytes) -> None: + _git( + root, + "notes", + "--ref=refs/notes/didrun", + "add", + "-f", + "-F", + "-", + commit, + stdin=body, + ) + + +# --- the ledger side --------------------------------------------------------- + + +def _event(i: int) -> Event: + return Event( + argv=("pytest", "-q"), + cwd=_SYNTHETIC_CWD, + env_fingerprint=_SYNTHETIC_FINGERPRINT, + observed_via="wrapper", + coverage="complete", + exit_code=0, + started_at=1000000.0 + i, + ended_at=1000001.0 + i, + stdout_blob=sha256_hex(f"synthetic-stdout-{i}".encode("ascii")), + stderr_blob=None, + transcript_blob=None, + tree_before=_SYNTHETIC_TREE, + tree_after=_SYNTHETIC_TREE, + submodule_dirty=False, + ) + + +def _chain(count: int) -> list: + """``count`` well-formed, correctly chained records.""" + records = [] + prev = GENESIS_HASH + for i in range(count): + entry = ChainEntry(index=i, prev_hash=prev, event=_event(i)) + entry.entry_hash = entry.compute_hash() + records.append( + { + "index": i, + "prev_hash": prev, + "entry_hash": entry.entry_hash, + "event": entry.event.to_dict(), + } + ) + prev = entry.entry_hash + return records + + +def _write_log(path: Path, records: list, torn_tail: bool = False) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("wb") as fh: + for rec in records: + fh.write(canonical_json(rec) + b"\n") + if torn_tail: + # A process killed mid-append: a partial line, no newline. This is + # the commonest real-world damage and it must not read as tamper. + fh.write(canonical_json(_chain(1)[0])[:37]) + + +def _write_objects(objects: Path, flat: int, fanout: int) -> None: + """An objects/ dir carrying both namespaces, like 113 of the real archives.""" + objects.mkdir(parents=True, exist_ok=True) + for i in range(flat): + digest = sha256_hex(f"synthetic-blob-{i}".encode("ascii")) + (objects / digest).write_bytes(b"synthetic blob\n") + for i in range(fanout): + loose = sha256_hex(f"synthetic-loose-{i}".encode("ascii"))[:40] + d = objects / loose[:2] + d.mkdir(exist_ok=True) + (d / loose[2:]).write_bytes(b"synthetic loose object\n") + + +# --- the note side ----------------------------------------------------------- + + +def _claim_entry( + ctype: str, + label: str, + grade: str, + supporting: Optional[int], + declared_at: int, + pathspecs: tuple = (), + exit_code: Optional[int] = 0, +) -> dict: + """One graded claim in the shape seal() writes into a note.""" + result = GradeResult( + claim=Claim( + ctype=ctype, + label=label, + event_indices=(supporting,) if supporting is not None else (), + pathspecs=pathspecs, + declared_at_index=declared_at, + ), + grade=grade, + reason="synthetic fixture", + delta=[], + supporting_event_index=supporting, + exit_code=exit_code, + ) + d = result.to_dict() + d["claim"]["argv_preview"] = ["pytest", "-q"] + return d + + +def _conjunction_entry( + label: str, + grade: str, + conjunct_grades: tuple, + declared_at: int, +) -> dict: + """One graded conjunction, in the shape seal() writes into a v3 note. + + ``grade`` is declared, not derived: every value in this fixture is a literal + so the legs measure the instrument against a shape whose every field is + stated. It binds no event and declares no pathspec, so its `argv_preview` is + empty — the same thing seal() writes for a claim with no supporting event. + """ + result = GradeResult( + claim=Claim( + ctype="conjunction", + label=label, + event_indices=(), + declared_at_index=declared_at, + conjuncts=tuple(name for name, _g in conjunct_grades), + ), + grade=grade, + reason="synthetic fixture", + conjunct_grades=conjunct_grades, + ) + d = result.to_dict() + d["claim"]["argv_preview"] = [] + return d + + +def _strip_post_v2_claim_keys(body: dict) -> None: + """Remove the claim keys no note before v3 carried. + + ``Claim.to_dict`` emits ``conjuncts`` for every claim, so a body built from + the current dataclass carries it whatever version number the fixture stamps + on the manifest. A note that says version 1 and holds a v3 claim shape would + make the older legs read as passing on a shape they never see in an archive. + """ + for entry in body.get("claims", []): + entry.get("claim", {}).pop("conjuncts", None) + + +def _coverage() -> dict: + return {"events": 3, "complete": 3, "observed_text_only": 0, "display_only": 0} + + +def build_corpus(root: Path) -> SyntheticCorpus: + """Build the good-path fixture and return it with its declared expectations. + + Ledger directories, in the sorted order the harness walks them: + + 0 unit-01-good-nested nested /.didrun/, 3 entries, intact + 1 unit-02-good-flat flat /, 1 entry, intact + 2 unit-03-torn-tail 2 entries + a partial final line + 3 unit-04-unknown-key an event key this binary does not know + 4 unit-05-unknown-preimage a preimage version this binary does not know + 5 unit-06-empty a zero-byte log + 6 unit-07-broken-hash 3 entries, entry 1's stored hash mutated + """ + root = Path(root) + _init_repo(root) + history = root / ".didrun-history" + + nested = history / "unit-01-good-nested" / ".didrun" + _write_log(nested / "session.log", _chain(3)) + _write_objects(nested / "objects", flat=2, fanout=2) + # Present in 94 / 62 of the real archives; the legs only stat their shape. + (nested / "claims.jsonl").write_bytes(b"") + (nested / "seals.jsonl").write_bytes(b"") + + flat = history / "unit-02-good-flat" + _write_log(flat / "session.log", _chain(1)) + _write_objects(flat / "objects", flat=1, fanout=1) + + _write_log( + history / "unit-03-torn-tail" / ".didrun" / "session.log", + _chain(2), + torn_tail=True, + ) + + unknown_key = _chain(1) + unknown_key[0]["event"]["signal"] = 9 + _write_log(history / "unit-04-unknown-key" / ".didrun" / "session.log", unknown_key) + + unknown_preimage = _chain(1) + unknown_preimage[0]["preimage_version"] = 2 + _write_log( + history / "unit-05-unknown-preimage" / ".didrun" / "session.log", + unknown_preimage, + ) + + empty = history / "unit-06-empty" / ".didrun" / "session.log" + empty.parent.mkdir(parents=True, exist_ok=True) + empty.write_bytes(b"") + + broken = _chain(3) + stored = broken[1]["entry_hash"] + broken[1]["entry_hash"] = ("0" if stored[0] != "0" else "1") + stored[1:] + _write_log(history / "unit-07-broken-hash" / ".didrun" / "session.log", broken) + + # Four notes, one per manifest version this binary must read plus one that + # exercises the additive-key allowlist. The second deliberately omits + # `secrets_override` — the one key from_json already reads with a + # v1-reproducing default — so the allowlist is exercised by the good path and + # not only by a meta-test. The version ladder is covered on purpose: v3 is + # what this build writes, and a leg that only ever reads the version it wrote + # would say nothing about the 65 v1 notes already published. + commit_a = _commit(root, "work-a.txt", "one\n") + manifest_a = Manifest( + version=MANIFEST_VERSION, + commit=commit_a, + tree=_SYNTHETIC_TREE, + claims=[ + _claim_entry("tests-pass", "suite", "tree-exact", 0, 0), + _claim_entry( + "lint-clean", "lint", "scope-exact", 1, 2, pathspecs=("src/",) + ), + _claim_entry("command-succeeded", "build", "stale", 0, 6), + ], + coverage=_coverage(), + secrets_override=False, + ) + _attach_note(root, commit_a, manifest_a.to_json()) + + # A v2 note: the version before the claim vocabulary gained `conjunction`. + # Pinned to the literal 2 rather than MANIFEST_VERSION so this leg keeps + # reading a note one version behind whatever this build writes. + commit_b = _commit(root, "work-b.txt", "two\n") + body_b = json.loads( + Manifest( + version=2, + commit=commit_b, + tree=_SYNTHETIC_TREE, + claims=[_claim_entry("tests-pass", "suite", "failed", 0, 0, exit_code=1)], + coverage=_coverage(), + secrets_override=False, + ).to_json() + ) + del body_b["secrets_override"] + _strip_post_v2_claim_keys(body_b) + _attach_note(root, commit_b, canonical_json(body_b)) + + # A v3 note carrying a conjunction: the claim shape the vocabulary extension + # added. It binds no event and names its members by label, and the note holds + # what each member earned beside the grade the conjunction took from them. + commit_c = _commit(root, "work-c.txt", "three\n") + manifest_c = Manifest( + version=MANIFEST_VERSION, + commit=commit_c, + tree=_SYNTHETIC_TREE, + claims=[ + _claim_entry("tests-pass", "unit", "tree-exact", 0, 0), + _claim_entry("lint-clean", "lint", "tree-exact", 1, 1), + _conjunction_entry( + "release", + "tree-exact", + (("unit", "tree-exact"), ("lint", "tree-exact")), + 2, + ), + ], + coverage=_coverage(), + secrets_override=False, + ) + _attach_note(root, commit_c, manifest_c.to_json()) + + # A v1 note, in the shape the archives actually hold one: none of the keys + # any later version added. Every one of them is on v1's additive allowlist, + # so this is the leg's statement that a note published before v0.2 still + # round-trips under a reader three versions later. + commit_d = _commit(root, "work-d.txt", "four\n") + body_d = json.loads( + Manifest( + version=1, + commit=commit_d, + tree=_SYNTHETIC_TREE, + claims=[_claim_entry("tests-pass", "suite", "tree-exact", 0, 0)], + coverage=_coverage(), + ).to_json() + ) + for added_after_v1 in ( + "secrets_override", + "secrets", + "claims_from", + "claims_to", + "authority", + ): + del body_d[added_after_v1] + _strip_post_v2_claim_keys(body_d) + _attach_note(root, commit_d, canonical_json(body_d)) + + expected = Expectations( + log_statuses=( + "intact", + "intact", + "torn-tail", + "unverifiable@0", + "unverifiable@0", + "empty", + "broken@1", + ), + log_entries=(3, 1, 2, 1, 1, 0, 3), + objects={"dirs": 2, "flat-64hex": 3, "git-fanout": 3}, + notes=4, + claim_shape={ + "notes": 4, + "claims": 8, + # The conjunction binds no event, so it is not a multi-index claim + # and never becomes one: that shape is what the design review ruled + # out, and this counter is the leg that would catch it arriving. + "multi-index": 0, + "max-pathspecs": 1, + "pathspecs": {0: 7, 1: 1}, + "gap-ge-1": 2, + "gap-ge-5": 1, + "grades": {"failed": 1, "scope-exact": 1, "stale": 1, "tree-exact": 5}, + }, + ) + return SyntheticCorpus(root=root, expected=expected) + + +def build_known_bad(root: Path) -> Path: + """A second tree whose *note* and *objects* legs must come back red. + + Kept separate from build_corpus so the good fixture stays green: a leg that + is red for a reason that is actually correct behaviour is a leg nobody will + trust. + """ + root = Path(root) + _init_repo(root) + history = root / ".didrun-history" + + # A third path shape in objects/: neither a flat 64-hex blob nor a git + # fan-out dir. The classifier must fail closed on it rather than ignore it. + residue = history / "unit-bad-objects" / ".didrun" / "objects" + _write_objects(residue, flat=1, fanout=1) + (residue / "pack").mkdir() + (residue / "notes.txt").write_bytes(b"third shape\n") + _write_log(history / "unit-bad-objects" / ".didrun" / "session.log", _chain(1)) + + commit = _commit(root, "work.txt", "bad\n") + # A note from a manifest version this binary must refuse, not coerce. + too_new = json.loads( + Manifest( + version=MANIFEST_VERSION, + commit=commit, + tree=_SYNTHETIC_TREE, + claims=[], + coverage=_coverage(), + ).to_json() + ) + too_new["version"] = MANIFEST_VERSION + 1 + _attach_note(root, commit, canonical_json(too_new)) + return root + + +def multi_index_note_body(commit: str = "a" * 40) -> bytes: + """A note carrying the claim shape the corpus says does not exist. + + 0 of 846 real claims bind more than one event; two of the audit's blocker + findings turned on that. This is the counter-example the leg must catch. + """ + entry = _claim_entry("tests-pass", "suite", "tree-exact", 0, 0) + entry["claim"]["event_indices"] = [0, 1] + two_specs = _claim_entry( + "lint-clean", "lint", "scope-exact", 1, 1, pathspecs=("src/", "tests/") + ) + return Manifest( + version=MANIFEST_VERSION, + commit=commit, + tree=_SYNTHETIC_TREE, + claims=[entry, two_specs], + coverage=_coverage(), + ).to_json() diff --git a/tests/compat/test_corpus_replay.py b/tests/compat/test_corpus_replay.py new file mode 100644 index 0000000..423d2d4 --- /dev/null +++ b/tests/compat/test_corpus_replay.py @@ -0,0 +1,759 @@ +"""The compat corpus replay harness — four legs, run content-blind. + +THE CONTENT RULE (non-negotiable): this module may READ ledger bytes; it may +never print, log, echo, or assert on them. Every assertion below compares +digests, booleans, counts, or path shapes. ``assert computed == stored`` over +two hashes is fine — a hash is not content. ``assert line == ...`` is +forbidden, because pytest prints both sides of a failed comparison, and a +ledger line is a verbatim record of everything a session ran and printed: +argv, stdout, stderr, and whatever credential passed through them. The tree +this harness is aimed at is live, secret-bearing evidence for a build that is +still running; a harness that leaks one line of it into a CI log or a chat +transcript has done more damage than any bug it could have found. + +``_assert_no_content`` is the net under that rule. It is a bound on the SHAPE +of what can escape into an assertion message, not a proof that nothing does — +the only real guarantee is that no leg ever puts a ledger-derived value into a +payload in the first place. + +Why the legs are these four (red-team §3.5): a harness asserting the corpus's +815/30/1 grade histogram would have passed on every defect in the audit. A +histogram is a snapshot, not a regression test. The legs that earn their keep +are the chain recompute — which would have caught a schema break across 5,558 +archived events before it shipped — and the claim-SHAPE invariants, which is +where two of the audit's blocker findings actually lived. + +The corpus leg is env-gated and OFF by default. A green synthetic run is not +corpus validation, and the run summary says so in those words. +""" + +from __future__ import annotations + +import json +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +import pytest + +from didrun.ledger import ( + GENESIS_HASH, + ChainEntry, + Event, + LedgerError, + sha256_hex, + unknown_event_keys_of, +) +from didrun.manifest import MANIFEST_VERSION, NOTES_REF, Manifest + +from . import synthetic +from .conftest import ( + ENV_VAR, + SKIP_LIVE, + SKIP_MISSING, + SKIP_REFUSED, + SKIP_UNSET, + has_git_repo, + is_refused, + path_digest, + record_summary, + resolve_corpus_root, +) + +# --- the content net --------------------------------------------------------- + +# Lowercase, no whitespace, bounded length: the shape of a status token, a key +# name, a grade, a digest, or a path-shape descriptor. Nothing a ledger line or +# a secret looks like survives it. +_STRUCTURAL = re.compile(r"\A[a-z0-9][a-z0-9_.:@#-]{0,63}\Z") + +_NON_STRUCTURAL = "non-structural-token" + + +def _safe_token(value) -> str: + """A source-derived string, reduced to a token or replaced outright. + + Manifest key names and grades come from a note body, which means a foreign + or corrupt note can put anything here. Anything that is not structural is + replaced rather than passed through, so the leg still reports the finding + without carrying the value into the failure message. + """ + if isinstance(value, str) and _STRUCTURAL.match(value): + return value + return _NON_STRUCTURAL + + +def _assert_no_content(obj) -> None: + """Fail if anything non-structural reached an assertion payload. + + The failure message deliberately reports only the TYPE and LENGTH of the + offending value. Echoing it is the exact accident this function exists to + prevent. + """ + stack = [obj] + while stack: + item = stack.pop() + if item is None or isinstance(item, (bool, int, float)): + continue + if isinstance(item, str): + if not _STRUCTURAL.match(item): + raise AssertionError( + f"a non-structural str of length {len(item)} reached an " + "assertion payload; the harness must never put ledger-derived " + "values in an assertion message (its value is withheld here " + "on purpose)" + ) + continue + if isinstance(item, dict): + stack.extend(item.keys()) + stack.extend(item.values()) + continue + if isinstance(item, (list, tuple, set, frozenset)): + stack.extend(item) + continue + raise AssertionError( + f"a {type(item).__name__} reached an assertion payload; only " + "counts, booleans, digests and shape tokens may appear there" + ) + + +def _summarize(leg: str, source: str, payload: dict) -> None: + _assert_no_content(payload) + body = json.dumps(payload, sort_keys=True, separators=(",", ":")) + record_summary(f"{leg} [{source}]: {body}") + + +# --- discovery --------------------------------------------------------------- + + +def ledger_dirs(root) -> list: + """Every directory holding a session.log, in a deterministic order. + + Covers both archive shapes without hardcoding either: nested + ``/.didrun/`` and flat ``/``. + """ + return sorted(p.parent for p in Path(root).rglob("session.log") if p.is_file()) + + +# --- leg 1: chain recompute -------------------------------------------------- + + +@dataclass(frozen=True) +class LogReplay: + """One log's outcome. + + ``label`` is an ordinal, not a path: archive directory names are not + ledger content, but they are not ours to put in a CI log or a transcript + either. The mapping stays recoverable locally without leaking anything — + the ordinal is the position in ``ledger_dirs(root)``. + """ + + label: str + shape: str # "nested" | "flat" + entries: int + status: str # empty | intact | torn-tail | malformed@N | broken@N | unverifiable@N + + +def _read_records(path: Path) -> tuple: + """Parse a log into records. Returns ``(records, torn_tail, malformed_at)``. + + A partial FINAL line is torn-tail — a process killed mid-append, which is + the commonest real damage and is never the same fact as tamper. A partial + line anywhere else is malformed and is reported at its position. + """ + raw = path.read_bytes() + if not raw: + return [], False, None + lines = raw.split(b"\n") + if lines and lines[-1] == b"": + lines.pop() + last = len(lines) - 1 + records: list = [] + torn = False + malformed_at: Optional[int] = None + for i, line in enumerate(lines): + stripped = line.strip() + if not stripped: + continue + try: + records.append(json.loads(stripped)) + except ValueError: + if i == last: + torn = True + elif malformed_at is None: + malformed_at = i + return records, torn, malformed_at + + +def _verify_records(records: list) -> tuple: + """Recompute the chain over parsed records. Returns ``(status, position)``. + + Mirrors Session.verify_chain_detail, but reads instead of constructing a + Session: Session.__init__ mkdirs its root and writes a .gitignore, which + would MODIFY an archive this harness is only ever allowed to read. + + Positions are the harness's own counter, never the stored index, so a + forged index cannot choose the number that gets reported. + """ + prev = GENESIS_HASH + for position, rec in enumerate(records): + try: + index = rec["index"] + prev_hash = rec["prev_hash"] + stored = rec["entry_hash"] + body = rec["event"] + except (KeyError, TypeError): + return "unverifiable", position + if not isinstance(body, dict): + return "unverifiable", position + if index != position or prev_hash != prev: + return "broken", position + if unknown_event_keys_of(body): + return "unverifiable", position + try: + entry = ChainEntry( + index=index, + prev_hash=prev_hash, + event=Event.from_dict(body), + entry_hash=stored, + preimage_version=rec.get("preimage_version", 1), + ) + recomputed = entry.compute_hash() + except (LedgerError, TypeError, ValueError): + return "unverifiable", position + if recomputed != stored: + return "broken", position + prev = stored + return "intact", None + + +def replay_chain(root) -> list: + out = [] + for i, d in enumerate(ledger_dirs(root)): + records, torn, malformed_at = _read_records(d / "session.log") + status, position = _verify_records(records) + if status != "intact": + status = f"{status}@{position}" + elif malformed_at is not None: + status = f"malformed@{malformed_at}" + elif torn: + status = "torn-tail" + elif not records: + status = "empty" + out.append( + LogReplay( + label=f"log#{i}", + shape="nested" if d.name == ".didrun" else "flat", + entries=len(records), + status=status, + ) + ) + return out + + +def _status_histogram(replays: list) -> dict: + hist: dict = {} + for r in replays: + key = r.status.split("@")[0] + hist[key] = hist.get(key, 0) + 1 + return hist + + +def test_leg1_chain_recompute(replay_source): + """Every archived entry_hash must still recompute under this binary. + + This is the leg that would have caught a chain-preimage change before it + shipped, across every archived event, instead of after. + """ + replays = replay_chain(replay_source.root) + payload = { + "logs": len(replays), + "entries": sum(r.entries for r in replays), + "nested": sum(1 for r in replays if r.shape == "nested"), + "flat": sum(1 for r in replays if r.shape == "flat"), + "status": _status_histogram(replays), + } + _summarize("leg1 chain-recompute", replay_source.name, payload) + + statuses = tuple(r.status for r in replays) + _assert_no_content(statuses) + if replay_source.expected is not None: + assert statuses == replay_source.expected.log_statuses + assert tuple(r.entries for r in replays) == replay_source.expected.log_entries + else: + # On a real archive tree the regression is `broken`: a stored hash that + # no longer recomputes. `unverifiable` is a different fact (written by + # something this binary does not fully understand) and is reported, not + # asserted away. + assert [r.label for r in replays if r.status.startswith("broken")] == [] + assert replays != [] + + +# --- leg 2: note round-trip -------------------------------------------------- + +# Keys `to_json` emits that a stored note of this version may legitimately +# lack, because `from_json` reads them with a v1-reproducing default. Hard +# coded per manifest version so a new manifest field is a deliberate one-line +# diff here and never a silent pass. +# v2 introduced no top-level manifest key — the evidence binding lives inside +# each claim entry. +# `secrets` is P3.2's: the scan-domain record beside `secrets_override`. It is +# additive with a v1-reproducing default ({}), so a stored note of EITHER +# version legitimately lacks it, and both allowlists carry it. This entry is +# the deliberate one-line diff the criterion exists to force — without it the +# round-trip leg would go red across every stored note at once. +# `claims_from`/`claims_to` are P4.1's: the half-open claim window the seal +# covered, which made a truncating re-seal visible in the artifact. Both are read +# with a default (0 / len(claims)), so a stored note of either version +# legitimately lacks them and both allowlists carry them. Second deliberate +# one-line diff this criterion has forced, and the reason it exists. +# Version 3 is P4.2's: the claim-type vocabulary gained `conjunction`. That bump +# added no TOP-LEVEL manifest key — the conjunct list and the per-conjunct grades +# live inside a claim entry, and a claim entry round-trips verbatim — so v3's +# allowlist is v2's. It is spelled out rather than aliased so that a v3-only +# additive key stays a deliberate one-line diff here, exactly as for v1 and v2. +# `authority` is P5.1's: the authorisation artifacts a seal cited, each carrying +# its digest and the limitation constant. Additive with a v1-reproducing default +# ([]) and NO version bump — an authority is a manifest field, never an Event +# field, precisely so it stays compat-neutral. A stored note of any version may +# legitimately lack it (every note published before P5.1 does), so all three +# allowlists carry it. Third deliberate one-line diff this criterion has forced. +MANIFEST_ADDITIVE_KEYS = { + 1: frozenset({"secrets_override", "secrets", "claims_from", "claims_to", "authority"}), + 2: frozenset({"secrets_override", "secrets", "claims_from", "claims_to", "authority"}), + 3: frozenset({"secrets_override", "secrets", "claims_from", "claims_to", "authority"}), +} + + +def note_violations(raw: bytes, ordinal: int, additive_keys=None) -> list: + """Apply the pack's three-part note round-trip criterion to one note body. + + Naive byte equality (`from_json(body).to_json() == body`) is NOT the + criterion and must not be written: it is false on every real note today + (git notes appends one trailing newline to every body) and it becomes false + a second way as Manifest gains additive fields. Instead: + + (a) newline discipline — strip AT MOST one trailing newline; more than + one is a malformed note and fails closed rather than being rstrip'd; + (b) key-superset semantic equality — every key in the source body must + come back with an equal value, and any extra key must be on this + version's additive allowlist; + (c) serialization idempotence — re-serializing is byte-stable. This is + where canonical_json's stability is actually pinned. + + Returns ``(ordinal, code)`` pairs. Notes are identified by ordinal, and + every code is a structural token: nothing derived from the body's VALUES + reaches the caller. + """ + if additive_keys is None: + additive_keys = MANIFEST_ADDITIVE_KEYS + + body = raw[:-1] if raw.endswith(b"\n") else raw + if body.endswith(b"\n"): + return [(ordinal, "multiple-trailing-newlines")] + try: + source = json.loads(body) + except ValueError: + return [(ordinal, "parse-error")] + if not isinstance(source, dict) or "version" not in source: + return [(ordinal, "not-a-manifest")] + version = source["version"] + if not isinstance(version, int) or isinstance(version, bool): + return [(ordinal, "bad-version-type")] + if version > MANIFEST_VERSION: + return [(ordinal, "version-too-new")] + + try: + first = Manifest.from_json(body).to_json() + except Exception: + return [(ordinal, "parse-error")] + round_tripped = json.loads(first) + + violations = [] + for key in sorted(source): + token = _safe_token(key) + if key not in round_tripped: + violations.append((ordinal, f"missing-key:{token}")) + elif round_tripped[key] != source[key]: + violations.append((ordinal, f"value-mismatch:{token}")) + allowed = additive_keys.get(version, frozenset()) + for key in sorted(set(round_tripped) - set(source)): + if key not in allowed: + violations.append((ordinal, f"undeclared-extra-key:{_safe_token(key)}")) + + # Compared by digest, not by bytes: an `assert a == b` over two manifest + # bodies would print both of them. + second = Manifest.from_json(first).to_json() + if sha256_hex(second) != sha256_hex(first): + violations.append((ordinal, "not-idempotent")) + return violations + + +def note_bodies(root) -> list: + """Note bodies via git metadata commands. No ledger file is opened.""" + listing = subprocess.run( + ["git", "-C", str(root), "notes", f"--ref={NOTES_REF}", "list"], + capture_output=True, + ) + if listing.returncode != 0: + return [] + blobs = sorted( + line.split()[0].decode("ascii") + for line in listing.stdout.splitlines() + if line.strip() + ) + bodies = [] + for blob in blobs: + proc = subprocess.run( + ["git", "-C", str(root), "cat-file", "-p", blob], capture_output=True + ) + if proc.returncode == 0: + bodies.append(proc.stdout) + return bodies + + +def replay_notes(root) -> tuple: + bodies = note_bodies(root) + violations = [] + for ordinal, raw in enumerate(bodies): + violations.extend(note_violations(raw, ordinal)) + return len(bodies), violations + + +def test_leg2_note_round_trip(replay_source): + """Every published note must still parse, round-trip and re-serialize. + + Six later prompts cite this leg as their compat gate, so its definition is + load-bearing: it is the criterion above, not byte equality. + """ + if not has_git_repo(replay_source.root): + pytest.skip("no git repo at the replay root; the note legs need refs/notes/didrun") + count, violations = replay_notes(replay_source.root) + _summarize( + "leg2 note-round-trip", + replay_source.name, + {"notes": count, "violations": len(violations)}, + ) + _assert_no_content(violations) + assert violations == [] + assert count >= 1 + if replay_source.expected is not None: + assert count == replay_source.expected.notes + + +# --- leg 3: layout classifier ------------------------------------------------ + +_FLAT_BLOB = re.compile(r"\A[0-9a-f]{64}\Z") +_FANOUT_DIR = re.compile(r"\A[0-9a-f]{2}\Z") +_FANOUT_FILE = re.compile(r"\A[0-9a-f]{38}\Z") + + +def _shape_token(p: Path) -> str: + kind = "dir" if p.is_dir() else "file" + cls = "hex" if re.fullmatch(r"[0-9a-f]+", p.name) else "nonhex" + return f"{kind}:len{len(p.name)}:{cls}" + + +def classify_objects(root) -> tuple: + """Partition every objects/ entry into exactly one of two shapes. + + 113 of the archived object dirs carry BOTH namespaces — flat SHA-256 blobs + written by the ledger and git fan-out dirs from the sealed object + redirect. A third shape is residue and fails closed; it is reported as a + shape descriptor, never as a name. + """ + counts = {"dirs": 0, "flat-64hex": 0, "git-fanout": 0} + residue: dict = {} + for d in ledger_dirs(root): + objects = d / "objects" + if not objects.is_dir(): + continue + counts["dirs"] += 1 + for entry in sorted(objects.iterdir()): + if entry.is_file() and _FLAT_BLOB.match(entry.name): + counts["flat-64hex"] += 1 + elif entry.is_dir() and _FANOUT_DIR.match(entry.name): + for child in sorted(entry.iterdir()): + if child.is_file() and _FANOUT_FILE.match(child.name): + counts["git-fanout"] += 1 + else: + token = "fanout-child:" + _shape_token(child) + residue[token] = residue.get(token, 0) + 1 + else: + token = _shape_token(entry) + residue[token] = residue.get(token, 0) + 1 + return counts, residue + + +def test_leg3_layout_classifier(replay_source): + counts, residue = classify_objects(replay_source.root) + payload = dict(counts) + payload["residue"] = sum(residue.values()) + _summarize("leg3 layout-classifier", replay_source.name, payload) + _assert_no_content(residue) + assert residue == {} + if replay_source.expected is not None: + assert counts == replay_source.expected.objects + + +# --- leg 4: claim-shape invariants ------------------------------------------- + + +def claim_shape_from_bodies(bodies: list) -> dict: + """The shape facts nobody measured until they broke three conclusions.""" + notes = 0 + claims = 0 + multi_index = 0 + pathspecs: dict = {} + gap_ge_1 = 0 + gap_ge_5 = 0 + grades: dict = {} + for raw in bodies: + body = raw[:-1] if raw.endswith(b"\n") else raw + try: + parsed = json.loads(body) + except ValueError: + continue + if not isinstance(parsed, dict): + continue + notes += 1 + for entry in parsed.get("claims", []) or []: + if not isinstance(entry, dict): + continue + claims += 1 + claim = entry.get("claim") or {} + if len(claim.get("event_indices") or []) > 1: + multi_index += 1 + n_specs = len(claim.get("pathspecs") or []) + pathspecs[n_specs] = pathspecs.get(n_specs, 0) + 1 + supporting = entry.get("supporting_event_index") + declared_at = claim.get("declared_at_index", -1) + if ( + isinstance(supporting, int) + and isinstance(declared_at, int) + and declared_at >= 0 + ): + gap = declared_at - supporting + if gap >= 1: + gap_ge_1 += 1 + if gap >= 5: + gap_ge_5 += 1 + grade = _safe_token(entry.get("grade")) + grades[grade] = grades.get(grade, 0) + 1 + return { + "notes": notes, + "claims": claims, + "multi-index": multi_index, + "max-pathspecs": max(pathspecs) if pathspecs else 0, + "pathspecs": pathspecs, + "gap-ge-1": gap_ge_1, + "gap-ge-5": gap_ge_5, + "grades": grades, + } + + +def claim_shape_stats(root) -> dict: + return claim_shape_from_bodies(note_bodies(root)) + + +def test_leg4_claim_shape_invariants(replay_source): + """The leg the red-team says earns its keep. + + A grade histogram is a snapshot and is REPORTED, not asserted, against a + real corpus — asserting it would have passed on every defect in the audit. + The shape invariants are the regression: had they existed, two blocker + findings would have been caught here instead of by a reviewer. The + declared-vs-supporting gap is a signal, so it is reported too. + """ + if not has_git_repo(replay_source.root): + pytest.skip("no git repo at the replay root; the note legs need refs/notes/didrun") + stats = claim_shape_stats(replay_source.root) + _summarize("leg4 claim-shape", replay_source.name, stats) + _assert_no_content(stats) + assert stats["multi-index"] == 0 + assert stats["max-pathspecs"] <= 1 + if replay_source.expected is not None: + # Synthetic only: here the histogram is testing the instrument against + # a fixture whose every value is declared, which is not the same thing + # as pinning a corpus snapshot. + assert stats == replay_source.expected.claim_shape + + +# --- meta: the harness must skip cleanly, and must be able to fail ----------- + + +def test_corpus_legs_skip_when_env_unset(tmp_path): + """CI is green by construction: no env var, no corpus leg, no error.""" + root, reason = resolve_corpus_root({}) + assert root is None + assert reason == SKIP_UNSET + assert ENV_VAR in reason + + root, reason = resolve_corpus_root({ENV_VAR: str(tmp_path / "nope")}) + assert root is None + assert reason == SKIP_MISSING + + root, reason = resolve_corpus_root({ENV_VAR: " "}) + assert root is None + assert reason == SKIP_UNSET + + live = tmp_path / "live" + (live / ".didrun").mkdir(parents=True) + (live / ".git").mkdir() + root, reason = resolve_corpus_root({ENV_VAR: str(live)}) + assert root is None + assert reason == SKIP_LIVE + + ok = tmp_path / "copy" + (ok / ".didrun-history").mkdir(parents=True) + root, reason = resolve_corpus_root({ENV_VAR: str(ok)}) + assert root == ok + assert reason is None + + +def test_refused_trees_are_refused_by_digest_including_subpaths(tmp_path): + """The named-tree refusal, and the proof it also covers paths inside it. + + The digests themselves are frozen in conftest; this exercises the + mechanism against a synthetic path so no private path appears here. + """ + forbidden = tmp_path / "forbidden-tree" + (forbidden / "sub" / "deeper").mkdir(parents=True) + digests = frozenset({path_digest(forbidden)}) + assert is_refused(forbidden, digests) + assert is_refused(forbidden / "sub" / "deeper", digests) + assert is_refused(str(forbidden) + "/", digests) + assert not is_refused(tmp_path / "other", digests) + + # The wiring: a path on the frozen list takes the refusal branch, and it is + # checked BEFORE the directory-exists test, so a refused tree is refused + # whether or not it is currently mounted. + root, reason = resolve_corpus_root({ENV_VAR: str(forbidden)}, digests=digests) + assert root is None + assert reason == SKIP_REFUSED + root, reason = resolve_corpus_root( + {ENV_VAR: str(tmp_path / "absent")}, digests=frozenset({path_digest(tmp_path)}) + ) + assert reason == SKIP_REFUSED + + from .conftest import _REFUSED_PATH_DIGESTS + + assert len(_REFUSED_PATH_DIGESTS) >= 1 + assert all(re.fullmatch(r"[0-9a-f]{64}", d) for d in _REFUSED_PATH_DIGESTS) + + +def test_synthetic_known_bad_logs_are_detected(synthetic_corpus, tmp_path): + """The harness can actually fail: torn, unknown, broken are all detected.""" + replays = replay_chain(synthetic_corpus.root) + by_status = {r.status for r in replays} + _assert_no_content(by_status) + assert "torn-tail" in by_status + assert "unverifiable@0" in by_status + assert "broken@1" in by_status + assert "empty" in by_status + assert sum(1 for r in replays if r.status == "intact") == 2 + + # Independent of the generator's declarations: tamper a good chain here and + # confirm the recompute catches it. + records = synthetic._chain(3) + synthetic._write_log(tmp_path / "good" / ".didrun" / "session.log", records) + before = replay_chain(tmp_path / "good")[0].status + assert before == "intact" + + stored = records[2]["entry_hash"] + records[2]["entry_hash"] = ("0" if stored[0] != "0" else "1") + stored[1:] + synthetic._write_log(tmp_path / "tampered" / ".didrun" / "session.log", records) + after = replay_chain(tmp_path / "tampered")[0].status + assert after == "broken@2" + + +def test_synthetic_known_bad_notes_are_detected(known_bad_corpus): + """Each part of the round-trip criterion fails on the thing it targets. + + Every result is bound to a local before it is asserted on. pytest's + assertion rewriting explains a failing comparison by printing the CALL and + its arguments, so ``assert note_violations(    , 0) == []`` would + put a note body in the failure message. Same rule as everywhere else here, + and the reason the grep gate looks for exactly that shape. + """ + synthetic_note = synthetic.multi_index_note_body() + + # (a) newline discipline: one trailing newline is git's and is stripped; + # two is a malformed note and must fail closed rather than be rstrip'd. + clean = note_violations(synthetic_note, 0) + one_newline = note_violations(synthetic_note + b"\n", 0) + two_newlines = note_violations(synthetic_note + b"\n\n", 0) + assert clean == [] + assert one_newline == [] + assert two_newlines == [(0, "multiple-trailing-newlines")] + + # (b) the additive-key allowlist: with an empty allowlist, the one key a v1 + # note may legitimately lack becomes a reported violation. + stripped = json.loads(synthetic_note) + del stripped["secrets_override"] + trimmed = json.dumps(stripped, sort_keys=True, separators=(",", ":")).encode("ascii") + allowed = note_violations(trimmed, 3) + not_allowed = note_violations(trimmed, 3, additive_keys={1: frozenset()}) + assert allowed == [] + assert not_allowed == [(3, "undeclared-extra-key:secrets_override")] + + unparseable = note_violations(b"not json at all", 1) + not_manifest = note_violations(b'{"nope":1}', 2) + bad_version = note_violations(b'{"version":"1"}', 2) + assert unparseable == [(1, "parse-error")] + assert not_manifest == [(2, "not-a-manifest")] + assert bad_version == [(2, "bad-version-type")] + + # End to end through git: a note whose manifest version this binary must + # refuse rather than coerce. + count, violations = replay_notes(known_bad_corpus) + assert count == 1 + assert violations == [(0, "version-too-new")] + + +def test_synthetic_known_bad_objects_residue_is_detected(known_bad_corpus): + """A third path shape in objects/ fails closed and is named by shape only.""" + counts, residue = classify_objects(known_bad_corpus) + _assert_no_content(residue) + assert counts["flat-64hex"] == 1 + assert counts["git-fanout"] == 1 + assert residue == {"dir:len4:nonhex": 1, "file:len9:nonhex": 1} + + +def test_synthetic_known_bad_claims_are_detected(): + """The invariant leg catches the claim shapes the corpus says do not exist.""" + stats = claim_shape_from_bodies([synthetic.multi_index_note_body()]) + _assert_no_content(stats) + assert stats["multi-index"] == 1 + assert stats["max-pathspecs"] == 2 + assert stats["grades"] == {"tree-exact": 1, "scope-exact": 1} + + +def test_assert_no_content_refuses_content_and_never_echoes_it(): + """The net holds, and its own failure message does not leak the value.""" + _assert_no_content( + { + "logs": 7, + "status": {"intact": 2, "broken@1": 1}, + "ok": True, + "none": None, + "digest": "a" * 64, + "shape": "file:len9:nonhex", + } + ) + + secret = "AKIAIOSFODNN7EXAMPLE-with-a-space and a tail" + with pytest.raises(AssertionError) as exc: + _assert_no_content({"argv": [secret]}) + assert secret not in str(exc.value) + assert "AKIA" not in str(exc.value) + assert str(len(secret)) in str(exc.value) + + with pytest.raises(AssertionError): + _assert_no_content([Path("/w/repo")]) + + assert _safe_token("tree-exact") == "tree-exact" + assert _safe_token("Bearer ABC123") == _NON_STRUCTURAL + assert _safe_token(None) == _NON_STRUCTURAL diff --git a/tests/test_append_concurrency.py b/tests/test_append_concurrency.py new file mode 100644 index 0000000..417ceb3 --- /dev/null +++ b/tests/test_append_concurrency.py @@ -0,0 +1,610 @@ +"""P2.4 — the read-tail-and-append pair is atomic between didrun processes. + +The hazard being closed is corruption, not wall-clock. ``Session.append`` reads +the tail, derives ``(index, prev_hash)`` from it, then writes; two processes +that interleave those three steps both write the same index chained from the +same predecessor, and ``verify_chain`` reports a break that cannot be repaired +without rewriting the log — the one edit an evidence store must never make. + +Two things make the positive leg here mean something. + + - **The negative leg runs the identical harness.** ``test_1`` and ``test_2`` + drive the same child script with the same contention profile; the only + difference between them is whether ``ledger._append_lock`` is the real + context manager or ``contextlib.nullcontext``. A green positive leg that + could not fail is worth nothing, so the negative leg proves the harness + races. + - **The lock's extent is asserted, not assumed.** Test 4 pins that it is not + held across the wrapped child (the failure mode that would trade a rare + recoverable break for a routine hang), and test 3 pins that a read-only + verify never waits on it. + +Honest scope: the deployment class this protects — concurrent didrun +invocations against one ledger — has zero observed instances in the corpus this +work was measured against. The two preserved incidents behind it come from the +evidence documents, not from a race reproduced here. +""" + +from __future__ import annotations + +import contextlib +import os +import shutil +import stat +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from didrun import gitplumbing as gp +from didrun import ledger as ledger_mod +from didrun import manifest as manifest_mod +from didrun.claims import Claim +from didrun.ledger import Event, Session + +# flock is the whole mechanism. Off POSIX the manager is a declared no-op and +# there is nothing here to assert but the degradation (test 11). +fcntl = ledger_mod.fcntl + +ECHO = "/bin/echo" +SH = "/bin/sh" + +pytestmark = pytest.mark.skipif( + not ledger_mod.APPEND_LOCK_AVAILABLE + or not (os.path.exists(ECHO) and os.path.exists(SH)), + reason="POSIX flock semantics and a POSIX /bin layout", +) + +# How many writers race for one ledger. Eight is the figure the design was +# specified against and is enough to make a fork overwhelming rather than +# marginal when the lock is removed. +RACERS = 8 + +# Seconds each racer spends between reading the tail and writing the line, +# injected identically into both legs. See _race() for why this is here and why +# it does not weaken the positive leg. +WIDEN_SECONDS = 0.05 + + +# The racer. It drives the REAL CLI in a real process — the corruption this +# guards against is between processes, so an in-process thread test would not +# reach it. Everything it does is switched by environment variable so the +# locked and unlocked legs run byte-identical code. +_RACER = ''' +import contextlib +import os +import sys +import time + +import didrun.ledger as ledger +from didrun.ledger import Session + +if os.environ.get("DIDRUN_TEST_NO_LOCK") == "1": + # The negative leg. contextlib.nullcontext accepts and ignores the root, so + # this is exactly Session.append with no mutual exclusion: v0.1's behaviour. + ledger._append_lock = contextlib.nullcontext + +WIDEN = float(os.environ.get("DIDRUN_TEST_WIDEN") or 0.0) +RENDEZVOUS = os.environ.get("DIDRUN_TEST_RENDEZVOUS") or "" +PEERS = int(os.environ.get("DIDRUN_TEST_PEERS") or 0) + + +def _rendezvous(path, peers, timeout=120.0): + """Block until `peers` processes have arrived. Single-byte O_APPEND writes.""" + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600) + try: + os.write(fd, b"x") + finally: + os.close(fd) + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + if os.path.getsize(path) >= peers: + return + except OSError: + pass + time.sleep(0.005) + raise SystemExit("rendezvous timed out") + + +_real_last = Session._last +_real_append = Session.append + + +def _last(self): + out = _real_last(self) + if WIDEN: + # Between reading the tail and writing the line. Under the lock this is + # inside the critical section and merely slows the run down; without it + # this is the window every racer reads the same tail through. + time.sleep(WIDEN) + return out + + +def _append(self, event): + if RENDEZVOUS: + # BEFORE the lock, never inside it: a barrier inside the critical + # section would deadlock the locked leg by construction. + _rendezvous(RENDEZVOUS, PEERS) + return _real_append(self, event) + + +Session._last = _last +Session.append = _append + +from didrun.cli import main + +sys.exit(main(sys.argv[1:])) +''' + +_CLI_BOOT = "import sys; from didrun.cli import main; sys.exit(main(sys.argv[1:]))" + +# Acquires the ledger's append lock, publishes readiness, then blocks forever. +# Used to prove the lock is really held (test 3, test 6) and that the kernel +# releases it when the holder is killed (test 6). +_HOLDER = ''' +import os +import sys +import time + +from didrun.ledger import _append_lock + +root, ready = sys.argv[1], sys.argv[2] +with _append_lock(root): + tmp = ready + ".tmp" + open(tmp, "w").write(str(os.getpid())) + os.replace(tmp, ready) + time.sleep(600) +''' + + +def _wait_for(predicate, timeout: float, interval: float = 0.01) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(interval) + return predicate() + + +def _lock_is_held(root: Path) -> bool: + """True when some open file description holds the ledger's exclusive flock. + + A real probe even from the process that holds the lock: flock conflicts are + per open-file-description, so a second os.open in this process is refused + the same way another process would be. Test 7 pins that this probe + discriminates, so a False here is a measurement and not a tautology. + """ + path = Path(root) / ledger_mod.LOCK_FILENAME + fd = os.open(str(path), os.O_RDWR | os.O_CREAT, 0o600) + try: + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + return True + fcntl.flock(fd, fcntl.LOCK_UN) + return False + finally: + os.close(fd) + + +def _wrapper_event(argv=("echo", "hi"), exit_code=0, **kw) -> Event: + return Event( + argv=tuple(argv), + cwd="/tmp", + env_fingerprint="abc123", + observed_via="wrapper", + coverage="complete", + exit_code=exit_code, + **kw, + ) + + +def _read_session(repo: Path) -> Session: + """A read-only view of the ledger — the reader must never create anything.""" + return Session(repo / ".didrun", readonly=True) + + +def _race(repo: Path, tmp_path: Path, tag: str, *, no_lock: bool): + """Run RACERS concurrent `didrun run` invocations against one ledger. + + Returns ``(returncodes, verdict, indices)`` read back through a read-only + session. The contention profile is IDENTICAL in both legs — a start + rendezvous immediately before ``append`` plus WIDEN_SECONDS between the tail + read and the write — so the only variable across test 1 and test 2 is the + lock itself. + """ + script = tmp_path / "racer.py" + script.write_text(_RACER, encoding="utf-8") + env = dict(os.environ) + env["DIDRUN_TEST_WIDEN"] = str(WIDEN_SECONDS) + env["DIDRUN_TEST_RENDEZVOUS"] = str(tmp_path / f"rendezvous-{tag}") + env["DIDRUN_TEST_PEERS"] = str(RACERS) + if no_lock: + env["DIDRUN_TEST_NO_LOCK"] = "1" + procs = [ + subprocess.Popen( + [ + sys.executable, + str(script), + "--repo", + str(repo), + "run", + "--", + ECHO, + str(i), + ], + cwd=str(repo), + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + for i in range(RACERS) + ] + codes = [] + for proc in procs: + try: + proc.communicate(timeout=180) + except subprocess.TimeoutExpired: + proc.kill() + proc.communicate(timeout=30) + pytest.fail("a racer never finished — the lock is wider than the append") + codes.append(proc.returncode) + session = _read_session(repo) + verdict = session.verify_chain_detail() + indices = [e.index for e in session.entries()] + return codes, verdict, indices + + +# --- 1. the race, with the lock ---------------------------------------------- + + +def test_eight_concurrent_runs_leave_one_intact_chain(repo: Path, tmp_path: Path): + codes, verdict, indices = _race(repo, tmp_path, "locked", no_lock=False) + + assert codes == [0] * RACERS + assert verdict.status == "intact", verdict.reason + assert len(indices) == RACERS + assert set(indices) == set(range(RACERS)) + # Sequence, not just membership: a fork that happened to land on a complete + # set would still be out of order on disk. + assert indices == list(range(RACERS)) + + +# --- 2. the negative leg: the same harness, no lock -------------------------- + + +def test_the_same_race_forks_the_chain_without_the_lock(repo: Path, tmp_path: Path): + """The control. Without the lock this harness must actually corrupt. + + **What the harness had to do to make this fail** — recorded because a + negative leg that needed help is only honest if the help is named, and + because both helpers are applied to the LOCKED leg too: + + 1. A start rendezvous immediately before ``Session.append``, outside the + lock. Eight ``didrun run`` invocations otherwise finish their tree + digests at scattered times and mostly do not overlap at all; the + barrier makes them arrive together, which is the situation the lock + exists for and not one it gets to dodge. + 2. ``WIDEN_SECONDS`` of sleep between the tail read and the write. The + unlocked window is a few hundred microseconds of local I/O, so without + widening the outcome is a coin toss per attempt rather than a + demonstration. + + Neither weakens test 1: the identical profile runs there, and the lock has + to hold the chain together through it. + + Measured on the machine this was written on: 3 of 3 unlocked attempts forked + (duplicate indices, chain reported broken at index 0). The assertion stays + at "at least one" so a slower or busier machine cannot turn a real + demonstration into a red suite. + """ + attempts = 3 + forks = 0 + for attempt in range(attempts): + shutil.rmtree(repo / ".didrun", ignore_errors=True) + _codes, verdict, indices = _race( + repo, tmp_path, f"unlocked-{attempt}", no_lock=True + ) + duplicated = len(indices) != len(set(indices)) + if duplicated or verdict.status != "intact": + forks += 1 + + assert forks > 0, ( + "the unlocked race never forked the chain, so test 1 proves nothing — " + "raise RACERS or WIDEN_SECONDS until it does" + ) + + +# --- 3. the read path takes no lock ------------------------------------------ + + +def test_a_read_only_verify_does_not_wait_on_the_append_lock( + repo: Path, tmp_path: Path, git +): + """A held write lock must not stall a reader. Verification is the hot path + in CI and a reader that contends turns a corruption fix into an outage.""" + subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "run", "--", ECHO, "hi"], + cwd=str(repo), + capture_output=True, + check=True, + ) + subprocess.run( + [ + sys.executable, + "-c", + _CLI_BOOT, + "--repo", + str(repo), + "claim", + "tests-pass", + "--label", + "reader", + ], + cwd=str(repo), + capture_output=True, + check=True, + ) + git(repo, "commit", "-qm", "seal point", "--allow-empty") + subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "seal"], + cwd=str(repo), + capture_output=True, + check=True, + ) + + root = repo / ".didrun" + with ledger_mod._append_lock(root): + assert _lock_is_held(root), "the fixture did not actually hold the lock" + # No timeout= fallback: a reader that needs one has already failed. + proc = subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "verify"], + cwd=str(repo), + capture_output=True, + timeout=60, + ) + assert proc.returncode in (0, 1), proc.stderr.decode() + show = subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "show", "--session"], + cwd=str(repo), + capture_output=True, + timeout=60, + ) + assert show.returncode == 0, show.stderr.decode() + + +# --- 4. the lock is never held across the wrapped child ---------------------- + + +def test_the_lock_is_not_held_across_the_wrapped_child(repo: Path, tmp_path: Path): + """The failure this prompt was specified to avoid. + + Acquiring for the duration of `cli.main` would be trivially correct and + would make a second `didrun run` block for as long as the first command + takes — hours, in the workload this tool was built for. + """ + marker = tmp_path / "child-running" + slow = subprocess.Popen( + [ + sys.executable, + "-c", + _CLI_BOOT, + "--repo", + str(repo), + "run", + "--", + SH, + "-c", + f"touch {marker}; sleep 5", + ], + cwd=str(repo), + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + try: + assert _wait_for(marker.exists, 60), "the wrapped child never started" + started = time.monotonic() + quick = subprocess.run( + [ + sys.executable, + "-c", + _CLI_BOOT, + "--repo", + str(repo), + "run", + "--", + ECHO, + "x", + ], + cwd=str(repo), + capture_output=True, + timeout=60, + ) + elapsed = time.monotonic() - started + assert quick.returncode == 0, quick.stderr.decode() + # The wrapped child sleeps 5s and has barely started. Anything near that + # means the lock covered it. + assert elapsed < 3.0, f"the second run took {elapsed:.2f}s" + finally: + slow.kill() + slow.communicate(timeout=30) + + +# --- 5. lock file hygiene ---------------------------------------------------- + + +def test_lock_file_is_private_ignored_and_outside_every_tree_digest( + repo: Path, git +): + root = repo / ".didrun" + session = Session(root) + before = gp.tree_digest(repo, ledger_objects=session.blobs.root) + assert before is not None + + with ledger_mod._append_lock(root): + pass + + lock = root / ledger_mod.LOCK_FILENAME + assert lock.exists() + assert stat.S_IMODE(lock.stat().st_mode) == 0o600 + + # Ignored by the ledger's own "*" rule, so no `git add -A` can stage it. + ignored = subprocess.run( + ["git", "check-ignore", "-v", str(lock.relative_to(repo))], + cwd=str(repo), + capture_output=True, + text=True, + ) + assert ignored.returncode == 0, ignored.stdout + ignored.stderr + + # And excluded from the digest unconditionally, not merely by being ignored. + after = gp.tree_digest(repo, ledger_objects=session.blobs.root) + assert after == before + listing = subprocess.run( + ["git", "ls-tree", "-r", "--name-only", after], + cwd=str(repo), + capture_output=True, + text=True, + env={ + **os.environ, + "GIT_OBJECT_DIRECTORY": str(session.blobs.root), + "GIT_ALTERNATE_OBJECT_DIRECTORIES": str(gp.objects_dir(repo)), + }, + ) + assert listing.returncode == 0, listing.stderr + assert ".didrun" not in listing.stdout + + +# --- 6. a killed holder leaves no stale lock --------------------------------- + + +def test_a_sigkilled_holder_leaves_no_stale_lock(repo: Path, tmp_path: Path): + """flock is the reason this test is short. A PID file or an O_EXCL sentinel + would need liveness checks and a recovery path; the kernel drops an flock + when the holder dies, however it dies.""" + root = repo / ".didrun" + session = Session(root) + script = tmp_path / "holder.py" + script.write_text(_HOLDER, encoding="utf-8") + ready = tmp_path / "holder.ready" + + holder = subprocess.Popen( + [sys.executable, str(script), str(root), str(ready)], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + try: + assert _wait_for(ready.exists, 60), "the holder never acquired the lock" + assert _lock_is_held(root), "the holder claimed the lock but does not hold it" + finally: + holder.kill() + holder.communicate(timeout=30) + + assert _wait_for(lambda: not _lock_is_held(root), 10), "the lock outlived its holder" + started = time.monotonic() + entry = session.append(_wrapper_event()) + assert time.monotonic() - started < 5.0 + assert entry.index == 0 + + +# --- 7. the probe discriminates (so tests 3 and 6 are measurements) ---------- + + +def test_the_lock_probe_reports_false_when_nothing_holds_the_lock(repo: Path): + root = repo / ".didrun" + Session(root) + assert _lock_is_held(root) is False + with ledger_mod._append_lock(root): + assert _lock_is_held(root) is True + assert _lock_is_held(root) is False + + +# --- 8. a read-only session never touches the lock file ---------------------- + + +def test_a_read_only_session_never_creates_the_lock_file(tmp_path: Path): + root = tmp_path / ".didrun" + session = Session(root, readonly=True) + with pytest.raises(ledger_mod.LedgerError): + session.append(_wrapper_event()) + assert not root.exists() + assert not (root / ledger_mod.LOCK_FILENAME).exists() + + +# --- 9 + 10. the claim and seal appends take the same lock ------------------- + + +def _asserting_lock(recorded: list, real): + @contextlib.contextmanager + def probing(root): + with real(root): + recorded.append(_lock_is_held(Path(root))) + yield + + return probing + + +def test_declare_claim_holds_the_lock_while_it_writes(repo: Path, monkeypatch): + """A claims.jsonl line is not bounded by PIPE_BUF once a claim declares + pathspecs, so O_APPEND atomicity is not a guarantee here.""" + session = Session(repo / ".didrun") + session.append(_wrapper_event()) + held: list = [] + monkeypatch.setattr( + manifest_mod, "_append_lock", _asserting_lock(held, ledger_mod._append_lock) + ) + + manifest_mod.declare_claim( + session, + Claim( + ctype="tests-pass", + label="locked", + event_indices=(0,), + pathspecs=("src/", "tests/"), + declared_at_index=0, + ), + ) + + assert held == [True] + assert len(manifest_mod._load_claims(session)) == 1 + assert _lock_is_held(session.root) is False + + +def test_record_seal_holds_the_lock_while_it_writes(repo: Path, monkeypatch): + session = Session(repo / ".didrun") + held: list = [] + monkeypatch.setattr( + manifest_mod, "_append_lock", _asserting_lock(held, ledger_mod._append_lock) + ) + + manifest_mod._record_seal(session, "c" * 40, "t" * 40, 1) + + assert held == [True] + assert manifest_mod._last_seal_watermark(session) == 1 + assert _lock_is_held(session.root) is False + + +# --- 11. the platform degradation is recorded, not asserted away ------------- + + +def test_without_fcntl_the_manager_is_a_declared_no_op(tmp_path: Path, monkeypatch): + """docs/COMPAT.md states that append serialization is POSIX-only. That + sentence has to be backed by behaviour: no invented fallback, no lock file, + and appends that still work.""" + root = tmp_path / ".didrun" + monkeypatch.setattr(ledger_mod, "fcntl", None) + monkeypatch.setattr(ledger_mod, "APPEND_LOCK_AVAILABLE", False) + + session = Session(root) + assert session.append_serialized is False + with ledger_mod._append_lock(root): + pass + assert not (root / ledger_mod.LOCK_FILENAME).exists() + + entry = session.append(_wrapper_event()) + assert entry.index == 0 + assert not (root / ledger_mod.LOCK_FILENAME).exists() + assert session.verify_chain_detail().status == "intact" diff --git a/tests/test_capture_claims_manifest.py b/tests/test_capture_claims_manifest.py index 3ab5f89..6e92fc5 100644 --- a/tests/test_capture_claims_manifest.py +++ b/tests/test_capture_claims_manifest.py @@ -2,12 +2,16 @@ from __future__ import annotations +import json +import re import subprocess import sys from pathlib import Path import pytest +from didrun import capture +from didrun import cli from didrun.ledger import Session from didrun.capture import run_wrapped from didrun.claims import Claim, ClaimError, grade @@ -194,3 +198,143 @@ def test_strict_nonzero_on_stale(repo: Path): M.seal(s, repo) report = M.verify(s, repo) assert not report.all_verified # → CLI --strict returns nonzero + + +# --- P0.3: the format contract ------------------------------------------------ + +def _note_body(version: int, tree: str, commit: str = "0" * 40) -> bytes: + """A structurally complete note body at an arbitrary manifest version.""" + return json.dumps( + { + "version": version, + "commit": commit, + "tree": tree, + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + "secrets_override": False, + } + ).encode("ascii") + + +def _attach_raw_note(repo: Path, commit: str, body: bytes) -> None: + """Publish arbitrary bytes as a note, bypassing Manifest entirely.""" + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-F", "-", commit], + cwd=str(repo), + input=body, + capture_output=True, + check=True, + ) + + +def _note_of(repo: Path, commit: str): + """The note attached to ``commit``, or None — read with raw git, not didrun.""" + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +def _commit_file(repo: Path, name: str) -> str: + """A commit with a tree distinct from every other commit in the fixture.""" + (repo / name).write_text(f"content of {name}\n") + _git(repo, "add", name) + _git(repo, "commit", "-qm", f"add {name}") + return gp.head_commit(repo) + + +def test_from_json_refuses_a_newer_manifest_version(repo: Path): + """Refuse, never coerce: a v99 note may have redefined a field this binary + reads, so grading it under v1 semantics would be a confident lie.""" + with pytest.raises(M.ManifestError) as exc: + M.Manifest.from_json(_note_body(99, "a" * 40)) + msg = str(exc.value) + assert "version 99" in msg # the version it met + assert f"max {M.MANIFEST_VERSION}" in msg # the version it understands + assert "upgrade didrun" in msg + + +def test_from_json_accepts_the_current_manifest_version(): + """The gate is `>`, not `!=` — every note already published is version 1.""" + m = M.Manifest.from_json(_note_body(M.MANIFEST_VERSION, "b" * 40)) + assert m.version == M.MANIFEST_VERSION + assert m.tree == "b" * 40 + + +def test_fallback_skips_an_unparseable_note_and_counts_it(repo: Path): + """A foreign or corrupt note under the same ref must not hide every good + one — but the skip is counted, never silently reported as "no manifest".""" + stranger = gp.head_commit(repo) + target = _commit_file(repo, "target.py") + _attach_raw_note(repo, stranger, b"not json") + assert _note_of(repo, target) is None # the commit-exact path finds nothing + + manifest, resolved_by, skipped = M._resolve_manifest( + repo, target, gp.commit_tree(repo, target) + ) + assert manifest is None and resolved_by == "none" + assert skipped == 1 + assert M.verify(_session(repo), repo, commitish=target).notes_skipped == 1 + + +def test_fallback_refuses_a_future_version_it_scans_past(repo: Path): + """Fail closed on the version even in the fallback loop, and do it + order-independently: NEITHER note's tree matches the commit under test, so + the loop cannot exit early on a match and must reach the v99 note whichever + order `git notes list` yields.""" + garbage_commit = gp.head_commit(repo) + future_commit = _commit_file(repo, "future.py") + target = _commit_file(repo, "target.py") + + target_tree = gp.commit_tree(repo, target) + future_tree = gp.commit_tree(repo, future_commit) + assert future_tree != target_tree # the fixture's whole point + + _attach_raw_note(repo, garbage_commit, b"not json") + _attach_raw_note(repo, future_commit, _note_body(99, future_tree, future_commit)) + assert _note_of(repo, target) is None # commit-exact found nothing first + + with pytest.raises(M.ManifestError) as exc: + M._resolve_manifest(repo, target, target_tree) + assert "version 99" in str(exc.value) + + +def test_cli_refuses_a_future_note_as_a_message_not_a_traceback(repo: Path, capsys): + """Fail-closed has to reach the operator as a refusal. A traceback is a + crash report, and a CI log full of stack frames reads as "didrun is broken" + rather than "this note was written by a newer didrun". Exit 2, not + --strict's 1: unreadable evidence is a different fact from bad evidence.""" + head = gp.head_commit(repo) + _attach_raw_note(repo, head, _note_body(99, gp.commit_tree(repo, head), head)) + rc = cli.main(["--repo", str(repo), "verify", "--strict"]) + err = capsys.readouterr().err + assert rc == 2 + assert "version 99" in err and "upgrade didrun" in err + assert "Traceback" not in err + + +def test_env_fingerprint_carries_its_version_in_band(): + # The shape is pinned in tests/test_env_fingerprint.py, which owns the v2 + # contract; what this asserts is that the version is READABLE off the wire + # and that an unversioned v0.1 digest still reports as unversioned. + assert capture.fingerprint_version(capture.env_fingerprint({"PATH": "/bin"})) == 2 + assert capture.fingerprint_version("v1:0123456789abcdef") == 1 + # A bare v0.1 digest: incomparable with a versioned one, not drifted. + assert capture.fingerprint_version("0123456789abcdef") is None + + +def test_env_fingerprint_is_deterministic_and_env_sensitive(): + """Determinism is a trust-path invariant; sensitivity is the point of the + field. No hard-coded digest — the key set has changed once already.""" + env = {"PATH": "/usr/bin", "SHELL": "/bin/zsh", "LANG": "C"} + assert capture.env_fingerprint(env) == capture.env_fingerprint(dict(env)) + assert capture.env_fingerprint({**env, "SHELL": "/bin/bash"}) != capture.env_fingerprint(env) + + +def test_recorded_events_carry_the_versioned_fingerprint(repo: Path): + """The prefix reaches the ledger, not just the helper's return value.""" + s = _session(repo) + ev = run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + assert capture.fingerprint_version(ev.env_fingerprint) == 2 diff --git a/tests/test_chain_gate.py b/tests/test_chain_gate.py new file mode 100644 index 0000000..f889844 --- /dev/null +++ b/tests/test_chain_gate.py @@ -0,0 +1,395 @@ +"""P1.3 — `verify` consults the ledger hash chain and fails closed. + +The defect these pin: `verify_chain` had exactly one caller in the whole +package and it was `show --session`. `didrun verify --strict` — the command an +entire discipline gates on — never recomputed the chain, so a forked ledger +(duplicate indices) or a mid-file rewrite certified green and exited 0. + +What a chain check buys, stated as narrowly as it deserves: accident and drift +detection on the ledger the verdict was read out of. It is not a forger +barrier — whoever can rewrite a record can recompute every hash after it +(ledger.py's declared non-guarantee). A `broken` chain says an entry did not +recompute; it does not say who moved it. + +Two states that are deliberately NOT faults: `absent` (the ledger is elsewhere, +which P1.2 already reports per claim as witness-unavailable) and `empty`. +Treating either as tamper would make every verify against an archived ledger +read like an attack, which is the failure mode this file's tests 4 and 5 exist +to stop. +""" + +from __future__ import annotations + +import json +import shutil +import sys +from pathlib import Path + +from didrun import cli +from didrun import manifest as M +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import ChainVerdict, Session, canonical_json + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _sealed(repo: Path, events: int = 3) -> Session: + """`events` recorded successes, one claim over event 0, sealed clean. + + The claim binds event 0 on purpose: every corruption below lands on a LATER + record, so the claim's own evidence stays intact and gradeable. A test whose + claim went witness-unavailable would fail for the wrong reason and stop + testing the chain gate. + """ + s = _session(repo) + for i in range(events): + run_wrapped([sys.executable, "-c", f"print({i!r})"], s, repo) + M.declare_claim( + s, Claim(ctype="tests-pass", label="suite", event_indices=(0,), declared_at_index=0) + ) + M.seal(s, repo) + return s + + +def _records(s: Session) -> list[dict]: + return [ + json.loads(line) + for line in s.log_path.read_text(encoding="ascii").splitlines() + if line.strip() + ] + + +def _rewrite(s: Session, mutate) -> None: + """Round-trip the log through canonical JSON, mutating it via ``mutate``. + + The assertion is the load-bearing part: it proves the round-trip is + byte-stable on the records nobody touched, so a corruption test can never + pass merely because re-serializing the log moved a byte somewhere else. + """ + original = s.log_path.read_bytes() + recs = _records(s) + assert b"".join(canonical_json(r) + b"\n" for r in recs) == original, ( + "the canonical round-trip is not byte-stable; every corruption test in " + "this file would then be testing the round-trip, not the corruption" + ) + mutate(recs) + s.log_path.write_bytes(b"".join(canonical_json(r) + b"\n" for r in recs)) + + +def _flip(h: str) -> str: + return ("0" if h[0] != "0" else "1") + h[1:] + + +def _corrupt_middle_hash(s: Session) -> None: + """Flip a byte of the STORED hash of a middle record. + + Chosen over corrupting the event body so the events themselves are + untouched: every claim still grades exactly as it did, and the only thing + that changed in the whole report is the chain. That is what makes the + falsification test below a clean control. + """ + _rewrite(s, lambda recs: recs[1].__setitem__("entry_hash", _flip(recs[1]["entry_hash"]))) + + +# --- test 1: a broken chain fails --strict ----------------------------------- + + +def test_broken_chain_dominates_the_verdict_and_fails_strict(repo: Path, monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + _corrupt_middle_hash(s) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "broken" + assert report.chain_broken_index == 1 + assert report.worst_status == "chain-broken" + assert not report.all_verified + + out = render.render_verdict(report) + assert "ledger chain BROKEN at index 1" in out + assert "every grade below is unreliable" in out + assert "ALL RECORDED-EXACT" not in out + + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +def test_the_chain_check_is_the_only_thing_failing_that_report(repo: Path, monkeypatch): + """The falsification half of test 1, on the SAME corrupted fixture. + + Neutralize only the chain check and the identical corrupted ledger goes + green — which is the proof that the chain gate, and nothing else in the + report, is what test 1 detected. If this test ever fails, test 1 has started + passing for some other reason and is no longer evidence of anything. + """ + s = _sealed(repo) + _corrupt_middle_hash(s) + + monkeypatch.setattr( + Session, "verify_chain_detail", lambda self: ChainVerdict("intact") + ) + report = M.verify(_session(repo), repo) + assert report.chain_status == "intact" + assert report.all_verified, ( + "with the chain check neutralized this corrupted ledger must grade " + "green; if it does not, test 1's red came from somewhere else" + ) + assert report.worst_status == "tree-exact" + assert "ledger chain" not in render.render_verdict(report) + + +# --- test 2: a fork (duplicate index) ---------------------------------------- + + +def test_duplicate_index_reads_as_a_broken_chain(repo: Path, monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + # Re-append the last record verbatim: same index, same prev_hash, same + # stored hash. Every hash in the file still recomputes -- only the SEQUENCE + # is wrong, which is the shape a forked ledger actually has. + _rewrite(s, lambda recs: recs.append(dict(recs[-1]))) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "broken" + assert report.chain_broken_index == 2 + assert not report.all_verified + assert "ledger chain BROKEN at index 2" in render.render_verdict(report) + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +# --- test 3: unverifiable is not "intact", and is not "broken" either -------- + + +def test_unknown_preimage_version_is_unverifiable_not_broken(repo: Path, monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + _rewrite(s, lambda recs: recs[1].__setitem__("preimage_version", 2)) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "unverifiable" + assert report.chain_broken_index is None # nothing was shown not to recompute + assert report.worst_status == "chain-broken" + assert not report.all_verified + + # The verdict TOKEN is CHAIN-BROKEN for both fault states — one registered + # verdict, per the pack. The distinction lives in the banner, which is the + # line that says what actually happened, and it must not say BROKEN: a + # record this binary cannot recompute is not evidence of mutation, and + # saying so would accuse a future writer of tampering. + banner = render.chain_banner_text(report) + assert "unverifiable" in banner + assert "BROKEN" not in banner + assert "unknown chain preimage version: 2" in banner # the reason, named + + out = render.render_verdict(report) + assert banner in out + assert "ALL RECORDED-EXACT" not in out + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +# --- test 4: an absent ledger is not tamper ---------------------------------- + + +def test_absent_ledger_is_witness_unavailable_not_a_broken_chain( + repo: Path, tmp_path: Path, monkeypatch +): + monkeypatch.setenv("NO_COLOR", "1") + _sealed(repo) + shutil.move(str(repo / ".didrun"), str(tmp_path / "archived-ledger")) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "absent" + assert report.worst_status == "witness-unavailable" # the claims, not the chain + assert [r.grade for r in report.results] == ["witness-unavailable"] + assert not report.all_verified + + out = render.render_verdict(report) + assert "ledger chain" not in out, "a ledger that is elsewhere has not been tampered with" + assert "BROKEN" not in out + assert "witness ledger unavailable" in out + + # --strict still refuses, for the honest reason: nobody has seen the evidence. + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +def test_empty_log_is_not_a_chain_fault(repo: Path, monkeypatch): + """`empty` is a present-but-zero-length log, and it is not tamper either. + + Distinct from `absent` (no file at all) so the two stay tellable apart, and + excluded from the fault set for the same reason: nothing failed to + recompute. The claims still refuse --strict on their own. + """ + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + s.log_path.write_bytes(b"") + + report = M.verify(_session(repo), repo) + assert report.chain_status == "empty" + assert report.worst_status == "witness-unavailable" + assert not report.all_verified + assert "ledger chain" not in render.render_verdict(report) + + +def test_unreadable_log_refuses_instead_of_crashing_a_commit_with_no_note(repo: Path): + """An unreadable ledger must not start crashing verify on an unsealed commit. + + The chain check runs before the manifest is resolved, so it reads a ledger + on a path that previously never touched one. Nothing is sealed here: before + P1.3 this returned a clean "no claims" report, and it still must — as a + refusal that names the problem, not a traceback. + + The damage used here is corruption in the MIDDLE of the log. P2.3 gave the + reader a narrow tolerance for a torn FINAL record — a lost event, not + tamper, pinned by the test below — and that tolerance stops at the last + line on purpose: a hole with records after it is still unreadable, and + still a graded refusal rather than a crash. + """ + s = _session(repo) + for i in range(3): + run_wrapped([sys.executable, "-c", f"print({i!r})"], s, repo) + lines = s.log_path.read_bytes().splitlines(keepends=True) + assert len(lines) == 3 + s.log_path.write_bytes(lines[0] + lines[1][:40] + b"\n" + lines[2]) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "unverifiable" + assert report.results == [] + assert "could not be read" in report.chain_reason + assert "ledger chain unverifiable" in render.render_verdict(report) + + +def test_a_torn_final_record_is_a_lost_event_not_a_chain_fault(repo: Path): + """P2.3: the tail an interrupted append leaves must not read as tamper. + + Before P2.3 this was `unverifiable` — the log could not be read at all, so + the ledger's last word on an interrupted session was "this binary cannot + recompute me". It is now the honest thing: every link that survives does + recompute, the chain is `intact` over that prefix, and the truncation + travels in the reason. The fault this file exists to make dominant must NOT + fire here, because nothing failed to recompute. + """ + s = _session(repo) + for i in range(2): + run_wrapped([sys.executable, "-c", f"print({i!r})"], s, repo) + lines = s.log_path.read_bytes().splitlines(keepends=True) + assert len(lines) == 2 + # A partial final record: the shape a crash mid-append leaves behind. Cut at + # a byte offset, not a line boundary, or the log is merely SHORT and parses. + s.log_path.write_bytes(lines[0] + lines[1][:40]) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "intact" + assert report.results == [] + assert "torn at byte" in report.chain_reason + # Surfaced, but never as a chain fault: no banner, no tamper sentence. + assert "ledger chain" not in render.render_verdict(report) + + +# --- test 5: the happy path is untouched ------------------------------------- + + +def test_intact_chain_adds_nothing_to_the_verdict(repo: Path, monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + _sealed(repo) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "intact" + assert report.chain_broken_index is None + assert report.worst_status == "tree-exact" + assert report.all_verified + + out = render.render_verdict(report) + assert "ALL RECORDED-EXACT" in out + assert "chain" not in out.lower(), "an intact chain is not news; it prints nothing" + # The class is always in the stylesheet; the block must not be in the body. + assert '
    ' not in render.render_html(report) + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_show_session_is_unchanged(repo: Path, capsys): + """`show --session` was the sole caller of verify_chain and keeps its shape.""" + s = _sealed(repo) + assert cli.main(["--repo", str(repo), "show", "--session"]) == 0 + assert "chain intact" in capsys.readouterr().out + + _corrupt_middle_hash(s) + assert cli.main(["--repo", str(repo), "show", "--session"]) == 1 + assert "BROKEN at index 1" in capsys.readouterr().out + + +# --- test 6: the render surfaces cannot silently drop it --------------------- + + +def test_chain_broken_is_registered_in_both_render_tables(): + """An unregistered verdict renders as UNKNOWN through `.get(…, UNKNOWN)`. + + That is the silent lie render.py's own comment warns about, and it would + turn the strongest statement didrun can make about a ledger into its + weakest. + """ + assert "chain-broken" in render._GRADE_DISPLAY + assert "chain-broken" in render._SORT_RANK + assert render._SORT_RANK["chain-broken"] < render._SORT_RANK["failed"] + assert render._GRADE_DISPLAY["chain-broken"][0] == "CHAIN-BROKEN" + + +def test_neither_surface_reads_as_passed_when_the_chain_is_broken( + repo: Path, monkeypatch +): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + _corrupt_middle_hash(s) + report = M.verify(_session(repo), repo) + + out = render.render_verdict(report) + assert "CHAIN-BROKEN" in out + assert "ALL RECORDED-EXACT" not in out + assert "UNKNOWN" not in out, "the verdict must not fall back to UNKNOWN" + for overclaim in ("VERIFIED", "PASSED", "PROVEN"): + assert overclaim not in out.upper() + + html_out = render.render_html(report) + assert "CHAIN-BROKEN" in html_out + assert '
    ' in html_out + assert "ledger chain BROKEN at index 1" in html_out + assert "ALL RECORDED-EXACT" not in html_out + + +def test_quiet_html_still_prints_the_chain_banner(repo: Path, tmp_path: Path, capsys, monkeypatch): + """--quiet suppresses the verdict, never the ledger's integrity.""" + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + _corrupt_middle_hash(s) + out_html = tmp_path / "report.html" + + rc = cli.main( + ["--repo", str(repo), "verify", "--strict", "--html", str(out_html), "--quiet"] + ) + printed = capsys.readouterr().out + assert rc == 1 + assert "ledger chain BROKEN at index 1" in printed + assert "STATUS" not in printed, "--quiet still suppresses the per-claim table" + + +def test_no_claims_report_still_reports_a_broken_chain(repo: Path, monkeypatch): + """A commit with nothing sealed still gets the sentence. + + The ledger is what every other command reads; "nothing sealed here" is not a + reason to stay quiet about it not recomputing. + """ + monkeypatch.setenv("NO_COLOR", "1") + s = _session(repo) + for i in range(3): + run_wrapped([sys.executable, "-c", f"print({i!r})"], s, repo) + _corrupt_middle_hash(s) + + report = M.verify(_session(repo), repo) + assert report.results == [] + assert report.chain_status == "broken" + assert report.worst_status == "chain-broken" + out = render.render_verdict(report) + assert "NO CLAIMS" in out + assert "ledger chain BROKEN at index 1" in out diff --git a/tests/test_claim_binding.py b/tests/test_claim_binding.py new file mode 100644 index 0000000..a7809d4 --- /dev/null +++ b/tests/test_claim_binding.py @@ -0,0 +1,270 @@ +"""P5.2 — `didrun claim` must not receipt an older passing run. + +The defect, reachable through the documented happy path and nothing more +exotic: `cmd_claim` with no `--event` bound to the last event whose +``exit_code == 0`` rather than to the last event. So + + didrun run -- pytest -q # passes, event 0 + + didrun run -- pytest -q # FAILS, event 1 + didrun claim tests-pass # bound event 0 + +produced a claim backed by the older passing run. ``declared_at_index`` stayed +at ``len(events) - 1``, so the retroactive-binding rule (which fires only when +the claim was declared BEFORE its backing event) did not catch it, and if the +tree had not moved between the two runs the claim graded ``tree-exact``. The +only signal was the index printed in a success line. + +The fix is a refusal, not a better heuristic: bind to the last event, and if the +last event did not exit 0, refuse and say so. Binding an earlier event stays +available through an explicit ``--event N``, which now warns about the gap. +""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as _manifest +from didrun.capture import run_wrapped +from didrun.ledger import Event, Session + +PASS = [sys.executable, "-c", "print('ok')"] +FAIL = [sys.executable, "-c", "import sys; sys.exit(3)"] + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _claims(repo: Path) -> list: + return _manifest._load_claims(_session(repo)) + + +def _claim(repo: Path, *args: str) -> int: + return cli.main(["--repo", str(repo), "claim", *args]) + + +# --- 1. the headline case ----------------------------------------------------- + + +def test_a_failing_last_event_refuses_instead_of_receipting_the_older_pass( + repo: Path, capsys +): + """Pass, then fail, then claim with no --event. + + On pre-change code this SILENTLY receipted the older passing event 0 and + exited 0. The test therefore asserts three separate things, because "exit 2" + alone would still pass on a version that wrote the claim and then failed for + an unrelated reason: the exit code, that the message names the failing index + and its real exit code, and that claims.jsonl gained nothing at all. + """ + s = _session(repo) + run_wrapped(PASS, s, repo) + run_wrapped(FAIL, s, repo) + assert [ev.exit_code for ev in s.events()] == [0, 3] + capsys.readouterr() + + assert _claim(repo, "tests-pass", "--label", "unit") == 2 + err = capsys.readouterr().err + assert "index 1" in err, err + assert "exited 3" in err, err + assert _claims(repo) == [], "a refused claim must write nothing" + assert not (repo / ".didrun" / "claims.jsonl").exists() or ( + repo / ".didrun" / "claims.jsonl" + ).read_bytes() == b"" + + +def test_the_refusal_names_the_deliberate_escape(repo: Path, capsys): + """A refusal that does not say what to do instead gets worked around. + + The escape is `--event N`, and it must be named in the refusal itself: an + operator who really does mean the earlier run has a supported way to say so. + """ + s = _session(repo) + run_wrapped(PASS, s, repo) + run_wrapped(FAIL, s, repo) + capsys.readouterr() + assert _claim(repo, "tests-pass") == 2 + assert "--event" in capsys.readouterr().err + + +def test_a_last_event_that_witnessed_no_exit_refuses_too(repo: Path, capsys): + """``exit_code is None`` is not a pass either. + + An interrupted flight and a PTY transcript both record an event that + witnesses no exit code, and the old backwards search skipped straight over + them to an earlier success. "Not 0" includes "unknown", and the message says + which of the two it is rather than printing "exited None". + """ + s = _session(repo) + run_wrapped(PASS, s, repo) + s.append( + Event( + argv=("some-command",), + cwd=str(repo), + env_fingerprint="0" * 16, + observed_via="transcript", + coverage="display-only", + ) + ) + assert s.events()[-1].exit_code is None + capsys.readouterr() + + assert _claim(repo, "tests-pass") == 2 + err = capsys.readouterr().err + assert "index 1" in err, err + assert "no exit code" in err, err + assert "None" not in err, err + assert _claims(repo) == [] + + +# --- 2. the happy path is still the happy path ------------------------------- + + +def test_a_passing_last_event_binds_to_it(repo: Path, capsys): + s = _session(repo) + run_wrapped(FAIL, s, repo) + run_wrapped(PASS, s, repo) + capsys.readouterr() + + assert _claim(repo, "tests-pass", "--label", "unit") == 0 + (claim,) = _claims(repo) + assert claim.event_indices == (1,), "binds the LAST event, not the first pass" + assert capsys.readouterr().err == "", "the happy path warns about nothing" + + +def test_the_single_event_case_binds_event_zero(repo: Path): + """The documented one-run flow, unchanged: run something, claim it.""" + s = _session(repo) + run_wrapped(PASS, s, repo) + assert _claim(repo, "tests-pass") == 0 + (claim,) = _claims(repo) + assert claim.event_indices == (0,) + + +# --- 3. an explicit older binding warns but succeeds ------------------------- + + +def test_explicit_event_zero_of_four_warns_and_still_writes_the_claim( + repo: Path, capsys +): + """`--event` is a deliberate act, so it succeeds — but not silently. + + 197 of 846 claims in the corpus this tool was built against bound a + non-terminal event. Whether any of those gaps spanned a FAILURE is unknown + (it needs a read of archived logs). The warning is what makes the next one + visible at the moment it is created. + """ + s = _session(repo) + for _ in range(4): + run_wrapped(PASS, s, repo) + assert len(s.events()) == 4 + capsys.readouterr() + + assert _claim(repo, "tests-pass", "--event", "0") == 0 + err = capsys.readouterr().err + assert "event 0" in err, err + assert "3 event(s) before the most recent" in err, err + (claim,) = _claims(repo) + assert claim.event_indices == (0,), "the claim is written, not refused" + + +def test_explicit_binding_to_the_most_recent_event_does_not_warn(repo: Path, capsys): + """There is no gap, so there is nothing to say. A warning on every explicit + `--event` would be noise, and noise is how a real warning gets filtered.""" + s = _session(repo) + for _ in range(3): + run_wrapped(PASS, s, repo) + capsys.readouterr() + assert _claim(repo, "tests-pass", "--event", "2") == 0 + assert capsys.readouterr().err == "" + + +def test_an_explicit_binding_to_a_failing_event_is_not_refused_here( + repo: Path, capsys +): + """The refusal is about the DEFAULT, not about `--event`. + + Binding a claim to an event that failed is allowed and is not a lie: the + grading ladder witnesses the failure and grades it `failed`, which is the + honest record. Refusing at declare time would hide a failure instead of + recording one. + """ + s = _session(repo) + run_wrapped(FAIL, s, repo) + run_wrapped(PASS, s, repo) + capsys.readouterr() + assert _claim(repo, "tests-pass", "--event", "0") == 0 + (claim,) = _claims(repo) + assert claim.event_indices == (0,) + + +# --- 4. no events at all ----------------------------------------------------- + + +def test_no_events_at_all_keeps_the_existing_refusal_verbatim(repo: Path, capsys): + """Asserted verbatim: docs/COMPAT.md quotes this sentence.""" + capsys.readouterr() + assert _claim(repo, "tests-pass") == 2 + assert capsys.readouterr().err == ( + "didrun claim: no recorded events to bind to (run something first)\n" + ) + + +# --- 5. declared_at_index is untouched in every path ------------------------- + + +@pytest.mark.parametrize( + "extra,events,expected_indices", + [ + ([], 3, (2,)), + (["--event", "0"], 3, (0,)), + (["--event", "1"], 2, (1,)), + ], + ids=["default", "explicit-older", "explicit-latest"], +) +def test_declared_at_index_is_always_the_last_index( + repo: Path, extra, events, expected_indices +): + """The retroactive-binding rule reads this field, and P5.2 changes which + event is BOUND, never where the declaration sits. If this drifted, a claim + bound to the last event would start grading `unknown`. + """ + s = _session(repo) + for _ in range(events): + run_wrapped(PASS, s, repo) + assert _claim(repo, "tests-pass", *extra) == 0 + (claim,) = _claims(repo) + assert claim.event_indices == expected_indices + assert claim.declared_at_index == events - 1 + + +# --- 6. end to end through the real binary ----------------------------------- + +_CLI_BOOT = "import sys; from didrun.cli import main; sys.exit(main(sys.argv[1:]))" + + +def test_the_refusal_holds_in_a_subprocess_too(repo: Path): + """In-process `cli.main` shares this interpreter's state; a separate process + is the shape an agent actually invokes, and the exit code is what a shell + gate reads.""" + for argv in ([*PASS], [*FAIL]): + subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "run", "--", *argv], + cwd=str(repo), + capture_output=True, + ) + proc = subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "claim", "tests-pass"], + cwd=str(repo), + capture_output=True, + text=True, + ) + assert proc.returncode == 2, proc.stderr + assert "exited 3" in proc.stderr + assert _claims(repo) == [] diff --git a/tests/test_claim_supersession.py b/tests/test_claim_supersession.py new file mode 100644 index 0000000..b443f1d --- /dev/null +++ b/tests/test_claim_supersession.py @@ -0,0 +1,625 @@ +"""P4.1 — supersede a claim inside the window instead of truncating the record. + +The pair these pin. Within one seal window a single `failed` or `stale` claim +made `all_verified` false with no in-window way to retire it, and the only two +escapes were seal-early (needs a commit) and re-seal — which force-replaced the +commit's note with `all_claims[watermark:]` and silently destroyed everything +the earlier note held. Discarding the ledger and starting over was the rational +third option, which is what the observed atomicity wall was. + +So: a later claim with the same `(ctype, label)` retires an earlier one inside +the window, both stay in the note, and a narrowing re-seal is refused unless the +operator asks for it. The retirement subtracts from the VERDICT, never from the +RECORD — every test here that asserts a green verdict also asserts the retired +claim is still present, with its original grade and reason. + +What this does NOT do: prevent claim-until-green. Every attempt is retained and +counted, which makes it visible. That is the honest bound and it is strictly +better than the invisible truncation it replaces. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import Claim, grade +from didrun.ledger import Session +from didrun import gitplumbing as gp + + +def _git(repo: Path, *a: str) -> None: + subprocess.run(["git", *a], cwd=str(repo), capture_output=True, text=True, check=True) + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _run(session: Session, repo: Path, code: str): + return run_wrapped([sys.executable, "-c", code], session, repo) + + +def _claim(session: Session, ctype: str, label: str, event_index: int) -> Claim: + """Declare a claim exactly as the CLI would: one event, declared at the tail.""" + claim = Claim( + ctype=ctype, + label=label, + event_indices=(event_index,), + declared_at_index=len(session.events()) - 1, + ) + M.declare_claim(session, claim) + return claim + + +def _note_json(repo: Path, commit: str = "HEAD") -> dict: + """The note body on ``commit``, parsed — read with raw git, not didrun.""" + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + assert proc.returncode == 0, "the commit carries no note" + body = proc.stdout + assert body.endswith(b"\n") + return json.loads(body[:-1]) + + +def _attach_raw_note(repo: Path, commit: str, body: bytes) -> None: + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-F", "-", commit], + cwd=str(repo), + input=body, + capture_output=True, + check=True, + ) + + +def _gate_pair(session: Session, repo: Path, tmp_path: Path) -> None: + """One gate run twice in one window: fails, is fixed, passes. No commit. + + The fix lives OUTSIDE the repo on purpose. A fix that edited a tracked file + would move the working tree away from HEAD's tree, and the second claim would + grade `stale` for that reason rather than for anything to do with + supersession — the test would then be measuring the tree digest. This keeps + the tree fixed so the only variable is the two attempts at the same gate. + """ + flag = tmp_path / "gate-fixed" + code = f"import os, sys; sys.exit(0 if os.path.exists({str(flag)!r}) else 1)" + _run(session, repo, code) # event 0 — the gate fails + _claim(session, "tests-pass", "t", 0) + flag.write_text("fixed\n", encoding="ascii") + _run(session, repo, code) # event 1 — the same gate passes + _claim(session, "tests-pass", "t", 1) + + +# --- test 1: the commit-free checkpoint -------------------------------------- + + +def test_a_later_claim_retires_an_earlier_one_with_no_commit(repo: Path, tmp_path: Path): + """The point of the phase: the gate opens without an intervening commit, and + the failed attempt is still in the note.""" + s = _session(repo) + _gate_pair(s, repo, tmp_path) + head = gp.head_commit(repo) + + m = M.seal(s, repo) + + assert len(m.claims) == 2, "the record must keep both attempts" + assert m.claims[0]["grade"] == "failed" + assert m.claims[0]["superseded_by"] == 1 + assert m.claims[1]["grade"] == "tree-exact" + assert "superseded_by" not in m.claims[1], "the live claim must carry no mark" + assert (m.claims_from, m.claims_to) == (0, 2) + + report = M.verify(s, repo) + assert report.all_verified is True, "the gate must open with no commit" + assert report.worst_status == "tree-exact", "the retired FAILED must not vote" + assert report.superseded_count == 1 + assert len(report.results) == 2 + assert [r.grade for r in report.live_results] == ["tree-exact"] + assert gp.head_commit(repo) == head, "no commit may be needed to converge" + + # The CI gate, not just the report object. + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_the_verdict_line_says_how_many_were_retired(repo: Path, tmp_path: Path, monkeypatch): + """A green headline over a retained FAILED row has to explain itself.""" + monkeypatch.setenv("NO_COLOR", "1") + s = _session(repo) + _gate_pair(s, repo, tmp_path) + M.seal(s, repo) + + out = render.render_verdict(M.verify(s, repo)) + assert "ALL RECORDED-EXACT" in out + assert "1/1 claims recorded-exact" in out, "the denominator must be the live set" + assert "1 claim superseded within this seal window" in out + # The retired row is still shown, and it says which claim replaced it. + assert "FAILED" in out + assert "superseded by claim #1 (record, not verdict)" in out + # And it is below the live row: the record explains, the verdict is asked about. + lines = out.splitlines() + live = next(i for i, ln in enumerate(lines) if "TREE-EXACT" in ln) + retired = next(i for i, ln in enumerate(lines) if "FAILED" in ln) + assert live < retired + + +def test_three_attempts_all_point_at_the_one_that_stands(repo: Path, tmp_path: Path): + """`superseded_by` names the LIVE entry, not the immediate successor.""" + s = _session(repo) + _gate_pair(s, repo, tmp_path) + _run(s, repo, "import sys; sys.exit(0)") + _claim(s, "tests-pass", "t", 2) + + m = M.seal(s, repo) + + assert len(m.claims) == 3 + assert m.claims[0]["superseded_by"] == 2 + assert m.claims[1]["superseded_by"] == 2 + assert "superseded_by" not in m.claims[2] + assert M.verify(s, repo).superseded_count == 2 + assert M.verify(s, repo).all_verified is True + + +# --- test 2: identity is (ctype, label), and nothing wider ------------------- + + +def test_a_different_label_or_ctype_does_not_supersede(repo: Path): + """Two gates are two gates. Supersession is same-type AND same-label only.""" + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "unit", 0) + _claim(s, "tests-pass", "integration", 0) + _claim(s, "lint-clean", "unit", 0) + + m = M.seal(s, repo) + + assert len(m.claims) == 3 + assert all("superseded_by" not in c for c in m.claims) + report = M.verify(s, repo) + assert report.superseded_count == 0 + assert len(report.live_results) == 3 + assert report.all_verified is True + + +# --- test 3: the record grows, never shrinks --------------------------------- + + +def test_the_retired_entry_keeps_its_grade_and_reason_verbatim(repo: Path, tmp_path: Path): + """Nothing is relabelled, re-graded or dropped — checked against an + independent grading of the same claim, not against a string in this file.""" + s = _session(repo) + _gate_pair(s, repo, tmp_path) + + tree = gp.commit_tree(repo, "HEAD") + declared = M._load_claims(s) + independent = grade(declared[0], tree, s.events(), repo, s.blobs.root) + + m = M.seal(s, repo) + retired = m.claims[0] + + assert retired["grade"] == independent.grade == "failed" + assert retired["reason"] == independent.reason + assert retired["exit_code"] == independent.exit_code == 1 + assert retired["claim"]["label"] == "t", "the label is not rewritten either" + assert retired["supporting_event_index"] == 0 + + # And verify hands the same values back, marked rather than altered. + result = M.verify(s, repo).results[0] + assert result.grade == "failed" + assert result.reason == independent.reason + assert result.is_superseded and result.superseded_by == 1 + + +def test_a_retired_stale_claim_keeps_its_delta(repo: Path): + """`stale` must carry its delta wherever it appears, retired included.""" + s = _session(repo) + _run(s, repo, "print(1)") # event 0, at the old tree + _claim(s, "tests-pass", "t", 0) + (repo / "calc.py").write_text("def add(a, b):\n return a + b # edited\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit") + _run(s, repo, "print(1)") # event 1, at the new tree + _claim(s, "tests-pass", "t", 1) + + m = M.seal(s, repo) + + assert m.claims[0]["grade"] == "stale" + assert m.claims[0]["superseded_by"] == 1 + assert [c["path"] for c in m.claims[0]["delta"]] == ["calc.py"] + assert m.claims[1]["grade"] == "tree-exact" + assert M.verify(s, repo).all_verified is True + + +# --- test 4: the window is in the artifact ----------------------------------- + + +def test_the_note_records_the_claim_window(repo: Path): + """claims_from/claims_to match the watermark arithmetic on every seal.""" + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "one", 0) + first_commit = gp.head_commit(repo) + M.seal(s, repo) + + body = _note_json(repo, first_commit) + assert body["claims_from"] == 0 + assert body["claims_to"] == 1 + assert len(body["claims"]) == 1 + + (repo / "calc.py").write_text("def add(a, b):\n return a + b # two\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "unit two") + _run(s, repo, "print(2)") + _claim(s, "tests-pass", "two", 1) + second = M.seal(s, repo) + + assert (second.claims_from, second.claims_to) == (1, 2) + body = _note_json(repo, gp.head_commit(repo)) + assert (body["claims_from"], body["claims_to"]) == (1, 2) + assert len(body["claims"]) == 1, "the window is the note's scope, verbatim" + + # The earlier commit's note is untouched: history is never rewritten. + assert _note_json(repo, first_commit)["claims_from"] == 0 + assert _note_json(repo, first_commit)["claims_to"] == 1 + + +def test_a_v1_note_reads_back_with_the_window_defaults(): + """Additive with a reading default, so every stored note still parses.""" + body = json.dumps( + { + "version": 1, + "commit": "0" * 40, + "tree": "1" * 40, + "claims": [{"grade": "tree-exact"}, {"grade": "tree-exact"}], + "coverage": {"total_events": 2, "by_coverage": {"complete": 2}}, + "secrets_override": False, + } + ).encode("ascii") + + m = M.Manifest.from_json(body) + + assert m.claims_from == 0 + assert m.claims_to == 2, "the default is len(claims), never None" + assert json.loads(m.to_json())["claims_to"] == 2 + + +# --- test 5: a narrowing re-seal is refused ---------------------------------- + + +def _ten_gates(repo: Path) -> Session: + s = _session(repo) + _run(s, repo, "print(1)") + for i in range(10): + _claim(s, "tests-pass", f"gate-{i}", 0) + return s + + +def test_a_narrowing_reseal_is_refused_and_names_both_windows(repo: Path): + s = _ten_gates(repo) + M.seal(s, repo) + assert _note_json(repo)["claims_to"] == 10 + + _claim(s, "tests-pass", "gate-10", 0) + _claim(s, "tests-pass", "gate-11", 0) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + message = str(exc.value) + assert "[0,10)" in message, "the message must name the recorded window" + assert "[10,12)" in message, "and the window that would replace it" + assert "--reseal" in message + # Refused means refused: the note and the watermark are where they were. + assert _note_json(repo)["claims_to"] == 10 + assert M._last_seal_watermark(s) == 10 + + m = M.seal(s, repo, reseal=True) + assert (m.claims_from, m.claims_to) == (10, 12) + assert _note_json(repo)["claims_from"] == 10, "the narrowing is now visible" + assert len(_note_json(repo)["claims"]) == 2 + + +def test_a_superset_reseal_needs_no_flag(repo: Path): + """Re-covering the recorded window plus more loses nothing, so it is allowed. + + The realistic way to land here is watermark rotation — the gitignored + seals.jsonl is gone, so the seal covers the whole claim history again. + """ + s = _ten_gates(repo) + M.seal(s, repo) + _claim(s, "tests-pass", "gate-10", 0) + (s.root / "seals.jsonl").unlink() + + m = M.seal(s, repo) + + assert (m.claims_from, m.claims_to) == (0, 11) + assert len(_note_json(repo)["claims"]) == 11 + + +def test_the_seal_line_counts_the_same_set_the_verdict_does(repo: Path, tmp_path: Path, capsys): + """`seal` and `verify` describe the same note, so "1/2 recorded-exact" beside + an ALL RECORDED-EXACT verdict is one of them being wrong.""" + s = _session(repo) + _gate_pair(s, repo, tmp_path) + capsys.readouterr() + + assert cli.main(["--repo", str(repo), "seal"]) == 0 + + out = capsys.readouterr().out + assert "1/1 claims recorded-exact" in out + assert "1 superseded" in out + assert "1/2" not in out + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_the_cli_wires_reseal_and_refuses_without_it(repo: Path, capsys): + s = _ten_gates(repo) + assert cli.main(["--repo", str(repo), "seal"]) == 0 + capsys.readouterr() + _claim(s, "tests-pass", "gate-10", 0) + + assert cli.main(["--repo", str(repo), "seal"]) == 2 + captured = capsys.readouterr() + assert "refusing to replace the note" in captured.err + assert "--reseal" in captured.err + assert "Traceback" not in captured.err + + assert cli.main(["--repo", str(repo), "seal", "--reseal"]) == 0 + assert (_note_json(repo)["claims_from"], _note_json(repo)["claims_to"]) == (10, 11) + + +# --- test 6: a legacy note carries no window, so it is not overwritten blind -- + + +def test_a_legacy_note_requires_reseal(repo: Path): + """An absent window is UNKNOWN, not [0,N): the reading default must never + let the guard conclude a legacy note is containable.""" + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "t", 0) + head = gp.head_commit(repo) + legacy = json.dumps( + { + "version": 1, + "commit": head, + "tree": gp.commit_tree(repo, head), + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + "secrets_override": False, + } + ).encode("ascii") + _attach_raw_note(repo, head, legacy) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + message = str(exc.value) + assert "records no claim window" in message + assert "--reseal" in message + assert "[0,1)" in message, "the seal's own window is still named" + + m = M.seal(s, repo, reseal=True) + assert (m.claims_from, m.claims_to) == (0, 1) + + +def test_a_foreign_note_is_not_overwritten_blind(repo: Path): + """A body that is not a manifest at all reads as no window, for the same + reason: it is the case where an overwrite destroys the most.""" + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "t", 0) + _attach_raw_note(repo, gp.head_commit(repo), b"someone else's note body") + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + assert "records no claim window" in str(exc.value) + assert M.seal(s, repo, reseal=True).claims_to == 1 + + +# --- test 7: a missing watermark file is never silent ------------------------ + + +def test_a_missing_watermark_file_warns_and_proceeds(repo: Path, capsys): + s = _ten_gates(repo) + M.seal(s, repo) + capsys.readouterr() + seals = s.root / "seals.jsonl" + assert seals.exists() + seals.unlink() + + m = M.seal(s, repo) + + err = capsys.readouterr().err + assert "no seal watermark file" in err + assert "WHOLE claim history" in err + assert "10 declared claim(s)" in err + assert m.claims_from == 0, "the watermark really did reset" + assert seals.exists(), "and the seal proceeded" + + +def test_the_warning_is_silent_once_the_watermark_exists(repo: Path, capsys): + """It fires on absence, not on every seal — otherwise it is ignorable noise. + + The first seal of a session also has no watermark file and also warns: the + two are genuinely indistinguishable from inside `seal`, and the warning text + says so rather than picking one. + """ + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "one", 0) + M.seal(s, repo) + assert "no seal watermark file" in capsys.readouterr().err + + _git(repo, "commit", "-q", "--allow-empty", "-m", "unit two") + _claim(s, "tests-pass", "two", 0) + M.seal(s, repo) + assert "no seal watermark file" not in capsys.readouterr().err + + +def test_no_claims_no_watermark_warning(repo: Path, capsys): + """Nothing to re-cover, nothing to warn about.""" + s = _session(repo) + _run(s, repo, "print(1)") + M.seal(s, repo) + assert "no seal watermark file" not in capsys.readouterr().err + + +# --- test 8: supersession is window-local, so the documented loop is unchanged + + +def test_the_same_label_in_a_later_window_is_not_superseded(repo: Path): + """What test_fix_loop_converges encodes, asserted directly: the watermark + advanced between the two seals, so the second window holds one claim and + nothing in the first is reachable from it.""" + s = _session(repo) + _run(s, repo, "import sys; sys.exit(1)") + _claim(s, "tests-pass", "t", 0) + M.seal(s, repo) + first_commit = gp.head_commit(repo) + + (repo / "calc.py").write_text("def add(a, b):\n return a + b # fixed\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "fix") + _run(s, repo, "import sys; sys.exit(0)") + _claim(s, "tests-pass", "t", 1) + second = M.seal(s, repo) + + assert (second.claims_from, second.claims_to) == (1, 2) + assert len(second.claims) == 1 + assert all("superseded_by" not in c for c in second.claims) + report = M.verify(s, repo) + assert report.all_verified is True + assert len(report.results) == 1 + assert report.superseded_count == 0 + + # History is never rewritten: the first commit's note still shows FAILED and + # nothing retired it from the later window. + old = M.verify(s, repo, commitish=first_commit) + assert old.worst_status == "failed" + assert old.superseded_count == 0 + + +# --- the vacuous-green guard ------------------------------------------------ + + +def _result(label: str, grade_name: str, superseded_by=None): + from didrun.claims import GradeResult + + return GradeResult( + Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0), + grade_name, + reason=f"{grade_name} reason", + supporting_event_index=0, + superseded_by=superseded_by, + ) + + +def test_a_window_of_only_superseded_results_is_never_green(): + """`all([])` is True, so filtering the superseded out of `all_verified` + without re-checking non-emptiness would make a window whose every claim was + retired verify vacuously green — a green verdict over zero live claims.""" + report = M.VerifyReport( + commit="a" * 40, + tree="b" * 40, + resolved_by="commit", + results=[ + _result("t", "tree-exact", superseded_by=1), + _result("t", "tree-exact", superseded_by=1), + ], + coverage={}, + chain_status="intact", + ) + + assert report.live_results == [] + assert report.superseded_count == 2 + assert report.all_verified is False + # And it must not read as an unsealed commit either: there IS a record here. + assert report.worst_status == "unknown" + assert report.total == 2 + out = render.render_verdict(report) + assert "ALL RECORDED-EXACT" not in out + assert "NO CLAIMS" not in out + assert "2 claims superseded" in out + + +def test_an_empty_report_still_reads_as_empty(): + """The complement: no results at all is `empty`, which is a different fact.""" + report = M.VerifyReport( + commit="a" * 40, tree="b" * 40, resolved_by="none", results=[], coverage={} + ) + assert report.worst_status == "empty" + assert report.all_verified is False + assert report.superseded_count == 0 + + +@pytest.mark.parametrize( + "mark", + [ + 0, # itself + -1, # backwards + 2, # out of range + True, # a bool is not an index + "1", # not an int + None, # explicitly null + ], +) +def test_an_unusable_supersession_mark_counts_the_claim(repo: Path, mark): + """A mark that is not a forward index into this note's own claims is dropped, + and dropping it COUNTS the claim — the direction that refuses. + + This is not a forgery barrier and does not pretend to be one: whoever can + rewrite a note can rewrite a grade directly. It is the same fail-closed + reading didrun applies to every field it did not compute in this process. + """ + s = _session(repo) + _run(s, repo, "import sys; sys.exit(1)") + _claim(s, "tests-pass", "t", 0) + _run(s, repo, "import sys; sys.exit(0)") + _claim(s, "tests-pass", "t", 1) + m = M.seal(s, repo) + assert M.verify(s, repo).all_verified is True # the honest note is green + + body = _note_json(repo) + body["claims"][0]["superseded_by"] = mark + _attach_raw_note( + repo, + m.commit, + json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii"), + ) + + report = M.verify(s, repo) + assert report.superseded_count == 0, "an unusable mark must not retire a claim" + assert report.all_verified is False + assert report.worst_status == "failed" + + +def test_a_mark_pointing_at_a_different_gate_is_dropped(repo: Path): + """Supersession is defined by (ctype, label), so a mark across two different + gates is malformed — it would retire a claim nothing replaced.""" + s = _session(repo) + _run(s, repo, "import sys; sys.exit(1)") + _claim(s, "tests-pass", "unit", 0) + _run(s, repo, "import sys; sys.exit(0)") + _claim(s, "tests-pass", "integration", 1) + m = M.seal(s, repo) + + body = _note_json(repo) + assert all("superseded_by" not in c for c in body["claims"]) + body["claims"][0]["superseded_by"] = 1 + _attach_raw_note( + repo, + m.commit, + json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii"), + ) + + report = M.verify(s, repo) + assert report.superseded_count == 0 + assert report.all_verified is False diff --git a/tests/test_conjunction_claim.py b/tests/test_conjunction_claim.py new file mode 100644 index 0000000..4e14e4c --- /dev/null +++ b/tests/test_conjunction_claim.py @@ -0,0 +1,628 @@ +"""P4.2 — a conjunction claim, graded as the worst of its conjuncts. + +The property an 80-command proof ledger was hand-enforcing — "all N of these +gates passed against this tree" — had no representation in the tool. This gives +it one, and the whole design rests on a single negative: a conjunction must +never produce a grade its members do not already back. + +Test 1 is that negative, over every combination, and it is the gate. If any +combination comes out better than the worst of its pair, the design is weakening +a claim and the design review's refusal to substitute a marker for a witnessed +execution still stands. + +Test 2 is the one that fails if the verify-side resolution pass is missing: a +sealed conjunction regraded with no siblings resolves every conjunct to "no +match" and verifies `unknown`, so a note that sealed green fails --strict. It +proves the resolution reads the NOTE's own results by deleting claims.jsonl and +verifying again. +""" + +from __future__ import annotations + +import itertools +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import ( + CONJUNCTION, + GRADE_ORDER_WORST_FIRST, + Claim, + ClaimError, + GradeResult, + grade, +) +from didrun.ledger import Session +from didrun import gitplumbing as gp + +# tests/compat's leg 2 owns the note round-trip criterion. Test 6 imports it +# rather than restating it: a second copy of a three-part criterion is a second +# chance to write a weaker version of it, and the pack names that leg as the +# compat gate for exactly this reason. +from compat.test_corpus_replay import note_violations + +# The six grades a conjunct can carry, written out rather than imported, so the +# matrix below is a statement about the vocabulary and not a restatement of +# whatever the vocabulary happens to be. The equality against the severity order +# is what turns a new grade into a deliberate edit here. +GRADES = ( + "tree-exact", + "scope-exact", + "stale", + "unknown", + "failed", + "witness-unavailable", +) + + +def test_the_matrix_covers_every_grade_the_order_knows(): + """A grade added to the ladder must be added to test 1's matrix.""" + assert set(GRADES) == set(GRADE_ORDER_WORST_FIRST) + + +# --- helpers ----------------------------------------------------------------- + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _run(session: Session, repo: Path, code: str): + return run_wrapped([sys.executable, "-c", code], session, repo) + + +def _claim(session: Session, ctype: str, label: str, event_index: int) -> Claim: + """Declare an ordinary claim exactly as the CLI would.""" + claim = Claim( + ctype=ctype, + label=label, + event_indices=(event_index,), + declared_at_index=len(session.events()) - 1, + ) + M.declare_claim(session, claim) + return claim + + +def _conjunction(session: Session, label: str, *conjuncts: str) -> Claim: + claim = Claim( + ctype=CONJUNCTION, + label=label, + event_indices=(), + declared_at_index=len(session.events()) - 1, + conjuncts=tuple(conjuncts), + ) + M.declare_claim(session, claim) + return claim + + +def _result(label: str, grade_value: str) -> GradeResult: + """One graded sibling, at an arbitrary grade. Not produced by the ladder — + the point is to reach grade combinations no single fixture repo could.""" + return GradeResult( + claim=Claim( + ctype="tests-pass", + label=label, + event_indices=(0,), + declared_at_index=0, + ), + grade=grade_value, + reason="fixture sibling", + ) + + +def _report(*results: GradeResult) -> M.VerifyReport: + """A report over ``results`` alone, so `worst_status` is the only authority + this file consults for which of two grades is worse.""" + return M.VerifyReport( + commit="0" * 40, tree="1" * 40, resolved_by="commit", + results=list(results), coverage={}, + ) + + +def _graded_conjunction(*siblings: GradeResult, names=None) -> GradeResult: + conjuncts = names if names is not None else tuple(s.claim.label for s in siblings) + claim = Claim( + ctype=CONJUNCTION, + label="REL", + event_indices=(), + declared_at_index=9, + conjuncts=tuple(conjuncts), + ) + return grade(claim, "1" * 40, [], Path("."), None, siblings=list(siblings)) + + +def _note_body(repo: Path, commit: str = "HEAD") -> bytes: + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + assert proc.returncode == 0, "the commit carries no note" + return proc.stdout + + +def _note_json(repo: Path, commit: str = "HEAD") -> dict: + body = _note_body(repo, commit) + assert body.endswith(b"\n") + return json.loads(body[:-1]) + + +def _entry(body: dict, label: str) -> dict: + (entry,) = [c for c in body["claims"] if c["claim"]["label"] == label] + return entry + + +# --- test 1: never better than the worst (THE criterion) --------------------- + + +@pytest.mark.parametrize("left,right", list(itertools.product(GRADES, GRADES))) +def test_a_conjunction_is_never_better_than_its_worst_conjunct(left: str, right: str): + """The gate: over EVERY pair of grades, in both orders. + + Three assertions, because "equals the worst" alone is satisfiable by an + implementation that invents a grade nobody claimed: + + 1. it equals what `worst_status` says about the two conjuncts together — + the same order the verdict line above it is computed with; + 2. it is one of the two grades its conjuncts actually carry, never a + third one; + 3. adding either conjunct back alongside the conjunction does not make the + verdict worse, which is "the conjunction is already at least as bad as + each of them" stated without reusing the implementation's own helper. + """ + a, b = _result("a", left), _result("b", right) + result = _graded_conjunction(a, b) + + expected = _report(a, b).worst_status + assert result.grade == expected, ( + f"conjunction over ({left}, {right}) graded {result.grade}, " + f"but the worst of them is {expected}" + ) + assert result.grade in (left, right), "a conjunction invented a grade" + conj_only = _report(result).worst_status + assert _report(result, a).worst_status == conj_only + assert _report(result, b).worst_status == conj_only + # The reason names the conjunct the grade came from, and its grade. + named = "a" if expected == left else "b" + assert repr(named) in result.reason + assert expected in result.reason + # The per-conjunct pairs travel with the result, in declaration order. + assert result.conjunct_grades == (("a", left), ("b", right)) + + +def test_a_tie_names_the_first_declared_conjunct(): + """Determinism: which of two equally-bad members is named is the operator's + order, never an iteration artefact.""" + first = _graded_conjunction(_result("a", "stale"), _result("b", "stale")) + reversed_ = _graded_conjunction(_result("b", "stale"), _result("a", "stale")) + assert "'a'" in first.reason + assert "'b'" in reversed_.reason + + +def test_a_conjunction_over_one_conjunct_is_that_conjunct_s_grade(): + for g in GRADES: + result = _graded_conjunction(_result("only", g)) + assert result.grade == g + assert result.conjunct_grades == (("only", g),) + + +# --- test 2: an all-green conjunction verifies, from the note alone ---------- + + +def _three_green_gates(s: Session, repo: Path) -> None: + """Three self-stable commands, one claim each, plus a conjunction of them.""" + for i, label in enumerate(("unit", "lint", "integration")): + _run(s, repo, "pass") + _claim(s, "tests-pass", label, i) + _conjunction(s, "REL", "unit", "lint", "integration") + + +def test_an_all_green_conjunction_seals_and_verifies_tree_exact(repo: Path): + s = _session(repo) + _three_green_gates(s, repo) + + m = M.seal(s, repo) + body = _note_json(repo) + assert _entry(body, "REL")["grade"] == "tree-exact" + assert [c["grade"] for c in m.claims] == ["tree-exact"] * 4 + + report = M.verify(s, repo) + by_label = {r.claim.label: r for r in report.results} + assert by_label["REL"].grade == "tree-exact" + assert by_label["REL"].conjunct_grades == ( + ("unit", "tree-exact"), + ("lint", "tree-exact"), + ("integration", "tree-exact"), + ) + assert report.all_verified is True + assert report.worst_status == "tree-exact" + # A conjunction binds no event, so it is not evidence-bound and must not be + # counted as though it were. + assert report.evidence_bound_count == 3 + assert report.total == 4 + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_the_conjunction_resolves_against_the_note_not_the_live_claims_file(repo: Path): + """Delete claims.jsonl; the grade must not move. + + This is the test that fails when verify grades conjunctions in one pass with + no siblings — and the one that fails again if a later change reaches for the + live claims file to get them. `verify` reads the manifest and the repo, never + the session's declared claims: a conjunction resolved against claims.jsonl + would let a claim declared after the seal change an old commit's verdict. + """ + s = _session(repo) + _three_green_gates(s, repo) + M.seal(s, repo) + + before = {r.claim.label: r.grade for r in M.verify(s, repo).results} + claims_file = repo / ".didrun" / "claims.jsonl" + assert claims_file.exists() + claims_file.unlink() + + after_report = M.verify(s, repo) + after = {r.claim.label: r.grade for r in after_report.results} + assert after == before + assert after["REL"] == "tree-exact" + assert after_report.all_verified is True + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_a_conjunction_graded_with_no_siblings_is_unknown_never_green(repo: Path): + """What the second pass buys, stated as an assertion. + + Grading a conjunction without its siblings — one pass over the note, which + is what verify did before this unit — resolves every conjunct to no match. + The result is `unknown`, so the failure mode of forgetting the pass is a + refusal, never a green verdict over members nobody looked at. + """ + s = _session(repo) + _three_green_gates(s, repo) + tree = gp.commit_tree(repo, "HEAD") + conjunction = M._load_claims(s)[-1] + assert conjunction.ctype == CONJUNCTION + + unbound = grade(conjunction, tree, s.events(), repo, s.blobs.root) + assert unbound.grade == "unknown" + assert "unit" in unbound.reason + + +# --- test 3: an unresolvable conjunct is unknown, never tree-exact ---------- + + +def test_a_missing_conjunct_grades_unknown_and_names_it(repo: Path): + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "unit", "nosuch") + + m = M.seal(s, repo) + entry = _entry(json.loads(m.to_json()), "REL") + assert entry["grade"] == "unknown" + assert "'nosuch'" in entry["reason"] + assert "names no claim" in entry["reason"] + assert entry["conjunct_grades"] == [ + {"label": "unit", "grade": "tree-exact"}, + {"label": "nosuch", "grade": None}, + ] + + report = M.verify(s, repo) + rel = next(r for r in report.results if r.claim.label == "REL") + assert rel.grade == "unknown" + assert "'nosuch'" in rel.reason + assert report.all_verified is False + assert report.worst_status == "unknown" + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +def test_an_ambiguous_conjunct_grades_unknown_and_says_how_many(repo: Path): + """One label, two live claims: a member it cannot identify is a member it + cannot vouch for. Two different ctypes carrying one label are both live — + supersession is keyed on the (ctype, label) pair, so neither retires the + other.""" + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "gate", 0) + _claim(s, "lint-clean", "gate", 0) + _conjunction(s, "REL", "gate") + + m = M.seal(s, repo) + entry = _entry(json.loads(m.to_json()), "REL") + assert entry["grade"] == "unknown" + assert "'gate' names 2 claims" in entry["reason"] + assert M.verify(s, repo).all_verified is False + + +def test_a_conjunction_naming_a_conjunction_is_unknown(repo: Path): + """Resolution is one level deep: no recursion to bound, no cycle to detect, + and no grade asserted for a name this code will not follow.""" + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "INNER", "unit") + _conjunction(s, "OUTER", "INNER") + + m = M.seal(s, repo) + body = json.loads(m.to_json()) + assert _entry(body, "INNER")["grade"] == "tree-exact" + outer = _entry(body, "OUTER") + assert outer["grade"] == "unknown" + assert "'INNER' names no claim" in outer["reason"] + + +# --- test 4: superseded conjuncts are skipped ------------------------------- + + +def test_a_conjunction_resolves_to_the_surviving_claim(repo: Path, tmp_path: Path): + """P4.1's retired attempt shares its label with the claim that replaced it. + + The fix lives OUTSIDE the repo so the tree does not move: the only variable + is the two attempts at one gate. + """ + s = _session(repo) + flag = tmp_path / "gate-fixed" + code = f"import os, sys; sys.exit(0 if os.path.exists({str(flag)!r}) else 1)" + _run(s, repo, code) # event 0 — the gate fails + _claim(s, "tests-pass", "gate", 0) + flag.write_text("fixed\n", encoding="ascii") + _run(s, repo, code) # event 1 — the same gate passes + _claim(s, "tests-pass", "gate", 1) + _conjunction(s, "REL", "gate") + head = gp.head_commit(repo) + + m = M.seal(s, repo) + body = json.loads(m.to_json()) + assert m.claims[0]["grade"] == "failed" + assert m.claims[0]["superseded_by"] == 1, "P4.1's mark must be on the note" + assert m.claims[1]["grade"] == "tree-exact" + rel = _entry(body, "REL") + assert rel["grade"] == "tree-exact", "the retired FAILED must not be resolved to" + assert rel["conjunct_grades"] == [{"label": "gate", "grade": "tree-exact"}] + + report = M.verify(s, repo) + live = {r.claim.label: r.grade for r in report.live_results} + assert live == {"gate": "tree-exact", "REL": "tree-exact"} + assert report.superseded_count == 1 + assert report.all_verified is True + assert gp.head_commit(repo) == head, "no commit may be needed to converge" + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + # The record keeps the failed attempt, verbatim, and the row says why the + # green headline stands over it. + text = render.render_verdict(report) + assert "FAILED" in text + assert "superseded by claim #1" in text + + +def test_a_superseded_conjunction_does_not_vote(repo: Path): + """A conjunction is retired by the same (ctype, label) rule as any claim.""" + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "nosuch") # attempt 1: unresolvable + _conjunction(s, "REL", "unit") # attempt 2: the corrected declaration + + m = M.seal(s, repo) + assert m.claims[1]["grade"] == "unknown" + assert m.claims[1]["superseded_by"] == 2 + assert m.claims[2]["grade"] == "tree-exact" + report = M.verify(s, repo) + assert report.all_verified is True + assert report.superseded_count == 1 + + +# --- test 5: vocabulary invariants ------------------------------------------ + + +def test_a_conjunction_with_no_conjuncts_is_refused(): + with pytest.raises(ClaimError) as exc: + Claim(ctype=CONJUNCTION, label="REL", event_indices=()) + assert "conjunction" in str(exc.value) + + +def test_a_non_conjunction_carrying_conjuncts_is_refused(): + with pytest.raises(ClaimError): + Claim( + ctype="tests-pass", + label="t", + event_indices=(0,), + conjuncts=("unit",), + ) + + +def test_a_conjunction_binding_an_event_is_refused(): + """The illegitimate composite: a claim spanning several bound events grades + on the first success it finds and hides a witnessed failure. A conjunction + must not be built out of that shape, so its bound-event set is required to + be empty.""" + with pytest.raises(ClaimError) as exc: + Claim( + ctype=CONJUNCTION, + label="REL", + event_indices=(0,), + conjuncts=("unit",), + ) + assert "binds no event" in str(exc.value) + + +def test_the_cli_refuses_the_flags_a_conjunction_has_no_use_for(repo: Path, capsys): + """Refused, not silently ignored: an operator who passed --event meant it.""" + s = _session(repo) + _run(s, repo, "pass") + + assert cli.main(["--repo", str(repo), "claim", "conjunction", "--of", "unit", "--event", "0"]) == 2 + assert cli.main(["--repo", str(repo), "claim", "conjunction", "--of", "unit", "--path", "src/"]) == 2 + assert cli.main(["--repo", str(repo), "claim", "conjunction"]) == 2 + assert cli.main(["--repo", str(repo), "claim", "tests-pass", "--of", "unit"]) == 2 + assert M._load_claims(s) == [], "no refusal may have declared a claim" + + assert cli.main( + ["--repo", str(repo), "claim", "conjunction", "--label", "REL", "--of", " unit , lint "] + ) == 0 + (declared,) = M._load_claims(s) + assert declared.conjuncts == ("unit", "lint"), "--of is order-preserving and trimmed" + assert declared.event_indices == () + + +def test_the_cli_choices_are_the_vocabulary(): + """The parser's choices used to duplicate CLAIM_TYPES, which is how a CLI + keeps refusing a type the library has accepted for a release.""" + parser = cli.build_parser() + with pytest.raises(SystemExit): + parser.parse_args(["claim", "no-such-type"]) + assert parser.parse_args(["claim", "conjunction"]).type == CONJUNCTION + + +# --- test 6: serialization -------------------------------------------------- + + +def test_the_note_carries_the_conjuncts_and_round_trips(repo: Path): + """The published note says what each member earned, and the body satisfies + the pack's three-part round-trip criterion — newline discipline, key-superset + semantic equality, serialization idempotence — not byte equality.""" + s = _session(repo) + _run(s, repo, "pass") + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _claim(s, "lint-clean", "lint", 1) + _conjunction(s, "REL", "unit", "lint") + M.seal(s, repo) + + raw = _note_body(repo) + assert note_violations(raw, 0) == [] + + body = json.loads(raw[:-1]) + rel = _entry(body, "REL") + assert rel["claim"]["ctype"] == CONJUNCTION + assert rel["claim"]["conjuncts"] == ["unit", "lint"] + assert rel["claim"]["event_indices"] == [] + assert rel["conjunct_grades"] == [ + {"label": "unit", "grade": "tree-exact"}, + {"label": "lint", "grade": "tree-exact"}, + ] + # A conjunction binds no event, so the seal writes it no evidence block: + # "nothing to bind" and "bound to nothing" are different facts. + assert "evidence" not in rel + # Every OTHER claim keeps the keys it always had, and none of them gains a + # conjunct list it did not declare. + for entry in body["claims"]: + if entry["claim"]["label"] != "REL": + assert entry["claim"]["conjuncts"] == [] + assert "conjunct_grades" not in entry + + # The Claim survives the round trip through its own serialization. + reparsed = Claim.from_dict(rel["claim"]) + assert reparsed.ctype == CONJUNCTION + assert reparsed.conjuncts == ("unit", "lint") + + +def test_a_claim_dict_without_conjuncts_reads_as_carrying_none(): + """A v1/v2 claim entry lacks the key entirely; it reads with a default.""" + claim = Claim.from_dict( + {"ctype": "tests-pass", "label": "t", "event_indices": [0], "pathspecs": []} + ) + assert claim.conjuncts == () + assert claim.to_dict()["conjuncts"] == [] + + +def test_a_conjunct_label_carrying_a_secret_is_redacted_everywhere_it_is_published( + repo: Path, +): + """The conjunct list and the reason are exported operator-authored text. + + A token in a claim label blocked the seal and was then published verbatim on + the override — the defect the claim-field redaction pass closed. A + conjunction reintroduces the same shape twice: the conjunct NAME, and the + reason that names it. + """ + token = "ghp_" + "b3TA" * 9 + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", f"unit {token}", 0) + _conjunction(s, "REL", f"unit {token}") + + with pytest.raises(M.redact.SecretsBlocked): + M.seal(s, repo) + m = M.seal(s, repo, allow_secrets=True) + + published = _note_body(repo).decode("ascii") + assert token not in published, "a conjunct name was published verbatim" + rel = _entry(json.loads(m.to_json()), "REL") + mask = M.redact._mask("github-token") + assert rel["claim"]["conjuncts"] == [f"unit {mask}"] + assert rel["conjunct_grades"] == [{"label": f"unit {mask}", "grade": "tree-exact"}] + assert mask in rel["reason"] + fields = {f["field"] for f in rel["redaction"]["fields"]} + assert {"claim.conjuncts[0]", "reason"} <= fields + # The redaction is the same projection on both sides of the match, so the + # conjunction still resolves when verify regrades it from the note. + report = M.verify(s, repo) + assert next(r for r in report.results if r.claim.label == "REL").grade == "tree-exact" + + +# --- test 7: the version bump ---------------------------------------------- + + +def test_a_note_carrying_a_conjunction_is_version_3(repo: Path): + assert M.MANIFEST_VERSION == 3 + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "unit") + m = M.seal(s, repo) + + body = _note_body(repo) + assert json.loads(body[:-1])["version"] == 3 + assert m.version == 3 + # A v0.2 binary whose MANIFEST_VERSION is 3 accepts it. + assert M.Manifest.from_json(body[:-1]).version == 3 + + +def test_the_refuse_unknown_version_gate_rejects_version_4(repo: Path): + """P0.3's refuse-never-coerce rule, one version above this one.""" + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "unit") + M.seal(s, repo) + + body = json.loads(_note_body(repo)[:-1]) + body["version"] = 4 + too_new = json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii") + with pytest.raises(M.ManifestError) as exc: + M.Manifest.from_json(too_new) + assert "version 4" in str(exc.value) + assert "max 3" in str(exc.value) + + +# --- rendering -------------------------------------------------------------- + + +def test_the_surfaces_show_the_conjuncts_and_never_grade_an_unresolved_one( + repo: Path, monkeypatch +): + monkeypatch.setenv("NO_COLOR", "1") + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "unit", "nosuch") + M.seal(s, repo) + report = M.verify(s, repo) + + text = render.render_verdict(report) + assert "of unit: TREE-EXACT" in text + assert "of nosuch: UNRESOLVED" in text + + html = render.render_html(report) + assert "UNRESOLVED" in html + assert "nosuch" in html + # No new grade token, and an unresolved conjunct is not dressed as a grade. + assert "conjuncts" in html diff --git a/tests/test_credential_recall.py b/tests/test_credential_recall.py new file mode 100644 index 0000000..4f33b0e --- /dev/null +++ b/tests/test_credential_recall.py @@ -0,0 +1,399 @@ +"""P3 remediation — the two directions the tokenizer change was measured in. + +The P3.2 work removed `/` from the entropy sweep's token class because an +absolute filesystem path was tokenizing as one 90-character token and scoring +like a credential. That fixed the false positives and was measured. What was +not measured is the other direction: `/` is also 1 of the 64 characters of +base64, so dropping it stopped scoring a real credential class as well as +paths. A base64 secret whose slash-free runs are all shorter than the +24-character floor became invisible — and because replacement is driven off +`scan`'s findings, invisible means UNREDACTED, published verbatim into a note +under a seal line reading "0 findings". + +So this file pins BOTH directions, and each fixture asserts the structural +property that makes it a faithful witness rather than a convenient one: + +* the credential fixtures assert that every one of their `/`-delimited + segments is under the floor, which is exactly the class that went missing — + a fixture with a 24-character segment would pass on segment scoring alone and + prove nothing; +* the path fixtures assert they fired under the pre-P3.2 sweep, so "still + silent" is a statement about a real false positive and not about a string + that was always quiet. + +`_looks_like_path` is what tells the two apart, and it is deliberately biased: +one name-like segment is enough to call a run a path. That direction can only +move a run back to the segment scoring that P3.2 ships, so it cannot introduce +a false positive P3.2 does not already have — measured at 0 new ones over +46,403 real paths — and it pays for that in recall instead. + +Every credential-shaped string here is synthetic and structural. Every path is +synthetic too, for the reason the P3.2 file gives: the strings this was +measured against are a home directory and a private project name, and a test +fixture is a shipped file. +""" + +from __future__ import annotations + +import json +import math +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import manifest as M +from didrun import redact +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + + +# --- the P3.2 sweep, reimplemented so before/after is checkable here ---------- + +# The token class exactly as P3.2 shipped it: `/` removed, so a run is only +# ever scored in segments. Written out rather than imported, so a change to +# redact.py cannot make these before/after assertions agree with it. +_P32_TOKENISH = re.compile(r"[A-Za-z0-9_\-+=]{24,}") +_PRE_P32_TOKENISH = re.compile(r"[A-Za-z0-9_\-+/=]{24,}") +_HEXISH = re.compile(r"^[0-9a-fA-F]+$") + + +def _bits_per_char(s: str) -> float: + counts: dict = {} + for ch in s: + counts[ch] = counts.get(ch, 0) + 1 + n = len(s) + return -sum((c / n) * math.log2(c / n) for c in counts.values()) + + +def _sweep(text: str, tokenizer) -> list: + """The entropy sweep under an arbitrary tokenizer, structured hits removed.""" + covered = [] + for _kind, pat in redact._PATTERNS: + covered.extend(m.span() for m in pat.finditer(text)) + hits = [] + for m in tokenizer.finditer(text): + tok = m.group(0) + if _HEXISH.match(tok): + continue + if any(s <= m.start() < e for s, e in covered): + continue + if _bits_per_char(tok) >= redact._ENTROPY_MIN_BITS_PER_CHAR: + hits.append(m.span()) + return hits + + +def _entropy_hits(text: str) -> list: + return [f for f in redact.scan(text) if f.kind == redact._ENTROPY_KIND] + + +# --- direction 1: base64 credentials with a separator in them ---------------- + +# Synthetic base64-alphabet credentials. Each one carries `/` at an offset that +# chops it into pieces that are ALL under the 24-character floor, which is the +# precise shape that stopped being a candidate at all. +SEPARATED_CREDENTIALS = [ + "Kq7Xm2Bv9Rt4Wn6Zc/8Yd3Fj5Hg1Lp0Qs", + "Zr4Nb8Kw2Vt6Mx1Qy/Jd7Fs3Gh5Lc9Pn0Rv2Bt6Wq", + "Tj5Hn9Cw3Xq7Bz1Mv/Ld4Fp8Gs2Kr6Yt0Nx/Vb3Zc7Qw", +] + + +@pytest.mark.parametrize("cred", SEPARATED_CREDENTIALS) +def test_credential_fixtures_are_faithful_witnesses(cred): + """Every segment under the floor — otherwise the fixture proves nothing. + + A fixture with one 24-character slash-free segment is caught by segment + scoring alone, so it would pass against the very code that lost this + class. The whole run must clear the bar and no segment may reach it. + """ + segments = cred.split("/") + assert len(segments) > 1, "fixture must contain a separator" + assert max(len(s) for s in segments) < redact._ENTROPY_MIN_LEN, ( + "fixture is caught by segment scoring and cannot witness the defect" + ) + assert len(cred) >= redact._ENTROPY_MIN_LEN + assert _bits_per_char(cred) >= redact._ENTROPY_MIN_BITS_PER_CHAR + + +@pytest.mark.parametrize("cred", SEPARATED_CREDENTIALS) +def test_separated_credentials_were_lost_by_p32_and_are_found_again(cred): + """The regression, stated as a before/after on each fixture.""" + assert _sweep(cred, _PRE_P32_TOKENISH), "fixture did not fire before P3.2" + assert not _sweep(cred, _P32_TOKENISH), ( + "fixture is not in the class P3.2 lost, so it cannot witness the fix" + ) + assert _entropy_hits(cred), "the credential class is still invisible" + + +@pytest.mark.parametrize("cred", SEPARATED_CREDENTIALS) +def test_a_found_credential_is_actually_replaced(cred): + """Detection is not the point; removal is. + + `redact` drives replacement off `scan`'s findings, so a class that stops + being a candidate stops being scrubbed too. This asserts the consequence + rather than the mechanism. + """ + result = redact.redact(cred) + assert cred not in result.text + assert result.text == redact._mask(redact._ENTROPY_KIND) + # And the projection still reconstructs, so nothing was mangled. + assert redact.reconstruct(cred, result) == cred + + +def test_a_separated_credential_inside_a_command_line_is_replaced(): + """The shape that actually leaks: a secret sitting in a recorded argv.""" + cred = SEPARATED_CREDENTIALS[1] + text = f"deploy --token {cred} --region eu-west-1" + scrubbed = redact.redact(text).text + assert cred not in scrubbed + assert "deploy --token" in scrubbed and "--region eu-west-1" in scrubbed + + +# --- direction 2: the paths P3.2 silenced stay silent ------------------------ + +# The same class the P3.2 measurement was taken over: ordinary build, cache and +# toolchain paths, every one of them a false positive under the pre-P3.2 sweep. +PATHS_THAT_FIRED_BEFORE_P32 = [ + "/opt/homebrew/Cellar/go/1.24.0/libexec/pkg/tool/darwin_arm64", + "/var/folders/qs/tmpbuild/example-service/cmd/gateway/internal/transport/grpc_handler.go", + "/Users/exampleuser/Library/Caches/build-tool/artifacts/darwin_arm64/release/binary", + "/home/runner/_work/pipeline-Qz4/artifacts/Linux_x64/Release/bin/toolchain", + "/opt/build/Jenkins/workspace/Nightly_x64/out/Release/obj/gen/protoc_wrapper", +] + +# Paths whose segments are CamelCase or capitalised rather than lower case — +# the shape that a naive "a path segment is lower case" rule misreads as a +# credential. They must stay quiet too. +MIXED_CASE_PATHS = [ + "/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks", + "/opt/toolchain/PrivateFrameworks/CoreFoundation/Modules/Darwin/Headers", +] + + +@pytest.mark.parametrize("path", PATHS_THAT_FIRED_BEFORE_P32 + MIXED_CASE_PATHS) +def test_paths_silenced_by_p32_are_still_silent(path): + assert _entropy_hits(path) == [], "a path is being scored as a credential again" + assert redact.redact(path).text == path, "a path was rewritten in an export" + + +@pytest.mark.parametrize("path", PATHS_THAT_FIRED_BEFORE_P32) +def test_path_fixtures_are_faithful_witnesses(path): + """These fired before P3.2, so "still silent" is about a real false positive.""" + assert _sweep(path, _PRE_P32_TOKENISH), ( + "fixture did not fire under the pre-P3.2 sweep and proves nothing" + ) + + +def test_the_separator_is_classified_not_tokenized_away(): + """The actual mechanism, pinned directly. + + P3.2 answered "is this `/` a separator?" in the character class, which + answers it the same way for every string. It is answered per run now: a + path is scored in segments, a base64 blob whole. The segment tokenizer + still admits no separator, which is what keeps a path from re-merging. + """ + assert "/" not in redact._TOKENISH.pattern + assert "/" in redact._RUNISH.pattern + assert redact._looks_like_path(PATHS_THAT_FIRED_BEFORE_P32[0]) + assert redact._looks_like_path(MIXED_CASE_PATHS[0]) + for cred in SEPARATED_CREDENTIALS: + assert not redact._looks_like_path(cred) + + +def test_the_threshold_and_the_length_floor_still_did_not_move(): + """The candidate moved; the test applied to it did not.""" + assert redact._ENTROPY_MIN_BITS_PER_CHAR == 4.2 + assert redact._ENTROPY_MIN_LEN == 24 + + +def test_a_low_entropy_prefix_cannot_hide_a_high_entropy_tail(): + """Whole-run scoring must not become a way to dilute a secret below the bar. + + A long repetitive prefix drags the whole run's per-character entropy under + the threshold. Scoring only the run would miss the tail that segment + scoring catches, so a run that does not clear the bar whole is still + scored in segments. + """ + run = "a" * 40 + "/" + "Kq7Xm2Bv9Rt4Wn6Zc8Yd3Fj5Hg1Lp0Qs" + assert _bits_per_char(run) < redact._ENTROPY_MIN_BITS_PER_CHAR, ( + "fixture does not dilute the run and cannot witness the fallback" + ) + assert _entropy_hits(run), "the high-entropy tail was diluted away" + + +def test_a_git_hash_is_still_not_a_secret(): + """The hex exclusion survives whole-run scoring.""" + assert _entropy_hits("a" * 0 + "0123456789abcdef0123456789abcdef01234567") == [] + + +# --- the exported claim fields are redacted, not just blocked on ------------- + +_GHP = "ghp_" + "G" * 36 + + +def _one_event(session: Session, repo: Path, code: str): + return run_wrapped([sys.executable, "-c", code], session, repo) + + +def _note_text(repo: Path) -> str: + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", "HEAD"], + cwd=str(repo), capture_output=True, text=True, + ) + assert proc.returncode == 0, "no note was published" + return proc.stdout + + +def test_a_secret_in_a_claim_label_is_redacted_in_the_published_note(repo: Path): + """The gate and the publication used to disagree. + + A token in a claim label blocked the seal — and the refusal says "re-run + with --allow-secrets to export a REDACTED artifact anyway". On the + override the label went into refs/notes/didrun verbatim: a ref that gets + pushed. Blocking is not redacting, and the message promised both. + """ + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label=f"suite green with {_GHP}", + event_indices=(0,), + declared_at_index=0, + ), + ) + # Still blocks: this fix does not open the gate, it makes the override honest. + with pytest.raises(redact.SecretsBlocked): + M.seal(s, repo) + + m = M.seal(s, repo, allow_secrets=True) + assert m.secrets_override is True + + published = _note_text(repo) + assert _GHP not in published, "the token was published verbatim" + assert json.loads(published)["claims"][0]["claim"]["label"] == ( + f"suite green with {redact._mask('github-token')}" + ) + + +def test_the_label_redaction_declares_its_projection(repo: Path): + """A marker is plain text; the projection is what a consumer can check.""" + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + label = f"suite green with {_GHP}" + M.declare_claim( + s, + Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo, allow_secrets=True) + fields = m.claims[0]["redaction"]["fields"] + entry = next(f for f in fields if f["field"] == "claim.label") + assert entry["detector_id"] == "github-token" + assert entry["withheld"] is False + # The declared span names the source coordinates that were replaced. + assert label[entry["start"]:entry["end"]] == _GHP + + +def test_a_secret_in_a_pathspec_is_redacted_in_the_published_note(repo: Path): + """Pathspecs are operator-authored too, and they are exported.""" + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label="scoped", + event_indices=(0,), + pathspecs=(f"src/{_GHP}",), + declared_at_index=0, + ), + ) + m = M.seal(s, repo, allow_secrets=True) + published = _note_text(repo) + assert _GHP not in published + assert redact._mask("github-token") in m.claims[0]["claim"]["pathspecs"][0] + + +def test_a_secret_in_a_changed_path_is_redacted_in_the_published_note(repo: Path): + """A changed path is repository-authored text and rides in every stale grade.""" + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="t", event_indices=(0,), declared_at_index=0), + ) + # Move the tree after the evidence, with a filename that carries a secret. + (repo / f"{_GHP}.txt").write_text("x\n") + subprocess.run(["git", "add", "-A"], cwd=str(repo), capture_output=True, check=True) + subprocess.run( + ["git", "commit", "-qm", "add file"], cwd=str(repo), capture_output=True, check=True + ) + + m = M.seal(s, repo, allow_secrets=True) + result = m.claims[0] + assert result["delta"], "fixture did not produce a delta and proves nothing" + published = _note_text(repo) + assert _GHP not in published, "a changed path was published verbatim" + assert any( + redact._mask("github-token") in change["path"] for change in result["delta"] + ) + + +def test_every_exported_claim_string_field_is_scrubbed(repo: Path): + """The invariant, not the four instances. + + Anything in the published claim entry that carries operator- or + repository-authored text goes through the redaction pass. This walks the + serialized note rather than naming fields, so a field added later that + forgets the pass fails here. + """ + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label=f"label {_GHP}", + event_indices=(0,), + pathspecs=(f"spec/{_GHP}",), + declared_at_index=0, + ), + ) + (repo / f"{_GHP}.txt").write_text("x\n") + subprocess.run(["git", "add", "-A"], cwd=str(repo), capture_output=True, check=True) + subprocess.run( + ["git", "commit", "-qm", "add file"], cwd=str(repo), capture_output=True, check=True + ) + M.seal(s, repo, allow_secrets=True) + assert _GHP not in _note_text(repo) + + +def test_the_scan_still_blocks_on_a_field_it_redacts(repo: Path): + """Redacting a field must not quietly stop it blocking. + + Scrubbing the label before the manifest bytes are serialized removes it + from the whole-blob scan. If the field pass did not report what it removed, + this seal would succeed silently — the opposite failure, and just as bad. + """ + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label=f"label {_GHP}", + event_indices=(0,), + declared_at_index=0, + ), + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo) + message = str(exc.value) + assert "github-token" in message + # And it says WHERE, in claim coordinates: a label has no event and no line. + assert "label of claim 0" in message diff --git a/tests/test_detector_tiers.py b/tests/test_detector_tiers.py new file mode 100644 index 0000000..e49585b --- /dev/null +++ b/tests/test_detector_tiers.py @@ -0,0 +1,640 @@ +"""P3.2 — detector tiers, finding locality, and the export-domain block. + +Three things are pinned here, and they pull against each other on purpose. + +1. The tokenizer stops treating a whole filesystem path as one token. The + entropy sweep's token class used to include `/`, and a path accumulates + distinct characters far faster than any one of its segments does, so an + ordinary build path scored like a credential. `_old_entropy_findings` below + is a deliberate reimplementation of the pre-P3.2 sweep, so every "this no + longer fires" assertion is a measured BEFORE/AFTER inside one file rather + than a claim about code that no longer exists. + +2. Nothing in (1) may cost structured recall. The six shape-based detectors are + unchanged and every one of them is pinned firing at `block`, and the entropy + sweep is pinned still firing on a genuinely high-entropy token — it was + demoted to `notice`, never deleted. Deleting it would pass every "paths stop + firing" test in this file while removing the only detector that catches an + unstructured credential. + +3. Blocking follows the bytes that leave the machine. A secret in a claim label + is published and now blocks; a secret in a stdout blob stays in a gitignored + local ledger and now warns instead. + +Every credential-shaped string below is synthetic and structural — a shape the +detectors match, never a real token. Every path below is synthetic too: the +strings this work was measured against are a home directory and a private +project name, and a test fixture is a shipped file. The substitution is not +taken on faith — each synthetic path's entropy is asserted into the band its +group is supposed to occupy, so swapping the fixture for a differently-shaped +string fails here instead of quietly changing what is being tested. +""" + +from __future__ import annotations + +import json +import math +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import manifest as M +from didrun import redact +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +# --- the pre-P3.2 detector, reimplemented so before/after is checkable -------- + +# The token class as it stood before this prompt: `/` included, which is the +# entire defect. Written out rather than imported, so restoring the old class +# in redact.py cannot make these tests agree with it. +_OLD_TOKENISH = re.compile(r"[A-Za-z0-9_\-+/=]{24,}") +_HEXISH = re.compile(r"^[0-9a-fA-F]+$") + + +def _bits_per_char(s: str) -> float: + counts: dict = {} + for ch in s: + counts[ch] = counts.get(ch, 0) + 1 + n = len(s) + return -sum((c / n) * math.log2(c / n) for c in counts.values()) + + +def _old_entropy_findings(text: str) -> list: + """The entropy sweep exactly as it behaved before the tokenizer change.""" + covered = [] + for _kind, pat in redact._PATTERNS: + covered.extend(m.span() for m in pat.finditer(text)) + hits = [] + for m in _OLD_TOKENISH.finditer(text): + tok = m.group(0) + if _HEXISH.match(tok): + continue + if any(s <= m.start() < e for s, e in covered): + continue + if _bits_per_char(tok) >= redact._ENTROPY_MIN_BITS_PER_CHAR: + hits.append(m.span()) + return hits + + +def _max_token_bits(text: str, tokenizer=_OLD_TOKENISH) -> float: + toks = [m.group(0) for m in tokenizer.finditer(text) if not _HEXISH.match(m.group(0))] + return max((_bits_per_char(t) for t in toks), default=0.0) + + +# --- test 1: paths no longer tokenize whole ---------------------------------- + +# The band ordinary repository paths were measured to occupy. Paths in it were +# never the problem — they are the control group, and asserting the synthetics +# land in it is what makes the substitution faithful rather than convenient. +_ORDINARY_BAND = (3.94, 4.12) + +ORDINARY_PATHS = [ + "/home/runner/work/project-name/project-name/src/backend/services/authentication/handlers.py", + "/Users/exampleuser/src/example-project/pkg/adapters/outbound/persistence/migrations", +] + +# The knife edge: paths that scored between the ordinary band and the threshold. +# They did not fire, but a hair more variety in a directory name and they would +# have — which is the mechanism behind the observed 1-to-346 finding swing on +# runs that differed only in where the repo was checked out. +KNIFE_EDGE_PATHS = [ + "/Users/exampleuser/Documents/billing-reconciler/internal/messagebus/http/connection_lifecycle_test.go", +] + +# And the paths that actually fired: ordinary build and cache paths, every one +# of them a false positive, every one of them silent after the change. The +# homebrew path is used verbatim — it names no user and no private project. +DEEP_PATHS_THAT_FIRED = [ + "/opt/homebrew/Cellar/go/1.24.0/libexec/pkg/tool/darwin_arm64", + "/var/folders/qs/tmpbuild/example-service/cmd/gateway/internal/transport/grpc_handler.go", + "/Users/exampleuser/Library/Caches/build-tool/artifacts/darwin_arm64/release/binary", + "/Users/exampleuser/Documents/widget-platform-Qz4/internal/servicelayer/http/connection_lifecycle_test.go", +] + + +@pytest.mark.parametrize("path", ORDINARY_PATHS) +def test_ordinary_paths_are_faithful_substitutes_and_stay_quiet(path): + """The control group: in the measured band, silent before and after.""" + low, high = _ORDINARY_BAND + bits = _max_token_bits(path) + assert low <= bits <= high, ( + f"synthetic path is not a faithful substitute: {bits:.4f} bits/char is " + f"outside the measured ordinary-path band {_ORDINARY_BAND}" + ) + assert _old_entropy_findings(path) == [] + assert redact.scan(path) == [] + + +@pytest.mark.parametrize("path", KNIFE_EDGE_PATHS) +def test_knife_edge_paths_sit_between_the_band_and_the_threshold(path): + """Faithful to the near-miss class: above ordinary, still under the line.""" + bits = _max_token_bits(path) + assert _ORDINARY_BAND[1] < bits < redact._ENTROPY_MIN_BITS_PER_CHAR, ( + f"synthetic path is not a faithful knife-edge substitute: {bits:.4f}" + ) + assert _old_entropy_findings(path) == [] + assert redact.scan(path) == [] + + +@pytest.mark.parametrize("path", DEEP_PATHS_THAT_FIRED) +def test_deep_paths_fired_before_and_are_silent_after(path): + """The before/after difference, stated explicitly on each fixture.""" + bits = _max_token_bits(path) + assert bits >= redact._ENTROPY_MIN_BITS_PER_CHAR, ( + f"synthetic path is not a faithful substitute for a FIRING path: {bits:.4f}" + ) + before = _old_entropy_findings(path) + assert before, "fixture must fire under the old tokenizer or it proves nothing" + assert redact.scan(path) == [], "path still tokenizes whole" + + +def test_tokenizer_no_longer_admits_a_path_separator(): + """The one-character change, pinned directly. + + A path is now several tokens, not one, and the count is what changes the + entropy: the same characters spread over segments never reach the + threshold that the concatenation did. + """ + assert "/" not in redact._TOKENISH.pattern + path = DEEP_PATHS_THAT_FIRED[1] + old = _OLD_TOKENISH.findall(path) + new = redact._TOKENISH.findall(path) + assert any("/" in tok for tok in old), "fixture must span separators" + assert all("/" not in tok for tok in new) + # A path's segments are individually short, so the whole path no longer + # reaches the 24-character floor at all: it is not merely re-scored, it + # stops being a candidate. + assert max(len(t) for t in old) > max((len(t) for t in new), default=0) + + +def test_the_threshold_and_the_length_floor_did_not_move(): + """One variable at a time. If these move, the measurement above is void.""" + assert redact._ENTROPY_MIN_BITS_PER_CHAR == 4.2 + assert redact._ENTROPY_MIN_LEN == 24 + + +# --- test 2: structured recall is unchanged, and entropy still exists -------- + +STRUCTURED_SAMPLES = [ + ("aws-access-key", "AKIAIOSFODNN7EXAMPLE"), + ("github-token", "ghp_" + "A" * 36), + ("slack-token", "xoxb-1234567890-ABCDEFGHIJ"), + ("openai-key", "sk-AbCdEf0123456789XyZwVuTsQq"), + ("pem-private-key", "-----BEGIN RSA PRIVATE KEY-----"), + ("env-secret-assignment", "MY_SECRET=hunter2taffylongvalue"), +] + + +@pytest.mark.parametrize("kind,sample", STRUCTURED_SAMPLES) +def test_every_structured_detector_fires_at_block_tier(kind, sample): + findings = [f for f in redact.scan(sample) if f.kind == kind] + assert findings, f"{kind} lost recall" + assert all(f.tier == redact.TIER_BLOCK for f in findings) + assert all(f.detector_id == kind for f in findings) + + +def test_structured_detectors_all_declare_block_tier(): + """No structured pattern may quietly arrive at notice tier.""" + for kind, _pat in redact._PATTERNS: + assert redact._tier(kind) == redact.TIER_BLOCK + + +def test_entropy_sweep_still_fires_on_a_genuine_high_entropy_token(): + """DEMOTED, NOT DELETED. + + Deleting the sweep would satisfy every path assertion in this file while + removing the only detector that catches an unstructured credential. A + 24-character token drawn from a wide alphabet, with no `/` to be blamed on + the tokenizer, must still be reported. + """ + token = "aZ3kQ9mB7xT2vL5nR8wY4cF6" + assert "/" not in token and len(token) == redact._ENTROPY_MIN_LEN + findings = [f for f in redact.scan(token) if f.kind == "high-entropy"] + assert findings, "the entropy sweep was deleted, not demoted" + assert findings[0].tier == redact.TIER_NOTICE + + +def test_entropy_sweep_fires_inside_ordinary_surrounding_text(): + """The same token embedded in a command line is still found.""" + text = "curl -H 'X-Api: aZ3kQ9mB7xT2vL5nR8wY4cF6' https://example.invalid/v1" + assert any(f.kind == "high-entropy" for f in redact.scan(text)) + + +# --- test 3: locality -------------------------------------------------------- + +_BLOB = ( + "starting run\n" + "no secrets on this line\n" + "leaked ghp_" + "B" * 36 + " in output\n" + "done\n" +) + + +def test_findings_in_a_blob_carry_source_line_and_offset(): + source = redact.FindingSource("stdout", event_index=4) + findings = redact.scan(_BLOB, source) + hit = next(f for f in findings if f.kind == "github-token") + assert hit.source.kind == "stdout" + assert hit.source.event_index == 4 + assert hit.source.argv_index is None + assert hit.line == 3, "line must be 1-based within the scanned text" + # The offset indexes the text that was scanned, so a reader can go get it. + start = hit.source.byte_offset + assert _BLOB[start:start + (hit.span[1] - hit.span[0])] == _BLOB[hit.span[0]:hit.span[1]] + assert hit.fingerprint + + +def test_argv_findings_carry_an_argv_index_and_no_line(): + source = redact.FindingSource("argv", event_index=2, argv_index=3) + hit = redact.scan("MY_TOKEN=ghp_" + "C" * 36, source)[0] + assert hit.source.kind == "argv" + assert hit.source.argv_index == 3 + assert hit.line is None, "an argv element has no lines" + + +def test_findings_scanned_without_a_source_are_unlocated_not_wrong(): + """Back-compat: the original three-field call still works and lies about nothing.""" + hit = redact.scan("AKIAIOSFODNN7EXAMPLE")[0] + assert hit.source is None + assert hit.fingerprint == "" + assert redact.Finding(kind="k", span=(0, 1), preview="p").tier == redact.TIER_NOTICE + + +def test_fingerprint_is_stable_across_runs_over_identical_input(): + source = redact.FindingSource("stdout", event_index=4) + first = [f.fingerprint for f in redact.scan(_BLOB, source)] + second = [f.fingerprint for f in redact.scan(_BLOB, source)] + assert first == second + assert all(len(fp) == 16 for fp in first) + + +def test_fingerprint_locates_and_does_not_digest_the_token(): + """It is a LOCALITY digest, and that is a security property, not a shortcut. + + Digesting the token would publish an unkeyed hash of a short high-entropy + string next to the redaction that removed it — a brute-force oracle for the + thing just protected. So: two different secrets at the same location share a + fingerprint, and the same secret at two locations does not. + """ + source = redact.FindingSource("stdout", event_index=1) + a = redact.scan("ghp_" + "D" * 36, source)[0] + b = redact.scan("ghp_" + "E" * 36, source)[0] + assert a.fingerprint == b.fingerprint # same place, different token + + elsewhere = redact.FindingSource("stdout", event_index=2) + c = redact.scan("ghp_" + "D" * 36, elsewhere)[0] + assert a.fingerprint != c.fingerprint # same token, different place + + +def test_unknown_source_kind_is_refused(): + with pytest.raises(ValueError): + redact.FindingSource("stdin") + + +# --- test 4: the scope inversion is fixed ------------------------------------ + +_LABEL_TOKEN = "ghp_" + "F" * 36 + + +def _one_event(session: Session, repo: Path, code: str): + return run_wrapped([sys.executable, "-c", code], session, repo) + + +def _note_bytes(repo: Path): + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", "HEAD"], + cwd=str(repo), capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +def test_a_secret_in_a_claim_label_blocks_the_seal(repo: Path): + """The exported field that was never scanned. + + A claim label is operator-authored text that travels in the note to + wherever the note travels. Until now the gate never looked at it. + """ + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label=f"suite green with {_LABEL_TOKEN}", + event_indices=(0,), + declared_at_index=0, + ), + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo) + assert "github-token" in str(exc.value) + + +def test_a_secret_in_a_local_output_blob_warns_but_does_not_block(repo, capsys): + """The bytes that never leave: reported loudly, not a work stoppage. + + The token is assembled inside the child so it never appears in argv — argv + IS exported (as a redacted preview) and blocking on it is unchanged. + """ + s = _session(repo) + _one_event(s, repo, "print('gh' + 'p_' + 'B' * 36)") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="clean label", event_indices=(0,), + declared_at_index=0), + ) + m = M.seal(s, repo) + assert m.secrets["blocked"] is False + assert m.secrets_override is False + err = capsys.readouterr().err + assert "did not block" in err + assert "github-token" in err + assert "stdout of event 0" in err + + +def test_a_secret_in_argv_still_blocks_and_says_where(repo: Path): + """argv is exported, so argv still blocks — with an address this time.""" + s = _session(repo) + _one_event(s, repo, f"x = 'MY_TOKEN={_LABEL_TOKEN}'") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="clean", event_indices=(0,), declared_at_index=0), + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo) + message = str(exc.value) + assert "at argv of event 0 arg 2" in message + assert _LABEL_TOKEN not in message, "a refusal must not reprint the secret" + + +def test_a_deep_path_in_stdout_does_not_block_and_seals(repo, capsys): + """The regression the tokenizer change is for: a build path is not a secret.""" + s = _session(repo) + _one_event(s, repo, f"print({DEEP_PATHS_THAT_FIRED[0]!r})") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="build ok", event_indices=(0,), + declared_at_index=0), + ) + m = M.seal(s, repo) + assert m.secrets["findings_by_tier"] == {"block": 0, "notice": 0} + assert m.secrets["blocked"] is False + + +def test_allow_secrets_overrides_a_blocking_label(repo: Path): + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label=f"green {_LABEL_TOKEN}", event_indices=(0,), + declared_at_index=0), + ) + m = M.seal(s, repo, allow_secrets=True) + assert m.secrets_override is True + assert m.secrets["blocked"] is True + assert m.secrets["overridden"] is True + + +def test_allow_secrets_on_a_clean_seal_still_records_false(repo: Path): + """`secrets_override` keeps its exact meaning: an override actually happened.""" + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="clean", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo, allow_secrets=True) + assert m.secrets_override is False + assert m.secrets["overridden"] is False + + +# --- test 5: the scan domain is recorded ------------------------------------- + +def test_seal_records_what_was_actually_scanned(repo: Path): + """`secrets_override: false` used to be unfalsifiable — it was the same word + for "nothing was found" and "almost nothing was looked at".""" + s = _session(repo) + _one_event(s, repo, "print('one')") + _one_event(s, repo, "print('two')") + _one_event(s, repo, "print('three')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="first", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo) + secrets = m.secrets + # `claim-fields` is load-bearing in this string, not decoration: the label, + # the pathspecs and the changed paths are scanned as fields by the same + # pass that redacts them. Before that they reached the scan only inside the + # serialized manifest, which blocked a seal over a label and then published + # that label verbatim on --allow-secrets. + assert secrets["scan_domain"] == "claim-bound-events+claim-fields+manifest" + assert secrets["events_total"] == 3 + assert secrets["events_scanned"] == 1 + assert secrets["events_scanned"] <= secrets["events_total"] + assert secrets["bytes_scanned"] > 0 + assert secrets["detector_set_version"] == redact.DETECTOR_SET_VERSION + + +def test_the_secrets_block_survives_the_note_round_trip(repo: Path): + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="ok", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo) + body = _note_bytes(repo) + assert body is not None + parsed = M.Manifest.from_json(body.rstrip(b"\n")) + assert parsed.secrets == m.secrets + + +def test_a_note_without_a_secrets_block_reads_back_additively(): + """A stored v1/v2 note has no `secrets` key and must not become an error.""" + body = json.dumps( + { + "version": 1, + "commit": "0" * 40, + "tree": "1" * 40, + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + "secrets_override": False, + } + ).encode("ascii") + assert M.Manifest.from_json(body).secrets == {} + + +# --- change 4b: the detector-set version is declared, not inferred ----------- + +def test_detector_set_version_is_two_and_the_projection_version_is_not(repo: Path): + """Re-tiering every detector and changing what the entropy sweep matches IS + a detector-set change. The span-partition algorithm is untouched. + """ + assert redact.DETECTOR_SET_VERSION == 2 + assert redact.PROJECTION_VERSION == 1 + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="ok", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo) + assert m.claims[0]["redaction"]["detector_set_version"] == 2 + assert m.claims[0]["redaction"]["projection_version"] == 1 + assert m.secrets["detector_set_version"] == 2 + + +def test_a_v1_projection_is_distinguishable_from_a_v2_one(repo: Path): + """A consumer reads the declared version instead of inferring didrun's era.""" + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="ok", event_indices=(0,), declared_at_index=0), + ) + fresh = M.seal(s, repo).claims[0]["redaction"] + + stored_v1 = dict(fresh, detector_set_version=1) + assert stored_v1["projection_version"] == fresh["projection_version"] + assert stored_v1["detector_set_version"] != fresh["detector_set_version"] + + +# --- test 6: the refusal enumerates ------------------------------------------ + +def test_secrets_blocked_names_detector_source_and_line(): + source = redact.FindingSource("stdout", event_index=7) + findings = [f for f in redact.scan(_BLOB, source) if f.tier == redact.TIER_BLOCK] + message = str(redact.SecretsBlocked(findings)) + assert "github-token" in message + assert "stdout of event 7" in message + assert "line 3" in message + assert "fingerprint" in message + + +def test_secrets_blocked_caps_the_enumeration_and_counts_the_rest(): + """A refusal that prints 19,900 lines is as unusable as one that prints none.""" + text = "\n".join(f"line {i} ghp_" + "G" * 36 for i in range(50)) + findings = redact.scan(text, redact.FindingSource("manifest")) + assert len(findings) == 50 + message = str(redact.SecretsBlocked(findings)) + listed = [ln for ln in message.splitlines() if ln.startswith(" - ")] + assert len(listed) == redact.ENUMERATION_CAP + assert "and 30 more finding(s) not listed" in message + assert message.splitlines()[0].startswith("export blocked: 50 likely secret(s)") + + +# --- test 7: a blocked seal is free ------------------------------------------ + +def test_a_blocked_seal_advances_nothing(repo: Path): + """Currently unguarded, and it is what makes attempting a seal cheap: the + refusal must happen before the note is published and before the watermark + moves, or a blocked seal silently consumes the claims it refused. + """ + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="clean unit", event_indices=(0,), + declared_at_index=0), + ) + M.seal(s, repo) + + seals = s.root / "seals.jsonl" + seals_before = seals.read_bytes() + note_before = _note_bytes(repo) + assert note_before is not None + + M.declare_claim( + s, + Claim(ctype="tests-pass", label=f"second unit {_LABEL_TOKEN}", + event_indices=(0,), declared_at_index=0), + ) + with pytest.raises(redact.SecretsBlocked): + M.seal(s, repo) + + assert seals.read_bytes() == seals_before, "the watermark moved on a refused seal" + assert _note_bytes(repo) == note_before, "a refused seal republished the note" + + # And the refusal is genuinely free: fixing the label and re-sealing works, + # with the refused claim still in scope rather than swallowed. + (s.root / "claims.jsonl").write_bytes( + b"\n".join( + line for line in (s.root / "claims.jsonl").read_bytes().splitlines() + if _LABEL_TOKEN.encode("ascii") not in line + ) + + b"\n" + ) + M.declare_claim( + s, + Claim(ctype="tests-pass", label="second unit", event_indices=(0,), + declared_at_index=0), + ) + # `reseal=True` because this seals the SAME commit a second time with a + # narrower window ([1,2) over a note that records [0,1)), which P4.1 refuses + # by default — that narrowing is the data-loss path it closes. What this test + # is about is a blocked seal advancing nothing, so the flag keeps its subject + # intact rather than the assertion below being softened. + m = M.seal(s, repo, reseal=True) + assert [c["claim"]["label"] for c in m.claims] == ["second unit"] + + +# --- test 8: the false-positive pin ------------------------------------------ + +# Realistic developer strings and the tier each is expected to reach. This is a +# table so that any future tuning of the detectors shows up as a diff on a +# reviewed list rather than as a number nobody looked at. `notice` entries are +# not endorsements — they are the entropy sweep's remaining reach, recorded so +# that a change to it is visible. +DEVELOPER_STRINGS = [ + ("commit 356a192b7913b04c54574d18c28d46e6395428ab passed", set()), + ("sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", set()), + ("https://example.invalid/example-org/example-repo/pull/1284#issuecomment-2098765432", set()), + ("npm WARN deprecated request@2.88.2: request has been deprecated", set()), + ("registry.invalid/library/postgres@sha256:1a2b3c4d5e6f70819a2b3c4d5e6f7081", set()), + ("pytest -q tests/test_capture_claims_manifest.py::test_grade_tree_exact", set()), + ("go test ./internal/servicelayer/... -run TestServiceLifecycle -count=1", set()), + ("--define=GOFLAGS=-mod=readonly -trimpath", set()), + ("cargo build --release --target aarch64-apple-darwin", set()), + ("kubectl -n production get pods -l app.kubernetes.io/name=gateway", set()), + ("registry.invalid/example-org/example-service:v1.24.0-rc3", set()), + ("uuid 3f2504e0-4f89-11d3-9a0c-0305e82c3301 assigned", set()), + ("git rebase --onto origin/main feature/long-running-branch-name", set()), + ("Traceback (most recent call last):\n File \"app/main.py\", line 42", set()), + ("ruff check --select E,F,W --ignore E501 src/ tests/", set()), + ("terraform plan -var-file=environments/production/main.tfvars", set()), + ("ORDINARY_PATH " + ORDINARY_PATHS[0], set()), + ("DEEP_PATH " + DEEP_PATHS_THAT_FIRED[0], set()), + # Still reported, and deliberately so: these are the shapes an unstructured + # credential actually takes, and the sweep is the only detector that sees them. + ("Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9", {"notice"}), + ("integrity sha512-abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOP==", {"notice"}), + # And the six that must never be demoted. + ("AKIAIOSFODNN7EXAMPLE", {"block"}), + ("ghp_" + "A" * 36, {"block"}), + ("xoxb-1234567890-ABCDEFGHIJ", {"block"}), + ("sk-AbCdEf0123456789XyZwVuTsQq", {"block"}), + ("-----BEGIN OPENSSH PRIVATE KEY-----", {"block"}), + ("DEPLOY_TOKEN=hunter2taffylongvalue", {"block"}), +] + + +@pytest.mark.parametrize("text,expected", DEVELOPER_STRINGS) +def test_developer_string_tiers_are_pinned(text, expected): + assert {f.tier for f in redact.scan(text)} == expected + + +def test_the_false_positive_table_covers_both_outcomes(): + """A table that only contained clean strings would pass if scan() returned [].""" + tiers = {t for _s, expected in DEVELOPER_STRINGS for t in expected} + assert tiers == {"block", "notice"} diff --git a/tests/test_env_fingerprint.py b/tests/test_env_fingerprint.py new file mode 100644 index 0000000..ac6303a --- /dev/null +++ b/tests/test_env_fingerprint.py @@ -0,0 +1,484 @@ +"""P3.3 — the environment fingerprint is bound to something and read by someone. + +Before this unit the fingerprint was written twice and read nowhere: two green +wrapper events recorded under different umasks were indistinguishable in the +receipt, and the key set gated on the two worst false-drift sources on a +developer machine (PATH, PWD) while omitting the one that voided real chains +(umask). + +The comparison that makes it load-bearing is ACROSS TIME — sealed fingerprint +versus this process's. Comparing the fingerprints inside a single claim would be +vacuous: every claim didrun's CLI can declare binds exactly one event, and one +event trivially agrees with itself, so ``return False`` would pass such a test on +100% of real claims. +""" + +from __future__ import annotations + +import os +import re +import subprocess +import sys +import textwrap +from pathlib import Path + +from didrun import capture +from didrun import cli +from didrun import gitplumbing as gp +from didrun import manifest as M +from didrun import render +from didrun.claims import ( + ENV_DRIFT, + ENV_INCOMPARABLE, + ENV_MATCH, + ENV_NOT_RECORDED, + Claim, +) +from didrun.ledger import Event, Session + +# A structurally equivalent stand-in for a real environment, so no test here +# depends on the machine it runs on. +_ENV = { + "LANG": "en_US.UTF-8", + "SHELL": "/bin/zsh", + "TZ": "UTC", + "VIRTUAL_ENV": "/opt/envs/example", + "PATH": "/usr/bin:/bin", +} + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _ok(repo: Path, session: Session): + return capture.run_wrapped([sys.executable, "-c", "print(1)"], session, repo) + + +def _sealed_green(repo: Path) -> Session: + """One recorded success, one claim over it, sealed on a clean tree.""" + s = _session(repo) + _ok(repo, s) + M.declare_claim( + s, Claim(ctype="tests-pass", label="suite", event_indices=(0,), declared_at_index=0) + ) + M.seal(s, repo) + return s + + +def _seal_with_fingerprint(repo: Path, fingerprint: str) -> Session: + """Seal a green claim over an event carrying ``fingerprint`` verbatim. + + The event is constructed rather than recorded because the point is a + fingerprint THIS didrun would never produce — a bare v0.1 digest, or a v1 + value from before the key set changed. It is appended through the real + ``Session.append`` so the chain covers it exactly as it covers a captured + event; nothing here edits a log. + """ + s = _session(repo) + tree = gp.tree_digest(repo, s.blobs.root) + s.append( + Event( + argv=("example-suite", "--quiet"), + cwd=str(repo), + env_fingerprint=fingerprint, + observed_via="wrapper", + coverage="complete", + exit_code=0, + started_at=1.0, + ended_at=2.0, + tree_before=tree, + tree_after=tree, + ) + ) + M.declare_claim( + s, Claim(ctype="tests-pass", label="suite", event_indices=(0,), declared_at_index=0) + ) + M.seal(s, repo) + return s + + +def _run_child(script: str, *args: str, env=None, cwd=None) -> str: + proc = subprocess.run( + [sys.executable, "-c", textwrap.dedent(script), *args], + capture_output=True, + text=True, + check=True, + env=env, + cwd=cwd, + ) + return proc.stdout.strip() + + +# --- test 1: the falsification test (criterion per red-team §3.1(3)) --------- + +def test_a_changed_TZ_drifts_the_verdict_and_refuses_under_require_env_match( + repo: Path, monkeypatch, capsys +): + """Seal green, verify under a different TZ, and the verdict must move. + + THE FALSIFICATION: a no-op implementation fails BOTH halves. If nothing + compares the fingerprint across time (the pre-P3.3 state, or a reader that + always answers "match"), the two verdict texts are identical and + --require-env-match --strict still exits 0. Passing requires the comparison + to exist AND to be able to refuse. + + TZ is the mover on purpose: it is in the gate, it is trivially settable, and + it is not PATH or PWD — both of which are now deliberately unable to do + this. + """ + monkeypatch.setenv("TZ", "UTC") + s = _sealed_green(repo) + + before = render.render_verdict(M.verify(s, repo)) + assert "1/1 claims recorded-exact" in before + assert f"env: 1 {ENV_MATCH} / 0 drifted / 0 {ENV_INCOMPARABLE}" in before + assert ENV_DRIFT not in before + + monkeypatch.setenv("TZ", "Asia/Tokyo") + after = render.render_verdict(M.verify(s, repo)) + assert after != before + assert ENV_DRIFT in after + assert "0 match / 1 drifted" in after + # Drift did not un-record the command: the grade is untouched. + assert "1/1 claims recorded-exact" in after + + # Advisory by default... + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + capsys.readouterr() + # ...and a refusal on request. + assert cli.main(["--repo", str(repo), "verify", "--require-env-match", "--strict"]) == 1 + out = capsys.readouterr().out + assert ENV_DRIFT in out + + +def test_drift_alone_does_not_change_the_grade_or_the_default_verdict( + repo: Path, monkeypatch +): + """`all_verified` keeps its default membership. Drift is a fact about this + machine now, not about whether the recorded command ran.""" + monkeypatch.setenv("TZ", "UTC") + s = _sealed_green(repo) + monkeypatch.setenv("TZ", "Asia/Tokyo") + + drifted = M.verify(s, repo) + assert drifted.env_drift_count == 1 + assert [r.grade for r in drifted.results] == ["tree-exact"] + assert drifted.all_verified is True + assert drifted.worst_status == "tree-exact" + + gated = M.verify(s, repo, require_env_match=True) + assert gated.all_verified is False + # Still not a grade: the claim is recorded-exact and says so. + assert [r.grade for r in gated.results] == ["tree-exact"] + + +# --- test 2: umask is bound -------------------------------------------------- + +_RECORD_UNDER_UMASK = """ + import os, sys + os.umask(int(sys.argv[3], 8)) + from pathlib import Path + from didrun.capture import run_wrapped + from didrun.ledger import Session + ev = run_wrapped( + [sys.executable, "-c", "pass"], Session(Path(sys.argv[2])), Path(sys.argv[1]) + ) + print(ev.env_fingerprint) +""" + + +def test_umask_is_bound_into_the_gate(repo: Path, tmp_path: Path): + """Two events recorded under umask 022 and 077 must differ in the gate. + + This is the case that voided three chains and was invisible in the receipt: + same argv, same tree, same exit code, two different file-permission + outcomes, one indistinguishable pair of green events. + + Recorded in child processes because the umask is read once per process and + reused (capture.process_umask) — which is itself the property under test: + the value must reach the ledger, not merely the helper's return value. + """ + prints = [ + _run_child( + _RECORD_UNDER_UMASK, str(repo), str(tmp_path / f"ledger-{mask}"), mask + ) + for mask in ("022", "077") + ] + assert all(capture.fingerprint_version(p) == 2 for p in prints) + assert capture.fingerprint_gate(prints[0]) != capture.fingerprint_gate(prints[1]) + # Only the umask moved: the non-gated PATH digest is unchanged. + assert capture.fingerprint_path_digest(prints[0]) == capture.fingerprint_path_digest( + prints[1] + ) + + +def test_umask_reaches_the_fingerprint_directly_too(): + """The same fact without a subprocess, so a failure localizes.""" + a = capture.env_fingerprint(_ENV, umask=0o022) + b = capture.env_fingerprint(_ENV, umask=0o077) + assert capture.fingerprint_gate(a) != capture.fingerprint_gate(b) + assert capture.env_fingerprint(_ENV, umask=0o022) == a + + +# --- test 3: PWD no longer drifts -------------------------------------------- + +_RECORD_FROM_CWD = """ + import sys + from pathlib import Path + from didrun.capture import run_wrapped + from didrun.ledger import Session + ev = run_wrapped( + [sys.executable, "-c", "pass"], Session(Path(sys.argv[2])), Path(sys.argv[1]) + ) + print(ev.env_fingerprint) +""" + + +def test_the_invocation_directory_does_not_drift_the_gate(repo: Path, tmp_path: Path): + """The same command against the same repo, invoked from two directories. + + PWD was in v0.1's key set and is NOT the child's cwd — run_wrapped always + spawns with cwd=str(repo). Gating on it would mean two byte-identical + executions disagree because the operator was standing somewhere else. + + Both the real child cwd and the PWD variable are moved, because a shell sets + the second and a launcher may not set it at all. + """ + here = tmp_path / "here" + there = tmp_path / "there" + here.mkdir() + there.mkdir() + prints = [ + _run_child( + _RECORD_FROM_CWD, + str(repo), + str(tmp_path / f"ledger-{d.name}"), + env={**os.environ, "PWD": str(d)}, + cwd=str(d), + ) + for d in (here, there) + ] + assert capture.fingerprint_gate(prints[0]) == capture.fingerprint_gate(prints[1]) + assert prints[0] == prints[1] + + +def test_PWD_is_not_in_the_preimage_at_all(): + assert "PWD" not in capture._ENV_FINGERPRINT_KEYS + a = capture.env_fingerprint({**_ENV, "PWD": "/home/example/one"}, umask=0o022) + b = capture.env_fingerprint({**_ENV, "PWD": "/home/example/two"}, umask=0o022) + assert a == b + + +# --- test 4: PATH is visible but not a gate ---------------------------------- + +def test_PATH_moves_the_recorded_digest_but_never_the_gate(): + """Two different PATHs: same gate, different `path=` suffix. + + PATH varies with terminal tab, direnv, `nix develop`, tmux and homebrew + shellenv ordering. Gating on it produces a refusal that fires on a new shell + and is ignored within a week — but dropping it entirely would lose a real + signal, so it is recorded beside the gate instead of inside it. + """ + assert "PATH" not in capture._ENV_FINGERPRINT_KEYS + a = capture.env_fingerprint({**_ENV, "PATH": "/usr/bin:/bin"}, umask=0o022) + b = capture.env_fingerprint({**_ENV, "PATH": "/opt/example/bin:/bin"}, umask=0o022) + assert capture.fingerprint_gate(a) == capture.fingerprint_gate(b) + assert capture.fingerprint_path_digest(a) != capture.fingerprint_path_digest(b) + + +def test_a_PATH_difference_is_an_advisory_and_never_drift(repo: Path, monkeypatch): + """A verdict must not call a PATH change environment drift.""" + monkeypatch.setenv("PATH", os.environ.get("PATH", "/usr/bin:/bin")) + s = _sealed_green(repo) + monkeypatch.setenv("PATH", "/opt/example/bin:" + os.environ.get("PATH", "")) + + report = M.verify(s, repo) + assert [r.env_status for r in report.results] == [ENV_MATCH] + assert report.env_drift_count == 0 + assert "PATH differs" in report.results[0].env_reason + assert "recorded, not gated" in report.results[0].env_reason + assert M.verify(s, repo, require_env_match=True).all_verified is True + + +# --- test 5: legacy is incomparable, not drifted ----------------------------- + +def test_a_bare_v0_1_fingerprint_is_incomparable_and_never_renders_as_drift( + repo: Path, +): + """The archived-evidence case, and the reason this unit is compat-safe. + + A v0.1 digest carries no version prefix, so it is not a measurement of the + same key set. Reporting a mismatch against it as drift would make every note + published before v0.2 look like the environment moved. + """ + s = _seal_with_fingerprint(repo, "0123456789abcdef") + report = M.verify(s, repo) + + assert [r.env_status for r in report.results] == [ENV_INCOMPARABLE] + assert "unversioned v0.1 digest" in report.results[0].env_reason + assert "not evidence that anything drifted" in report.results[0].env_reason + + text = render.render_verdict(report) + assert ENV_DRIFT not in text + assert f"0 drifted / 1 {ENV_INCOMPARABLE}" in text + assert ENV_DRIFT not in render.render_html(report) + + # Incomparable never refuses, under either mode. This is the assertion that + # would break the corpus if it stopped holding. + assert report.all_verified is True + assert M.verify(s, repo, require_env_match=True).all_verified is True + + +def test_a_v1_fingerprint_from_an_older_didrun_is_incomparable_too(repo: Path): + """Sealed evidence from P0.3-era didrun (versioned, older key set). + + The version prefix exists precisely so this case is legible: v1 and v2 are + digests of different key sets, so they are incomparable by construction + rather than by luck. + """ + s = _seal_with_fingerprint(repo, "v1:0123456789abcdef") + report = M.verify(s, repo) + assert [r.env_status for r in report.results] == [ENV_INCOMPARABLE] + assert "version 1" in report.results[0].env_reason + assert "incomparable, not drifted" in report.results[0].env_reason + assert M.verify(s, repo, require_env_match=True).all_verified is True + + +def test_a_note_that_bound_no_fingerprint_is_not_recorded_not_drifted(repo: Path): + """A v0.1-shaped note has no evidence block at all — a third distinct fact. + + Reported separately from `incomparable` because "the seal never recorded + one" and "it recorded one this binary cannot compare" are different, and + collapsing them would hide which notes could start being compared. + """ + s = _sealed_green(repo) + head = gp.head_commit(repo) + body = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", head], + cwd=str(repo), + capture_output=True, + text=True, + check=True, + ).stdout + stripped = body.replace('"evidence"', '"evidence_removed"') + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-F", "-", head], + cwd=str(repo), + input=stripped.encode("ascii"), + capture_output=True, + check=True, + ) + + report = M.verify(s, repo) + assert [r.env_status for r in report.results] == [ENV_NOT_RECORDED] + assert "bound no environment fingerprint" in report.results[0].env_reason + text = render.render_verdict(report) + assert f"1 {ENV_NOT_RECORDED}" in text + assert ENV_DRIFT not in text + assert M.verify(s, repo, require_env_match=True).all_verified is True + + +# --- test 6: determinism ----------------------------------------------------- + +_FINGERPRINT_IN_CHILD = """ + import json, sys + from didrun import capture + env = json.loads(sys.argv[1]) + print(capture.env_fingerprint(env, umask=int(sys.argv[2], 8))) +""" + + +def test_fingerprint_is_reproducible_across_processes(tmp_path: Path): + """Determinism is a trust-path invariant (no wall clock, no randomness). + + Mirrors the cross-process chain-hash check in tests/test_ledger.py: a + same-process comparison cannot catch hash randomization or key-order + instability, because both are fixed for a process's lifetime. + """ + import json + + out = [ + _run_child(_FINGERPRINT_IN_CHILD, json.dumps(_ENV, sort_keys=True), "022") + for _ in range(2) + ] + assert out[0] == out[1] + assert out[0] == capture.env_fingerprint(_ENV, umask=0o022) + + +def test_the_umask_read_is_cached_not_repeated(monkeypatch): + """Read once, reuse. The read is a read-modify-write on process-global + state, so doing it per event would open that window on every wrapped + command.""" + monkeypatch.setattr(capture, "_UMASK", None) + reads = [] + real_umask = os.umask + + def counting_umask(mask): + reads.append(mask) + return real_umask(mask) + + monkeypatch.setattr(capture.os, "umask", counting_umask) + first = capture.process_umask() + assert reads == [0, first] # set to 0, then restored + for _ in range(5): + assert capture.process_umask() == first + assert reads == [0, first] # and never touched again + + +# --- test 7: no leak --------------------------------------------------------- + +def test_the_fingerprint_leaks_no_input_value(): + """Values are digested, never stored. The fingerprint must never become the + place a secret-bearing environment variable lands verbatim in a note.""" + inputs = { + "LANG": "xq_ZZ.KOI8-R", + "SHELL": "/opt/quixotic/bin/wobbleshell", + "TZ": "Antarctica/South_Pole", + "VIRTUAL_ENV": "/opt/envs/zephyrous-marmoset-3141", + "PATH": "/opt/quixotic/bin:/usr/lib/kumquat/sbin", + } + value = capture.env_fingerprint(inputs, umask=0o022) + assert re.fullmatch(r"v2:[0-9a-f]{16}:path=[0-9a-f]{8}", value) + for raw in inputs.values(): + assert raw not in value + for i in range(len(raw) - 3): + assert raw[i : i + 4] not in value + + +def test_the_drift_reason_carries_digests_not_environment_values( + repo: Path, monkeypatch +): + """The reason names both gates. They are 16-hex digests, so naming them + leaks nothing — which is what makes the message safe to print.""" + monkeypatch.setenv("TZ", "UTC") + s = _sealed_green(repo) + monkeypatch.setenv("TZ", "Antarctica/South_Pole") + reason = M.verify(s, repo).results[0].env_reason + assert "Antarctica" not in reason + assert len(re.findall(r"\b[0-9a-f]{16}\b", reason)) == 2 + + +# --- the parsers ------------------------------------------------------------- + +def test_fingerprint_accessors_refuse_rather_than_guess(): + value = capture.env_fingerprint(_ENV, umask=0o022) + assert capture.fingerprint_version(value) == 2 + assert capture.fingerprint_gate(value) == value.split(":")[1] + assert capture.fingerprint_path_digest(value) == value.split("path=")[1] + + # A bare v0.1 digest: no version, no gate, no path digest — INCOMPARABLE at + # every accessor, so no consumer can accidentally compare one. + assert capture.fingerprint_version("0123456789abcdef") is None + assert capture.fingerprint_gate("0123456789abcdef") is None + assert capture.fingerprint_path_digest("0123456789abcdef") is None + + # A v1 value: comparable within v1, and carries no path digest. + assert capture.fingerprint_version("v1:0123456789abcdef") == 1 + assert capture.fingerprint_gate("v1:0123456789abcdef") == "0123456789abcdef" + assert capture.fingerprint_path_digest("v1:0123456789abcdef") is None + + # Malformed shapes are refusals, never empty-string matches. + assert capture.fingerprint_gate("v2:") is None + assert capture.fingerprint_path_digest("v2:0123456789abcdef:") is None + assert capture.fingerprint_path_digest("v2:0123456789abcdef:other=aa") is None diff --git a/tests/test_evidence_binding.py b/tests/test_evidence_binding.py new file mode 100644 index 0000000..9919285 --- /dev/null +++ b/tests/test_evidence_binding.py @@ -0,0 +1,406 @@ +"""P1.2 — a sealed claim is bound to the evidence it was sealed against. + +The defect these pin: `verify` re-graded every claim from scratch against +whatever ledger was on disk, keyed by integer index, and threw away the grades +it sealed. So an archived ledger made every claim `unknown` with the same reason +string a genuinely unbacked claim gets, and a SUBSTITUTED ledger of unrelated +exit-0 commands on the same tree graded a full set of claims recorded-exact and +exited 0. + +What this closes, stated honestly: verify used to silently regrade against +whatever ledger is present, so archival, rebuild or index drift changed a sealed +verdict with no signal. Binding the claim to its entry hash buys ACCIDENT AND +DRIFT DETECTION on a published note. It does not stop a forger — whoever can +substitute a ledger can regenerate its chain and re-run claim+seal to mint fresh +hash-bound claims (ledger.py's declared non-guarantee). +""" + +from __future__ import annotations + +import json +import shutil +import subprocess +import sys +from pathlib import Path + +from didrun import cli +from didrun import gitplumbing as gp +from didrun import manifest as M +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import Claim, GradeResult +from didrun.ledger import Session, canonical_json + + +def _git(repo: Path, *a: str) -> None: + subprocess.run(["git", *a], cwd=str(repo), capture_output=True, text=True, check=True) + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _ok(repo: Path, session: Session, text: str): + return run_wrapped([sys.executable, "-c", f"print({text!r})"], session, repo) + + +def _sealed_green(repo: Path, label: str = "suite") -> Session: + """One recorded success, one claim over it, sealed on a clean tree.""" + s = _session(repo) + _ok(repo, s, "the unit under test") + M.declare_claim( + s, Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0) + ) + M.seal(s, repo) + return s + + +def _substitute_log(repo: Path, session: Session, tmp_path: Path, texts) -> None: + """Replace session.log with a ledger of DIFFERENT commands on the same tree. + + Built by running real commands into a fresh Session, never by splicing log + lines: a spliced log has broken prev_hash linkage, so it reports a broken + chain, which dominates and fails --strict regardless of whether the evidence + binding works at all. The test would then pass for the wrong reason and stop + testing this unit. + + The substitute ledger lives OUTSIDE the repo. A ledger directory inside it + under any name but `.didrun` would enter the tree digest, moving tree_after + away from the sealed tree — and the whole point of this fixture is a + substitute whose events still look tree-exact. + """ + alt = Session(tmp_path / "alt-ledger") + for text in texts: + _ok(repo, alt, text) + shutil.copyfile(alt.log_path, session.log_path) + + +# --- test 1: the falsification test (criterion per red-team §3.3) ------------ + + +def test_substituted_ledger_of_the_same_length_is_caught(repo: Path, tmp_path: Path): + """A same-length ledger of different commands on the same tree. + + This fails on pre-P1.2 code, which grades it `tree-exact` and exits 0 — the + substitute's events are self-stable against the sealed tree, which is all + the index-keyed regrade ever looked at. It would ALSO fail with only the + witness-unavailable half: `supporting_event_index >= len(events)` is false + here, because the substitute ledger is exactly as long as the one it + replaced. Only the hash binding detects it. + """ + s = _sealed_green(repo) + sealed_tree = gp.commit_tree(repo, "HEAD") + _substitute_log(repo, s, tmp_path, ["something else entirely"]) + + live = list(s.entries()) + assert len(live) == 1, "the substitute must be the SAME LENGTH, or test 2 covers it" + assert live[0].event.tree_after == sealed_tree, ( + "the substitute must still look tree-exact, or the tree check catches it " + "and the hash binding is never exercised" + ) + assert s.verify_chain_detail().status == "intact", ( + "a spliced log would report a broken chain and dominate the verdict; " + "evidence binding must be the only thing detecting this" + ) + + report = M.verify(s, repo) + assert [r.grade for r in report.results] == ["witness-unavailable"] + assert "hash mismatch" in report.results[0].reason + assert report.results[0].sealed_grade == "tree-exact" + assert not report.all_verified # → --strict exits non-zero + assert report.evidence_bound_count == 0 + assert "hash mismatch" in render.render_verdict(report) + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +# --- test 2: a ledger that is merely short ------------------------------------ + + +def test_truncated_ledger_names_the_out_of_range_index(repo: Path, tmp_path: Path): + s = _session(repo) + _ok(repo, s, "first") + _ok(repo, s, "second") + M.declare_claim( + s, Claim(ctype="tests-pass", label="late", event_indices=(1,), declared_at_index=1) + ) + M.seal(s, repo) + + lines = s.log_path.read_bytes().splitlines(keepends=True) + assert len(lines) == 2 + s.log_path.write_bytes(lines[0]) # drop the backing entry + + report = M.verify(s, repo) + assert [r.grade for r in report.results] == ["witness-unavailable"] + reason = report.results[0].reason + assert "index 1" in reason and "out of range" in reason + assert "1 event(s)" in reason + assert not report.all_verified + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +# --- test 3: the archived ledger ---------------------------------------------- + + +# The exact sentence a genuinely unbacked claim gets. An archived ledger used to +# produce this byte-for-byte, which made "the evidence is elsewhere" and "there +# never was any evidence" indistinguishable in the report. +_UNBACKED_REASON = "no witnessed successful command backs this claim" + + +def test_archived_ledger_reports_the_sealed_grade_not_a_bare_unknown( + repo: Path, tmp_path: Path +): + s = _sealed_green(repo) + shutil.move(str(repo / ".didrun"), str(tmp_path / "archived-ledger")) + + report = M.verify(_session(repo), repo) + assert [r.grade for r in report.results] == ["witness-unavailable"] + reason = report.results[0].reason + assert "sealed as tree-exact" in reason + assert _UNBACKED_REASON not in reason + assert report.results[0].sealed_grade == "tree-exact" + assert not report.all_verified + assert report.evidence_bound_count == 0 + + +# --- test 4: a genuinely unbacked claim is still `unknown` -------------------- + + +def test_unbacked_claim_stays_unknown_with_the_original_reason(repo: Path): + """The two states must remain distinguishable — that is the whole point. + + A claim bound to an index that never had an event has nothing to bind to, so + the seal writes no evidence block and verify says what it has always said. + """ + s = _session(repo) + _ok(repo, s, "unrelated") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="phantom", event_indices=(7,), declared_at_index=0), + ) + M.seal(s, repo) + + body = json.loads(_note_bytes(repo)) + assert "evidence" not in body["claims"][0], "nothing backed it; nothing to bind" + + report = M.verify(s, repo) + assert [r.grade for r in report.results] == ["unknown"] + assert report.results[0].reason == _UNBACKED_REASON + assert report.evidence_bound_count == 0 + + +# --- test 5: a legacy v1 note verifies exactly as it did before --------------- + +# Generated by running this fixture against the code as it stood BEFORE P1.2 +# (see the unit's report), then pasted in. Hard-coded on purpose: computing them +# from the code under test would make "identical to pre-change behaviour" a +# claim this test cannot actually make. +_LEGACY_EXPECTED = [ + ("fresh", "tree-exact", "self-stable command ran against the sealed tree", []), + ("drifted", "stale", "tree moved since evidence: 1 path(s) differ", [("M", "calc.py")]), + ("unbacked", "unknown", "no witnessed successful command backs this claim", []), + ("caught", "failed", "the recorded command exited 1 — claim is not backed", []), +] + + +def _v1_claim_entry(ctype, label, indices, declared_at, grade, supporting, exit_code): + """A claim entry in the shape a v1 note carried — no `evidence` key.""" + return { + "claim": { + "ctype": ctype, + "label": label, + "event_indices": list(indices), + "pathspecs": [], + "declared_at_index": declared_at, + "argv_preview": [], + }, + "grade": grade, + "reason": "sealed", + "delta": [], + "supporting_event_index": supporting, + "exit_code": exit_code, + } + + +def _attach_raw_note(repo: Path, commit: str, body: bytes) -> None: + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-F", "-", commit], + cwd=str(repo), + input=body, + capture_output=True, + check=True, + ) + + +def _note_bytes(repo: Path, commit: str = "HEAD") -> bytes: + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + check=True, + ) + return proc.stdout + + +def test_legacy_v1_note_verifies_byte_for_byte_as_before(repo: Path): + """The corpus-protection test: 65 published notes are all version 1. + + Every one of them carries no evidence block, so every one of them must take + the unbound path and produce the grades, reasons and deltas it produced + before this unit existed — down to the string. + """ + s = _session(repo) + _ok(repo, s, "1") # event 0 @T0 + run_wrapped([sys.executable, "-c", "import sys; sys.exit(1)"], s, repo) # event 1 @T0 + (repo / "calc.py").write_text("def add(a, b):\n return a + b # edit\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit") + _ok(repo, s, "2") # event 2 @T1 + + commit = gp.head_commit(repo) + body = canonical_json( + { + "version": 1, + "commit": commit, + "tree": gp.commit_tree(repo, commit), + "claims": [ + _v1_claim_entry("tests-pass", "fresh", (2,), 2, "tree-exact", 2, 0), + _v1_claim_entry("command-succeeded", "drifted", (0,), 0, "stale", 0, 0), + _v1_claim_entry("lint-clean", "unbacked", (7,), 2, "unknown", None, None), + _v1_claim_entry("tests-pass", "caught", (1,), 1, "failed", 1, 1), + ], + "coverage": {"total_events": 3, "by_coverage": {"complete": 3}}, + "secrets_override": False, + } + ) + _attach_raw_note(repo, commit, body) + + report = M.verify(s, repo) + observed = [ + (r.claim.label, r.grade, r.reason, [(c.status, c.path) for c in r.delta]) + for r in report.results + ] + assert observed == _LEGACY_EXPECTED + assert report.worst_status == "failed" + assert not report.all_verified + assert report.evidence_bound_count == 0 + assert report.total == 4 + assert all(r.sealed_grade is None for r in report.results) + + +# --- test 6: the happy path --------------------------------------------------- + + +def test_happy_path_is_evidence_bound_and_strict_clean(repo: Path): + s = _session(repo) + _ok(repo, s, "one") + _ok(repo, s, "two") + M.declare_claim( + s, Claim(ctype="tests-pass", label="a", event_indices=(0,), declared_at_index=1) + ) + M.declare_claim( + s, Claim(ctype="lint-clean", label="b", event_indices=(1,), declared_at_index=1) + ) + M.seal(s, repo) + + report = M.verify(s, repo) + assert [r.grade for r in report.results] == ["tree-exact", "tree-exact"] + assert all(r.evidence_bound for r in report.results) + assert report.evidence_bound_count == 2 and report.total == 2 + assert report.all_verified + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_seal_records_the_entry_hash_of_the_backing_event(repo: Path): + """The binding is the recorded entry's own chain hash, taken from the ledger.""" + s = _sealed_green(repo) + entry = list(s.entries())[0] + evidence = json.loads(_note_bytes(repo))["claims"][0]["evidence"] + assert evidence["event_index"] == 0 + assert evidence["entry_hash"] == entry.entry_hash + assert evidence["tree_after"] == entry.event.tree_after + assert evidence["env_fingerprint"] == entry.event.env_fingerprint + + +def test_amend_tree_fallback_still_resolves_and_stays_bound(repo: Path): + """The durable anchor keeps working: same tree, new commit id, still bound.""" + s = _sealed_green(repo) + original = gp.head_commit(repo) + _git(repo, "commit", "--amend", "-qm", "amended message") + assert gp.head_commit(repo) != original + + report = M.verify(s, repo) + assert report.resolved_by == "tree-fallback" + assert report.all_verified + assert report.evidence_bound_count == 1 + + +# --- test 7: the state is registered on the human surfaces ------------------- + + +def test_witness_unavailable_is_registered_in_both_render_tables(): + """Both tables are read through `.get(…, UNKNOWN)`, so an unregistered grade + renders as UNKNOWN — a silent lie rather than a loud error.""" + assert "witness-unavailable" in render._GRADE_DISPLAY + assert "witness-unavailable" in render._SORT_RANK + assert render._GRADE_DISPLAY["witness-unavailable"][0] == "WITNESS-UNAVAIL" + # Worse than stale, better than unknown — the order the verdict sorts by. + assert ( + render._SORT_RANK["unknown"] + < render._SORT_RANK["witness-unavailable"] + < render._SORT_RANK["stale"] + ) + + +def test_verdict_never_calls_a_witness_unavailable_report_recorded_exact( + repo: Path, tmp_path: Path, monkeypatch +): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed_green(repo) + _substitute_log(repo, s, tmp_path, ["something else entirely"]) + + out = render.render_verdict(M.verify(s, repo)) + assert "WITNESS-UNAVAIL" in out + assert "ALL RECORDED-EXACT" not in out + assert "0/1 claims evidence-bound" in out + for overclaim in ("VERIFIED", "PASSED", " OK"): + assert overclaim not in out + + +def test_html_report_shows_the_state_and_never_reads_as_passed( + repo: Path, tmp_path: Path +): + s = _sealed_green(repo) + _substitute_log(repo, s, tmp_path, ["something else entirely"]) + + html_out = render.render_html(M.verify(s, repo)) + assert "WITNESS-UNAVAIL" in html_out + assert "badge witness_unavailable" in html_out # styled, not bare ink + assert "ALL RECORDED-EXACT" not in html_out + assert "0/1 claims evidence-bound" in html_out + assert "VERIFIED" not in html_out and "PROVEN" not in html_out.upper() + + +def test_strict_never_accepts_witness_unavailable(): + """`all_verified`'s membership set is the one thing --strict rests on. + + Widening it to admit this state would make --strict pass on a claim whose + evidence nobody has seen, which is the exact lie the state exists to stop. + """ + report = M.VerifyReport( + commit="c" * 40, + tree="t" * 40, + resolved_by="commit", + results=[ + GradeResult( + Claim(ctype="tests-pass", label="x", event_indices=(0,)), + "witness-unavailable", + reason="sealed as tree-exact at commit cccccccccccc; " + "witness ledger unavailable (entry hash mismatch at index 0)", + sealed_grade="tree-exact", + ) + ], + coverage={}, + ) + assert not report.all_verified + assert report.worst_status == "witness-unavailable" diff --git a/tests/test_gitplumbing.py b/tests/test_gitplumbing.py index d4d2348..72818f8 100644 --- a/tests/test_gitplumbing.py +++ b/tests/test_gitplumbing.py @@ -2,6 +2,7 @@ from __future__ import annotations +import inspect import re from pathlib import Path @@ -100,6 +101,57 @@ def test_ledger_excluded_from_digest_when_gitignored(repo: Path): assert before is not None and before == after +def _archive(repo: Path) -> None: + """A rotated ledger sitting under the archive root, in its usual shape.""" + d = repo / gp.ARCHIVE_DIRNAME / "unit-01" / ".didrun" + d.mkdir(parents=True) + (d / "session.log").write_text("rotated\n") + + +def test_archive_root_is_in_the_default_exclusion(): + """The exclusion is a default, not an opt-in — a caller passing nothing + still gets the archive root kept out.""" + default = inspect.signature(gp.tree_digest).parameters["exclude"].default + assert default == (gp.LEDGER_DIRNAME, gp.ARCHIVE_DIRNAME) + + +def test_archive_gitignored_digest_is_unchanged_by_the_new_exclusion(repo: Path): + """The compat criterion: where the archive root IS gitignored, v0.2 digests + a tree to the same id v0.1 did, because `add -A` already skipped it.""" + (repo / ".gitignore").write_text(".didrun/\n.didrun-history/\n") + _archive(repo) + v01 = gp.tree_digest(repo, exclude=(gp.LEDGER_DIRNAME,)) + v02 = gp.tree_digest(repo) + assert v01 is not None and v01 == v02 + + +def test_archive_not_gitignored_is_a_digest_semantics_change(repo: Path): + """Where the archive root is NOT gitignored, v0.2 and v0.1 disagree — and + v0.2 is the one that excludes it. Pinned, not hidden: this is the release + note. Without it, archiving a ledger moves the digest of the tree that + just archived it and stales the unit that did the archiving.""" + empty = gp.tree_digest(repo) # no archive root present yet + _archive(repo) + v01 = gp.tree_digest(repo, exclude=(gp.LEDGER_DIRNAME,)) + v02 = gp.tree_digest(repo) + assert v01 is not None and v02 is not None + assert v01 != v02 # the semantics change, stated + assert v02 == empty # and v0.2 is the side that ignores the archive + + +def test_ledger_dir_still_excluded_in_both_configs_with_the_archive_present(repo: Path): + """Adding a second exclusion must not weaken the first: `.didrun/` stays + out regardless of gitignore state, which is the guarantee grading rests on.""" + _archive(repo) + for ignore in ("", ".didrun/\n.didrun-history/\n"): + (repo / ".gitignore").write_text(ignore) + before = gp.tree_digest(repo) + (repo / ".didrun" / "objects").mkdir(parents=True, exist_ok=True) + (repo / ".didrun" / "objects" / "junk").write_text("noise\n") + after = gp.tree_digest(repo) + assert before is not None and before == after + + def test_tree_delta_reports_exact_path(repo: Path, tmp_path: Path): ledger_obj = tmp_path / "obj" a = gp.tree_digest(repo, ledger_obj) diff --git a/tests/test_grading_honesty.py b/tests/test_grading_honesty.py new file mode 100644 index 0000000..d005616 --- /dev/null +++ b/tests/test_grading_honesty.py @@ -0,0 +1,386 @@ +"""P1.1 — the grading ladder must not turn a failed subprocess into a fact. + +Two defects are pinned here: + + 1. ``tree_delta`` failed open to ``[]``, so when the evidence tree's objects + were unreachable (the archived-ledger case) ``grade`` rendered one of two + false sentences — "tree moved since evidence: 0 path(s) differ" (which also + violates the module's own stale-MUST-carry-the-delta invariant) and + "evidence tree equals sealed tree but command was not self-stable" (an + assertion about tree equality derived from a subprocess that failed). + 2. ``grade`` consulted ``_supporting_event`` before ``_witnessed_failure``, so + a claim bound to [failed, success] graded on the success and the witnessed + failure was invisible. +""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import gitplumbing as gp +from didrun.capture import run_wrapped +from didrun.claims import WIDENING_HINT_CAP, Claim, _widening_hint, grade +from didrun.ledger import Session + +# The two sentences the old fail-open path emitted. Neither may ever appear in +# a reason derived from a delta that was not computed. +FORBIDDEN = ("0 path(s) differ", "evidence tree equals sealed tree") + + +def _git(repo: Path, *a: str) -> None: + subprocess.run(["git", *a], cwd=str(repo), capture_output=True, text=True, check=True) + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _uncommitted_evidence(repo: Path) -> tuple[Session, str]: + """An event whose tree_after lives ONLY in the ledger's object dir. + + The dirt is never committed, so the tree object write-tree produced was + written under GIT_OBJECT_DIRECTORY into ``.didrun/objects`` and exists + nowhere else. Pointing a later grade at a different object dir reproduces + the archived-ledger case exactly: the event is present, the objects are not. + """ + s = _session(repo) + (repo / "scratch.py").write_text("x = 1\n") + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + sealed_tree = gp.commit_tree(repo, "HEAD") + ev = s.events()[0] + assert ev.tree_after is not None and ev.tree_after != sealed_tree + return s, sealed_tree + + +@pytest.mark.parametrize("pathspecs", [(), ("docs",)], ids=["no-pathspecs", "pathspecs"]) +def test_uncomputable_delta_grades_unknown_not_a_confident_sentence( + repo: Path, tmp_path: Path, pathspecs: tuple[str, ...] +): + """Objects gone -> `unknown`, and neither false sentence is emitted.""" + s, sealed_tree = _uncommitted_evidence(repo) + claim = Claim(ctype="tests-pass", label="t", event_indices=(0,), + pathspecs=pathspecs, declared_at_index=0) + + gone = tmp_path / "empty-objects" + gone.mkdir() + assert gp.tree_delta(repo, s.events()[0].tree_after, sealed_tree, gone) is None + + r = grade(claim, sealed_tree, s.events(), repo, gone) + assert r.grade == "unknown" + assert "not computable" in r.reason + for phrase in FORBIDDEN: + assert phrase not in r.reason + assert r.delta == [] # an uncomputed delta is never displayed + assert r.supporting_event_index == 0 + + # Control: the SAME claim over the SAME evidence with the objects present + # still grades stale. That is what proves the unknown above is caused by + # unavailability and not by the fixture being ungradeable in the first place. + reachable = grade(claim, sealed_tree, s.events(), repo, s.blobs.root) + assert reachable.grade == "stale" + assert "not computable" not in reachable.reason + assert reachable.delta, "the reachable control must carry a concrete delta" + + +def test_empty_delta_still_means_the_trees_are_equal(repo: Path): + """`[]` is "computed, and equal" — collapsing it into `None` breaks this. + + A command that mutates the tree is not self-stable, so it cannot be + tree-exact even when its resulting tree IS the sealed tree; the delta is + computed, comes back empty, and the stale-clean branch fires unchanged. + """ + s = _session(repo) + run_wrapped( + [sys.executable, "-c", "open('made.py', 'w').write('y = 2\\n')"], s, repo + ) + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "commit what the command produced") + sealed_tree = gp.commit_tree(repo, "HEAD") + + ev = s.events()[0] + assert not ev.self_stable() # the command moved the tree + assert ev.tree_after == sealed_tree # ...and landed exactly on the sealed one + + delta = gp.tree_delta(repo, ev.tree_after, sealed_tree, s.blobs.root) + assert delta is not None and delta == [] + + claim = Claim(ctype="tests-pass", label="t", event_indices=(0,), + pathspecs=("src",), declared_at_index=0) + r = grade(claim, sealed_tree, s.events(), repo, s.blobs.root) + assert r.grade == "stale" + assert r.reason == "evidence tree equals sealed tree but command was not self-stable" + + +def test_witnessed_failure_dominates_a_witnessed_success(repo: Path): + """[failed, success] grades FAILED, on the failure, not on the success. + + Reachable only through the library API today: the CLI's ``--event`` is + ``type=int``, not ``action="append"``, so every claim the CLI can build + binds exactly one event (0 of 846 corpus claims are multi-index). This is a + Low-severity honesty fix, not a headline. + """ + s = _session(repo) + run_wrapped([sys.executable, "-c", "import sys; sys.exit(1)"], s, repo) # 0: fails + run_wrapped([sys.executable, "-c", "import sys; sys.exit(0)"], s, repo) # 1: passes + claim = Claim(ctype="tests-pass", label="t", event_indices=(0, 1), declared_at_index=1) + + r = grade(claim, gp.commit_tree(repo, "HEAD"), s.events(), repo, s.blobs.root) + assert r.grade == "failed" + assert r.supporting_event_index == 0 + assert r.exit_code == 1 + assert r.reason == "the recorded command exited 1 — claim is not backed" + + +# --- every single-event rung of the ladder still fires exactly as before ------- + + +def _case_tree_exact(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + return s, Claim(ctype="tests-pass", label="t", event_indices=(0,), + declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +def _case_scope_exact(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + (repo / "src").mkdir() + (repo / "src" / "a.py").write_text("x = 1\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "add src") + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + (repo / "src" / "a.py").write_text("x = 2\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit src") + return s, Claim(ctype="tests-pass", label="t", event_indices=(0,), + pathspecs=("src",), declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +def _case_stale(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + (repo / "calc.py").write_text("def add(a, b):\n return a + b # later\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit") + return s, Claim(ctype="tests-pass", label="t", event_indices=(0,), + declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +def _case_unknown(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + run_wrapped([sys.executable, "-c", "print(2)"], s, repo) + return s, Claim(ctype="command-succeeded", label="c", event_indices=(1,), + declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +def _case_failed(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + run_wrapped([sys.executable, "-c", "import sys; sys.exit(1)"], s, repo) + return s, Claim(ctype="tests-pass", label="t", event_indices=(0,), + declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +LADDER = { + "tree-exact": (_case_tree_exact, "self-stable command ran against the sealed tree"), + "scope-exact": (_case_scope_exact, "all 1 change(s) within declared pathspecs"), + "stale": (_case_stale, "tree moved since evidence: 1 path(s) differ"), + "unknown": (_case_unknown, + "retroactive binding: claim declared at index 0 precedes its backing event 1"), + "failed": (_case_failed, "the recorded command exited 1 — claim is not backed"), +} + + +@pytest.mark.parametrize("expected_grade", sorted(LADDER)) +def test_single_event_claims_are_unaffected(repo: Path, expected_grade: str): + """The hoist and the fail-closed delta must move no single-event grade. + + Reason strings are asserted verbatim, not by substring: downstream + validators read them, so a reworded rung is a break even when the grade is + right. + """ + build, expected_reason = LADDER[expected_grade] + s, claim, sealed_tree = build(repo) + r = grade(claim, sealed_tree, s.events(), repo, s.blobs.root) + assert r.grade == expected_grade + assert r.reason == expected_reason + + +# --- P5.2: a scope miss names the widening set it already measured ----------- +# +# Measured over the corpus this tool was built against: 178 of 846 claims +# declared a pathspec and 14 of them lost `scope-exact` because the delta held +# exactly ONE out-of-scope path. The set that would have covered it was already +# in `delta`; the reason reported only how many paths missed, so an operator had +# to re-derive by hand something the tool had computed. + + +def _out_of_scope_case( + repo: Path, later: dict, pathspecs: tuple = ("src",) +) -> tuple[Session, Claim, str]: + """One event, then a commit that edits `later`'s paths only. + + The event's tree is the pre-edit HEAD tree, so grading against the post-edit + HEAD yields a delta of exactly `later`'s paths — every one of them outside + ``pathspecs``. + """ + s = _session(repo) + (repo / "src").mkdir() + (repo / "src" / "a.py").write_text("x = 1\n") + for rel, body in later.items(): + p = repo / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(body) + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "add tree") + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + for rel in later: + (repo / rel).write_text("later\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit outside src") + return ( + s, + Claim( + ctype="tests-pass", + label="t", + event_indices=(0,), + pathspecs=pathspecs, + declared_at_index=0, + ), + gp.commit_tree(repo, "HEAD"), + ) + + +def test_one_out_of_scope_path_names_the_pathspec_that_would_cover_it(repo: Path): + """The 14-claim case. Grade, delta and count prefix are unchanged; the + reason gains the arithmetic fact of what widening would take.""" + s, claim, sealed = _out_of_scope_case(repo, {"docs/status/X.md": "one\n"}) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + + assert r.grade == "stale", "the hint must not promote the grade" + assert r.grade != "scope-exact" + assert [c.path for c in r.delta] == ["docs/status/X.md"], "delta unchanged" + assert r.reason == ( + "1 change(s) outside declared pathspecs" + " — add --path docs/status/ to make this scope-exact" + ) + # The old prefix is preserved verbatim, which is what a downstream validator + # matching on it reads. + assert r.reason.startswith("1 change(s) outside declared pathspecs") + + +def test_the_named_widening_set_actually_makes_it_scope_exact(repo: Path): + """The un-gameable half: the hint is a claim about `_within`'s arithmetic, so + declaring exactly what it names must produce `scope-exact`. A hint that + printed a plausible-looking directory would pass the assertion above and + fail here.""" + s, claim, sealed = _out_of_scope_case(repo, {"docs/status/X.md": "one\n"}) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + body = r.reason.split(" — ", 1)[1] + suffix = " to make this scope-exact" + assert body.startswith("add ") and body.endswith(suffix), body + named = [ + tok.removeprefix("--path ") + for tok in body[len("add ") : -len(suffix)].split(", ") + if tok.startswith("--path ") + ] + assert named == ["docs/status/"] + + widened = Claim( + ctype=claim.ctype, + label=claim.label, + event_indices=claim.event_indices, + pathspecs=claim.pathspecs + tuple(named), + declared_at_index=claim.declared_at_index, + ) + r2 = grade(widened, sealed, s.events(), repo, s.blobs.root) + assert r2.grade == "scope-exact", r2.reason + + +def test_twelve_out_of_scope_paths_in_three_directories_dedup_to_three(repo: Path): + """Dedup and sort, with nothing elided. + + Twelve out-of-scope paths across three directories are three prefixes, so + the hint names three and says nothing about a remainder — there is none. A + `+0 more` suffix here would be a false statement about what was dropped. + """ + later = { + f"{d}/f{i}.md": "one\n" + for d in ("zeta", "docs/status", "alpha/beta") + for i in range(4) + } + s, claim, sealed = _out_of_scope_case(repo, later) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + + assert r.grade == "stale" + assert len(r.delta) == 12 + assert r.reason == ( + "12 change(s) outside declared pathspecs — add --path alpha/beta/, " + "--path docs/status/, --path zeta/ to make this scope-exact" + ) + assert "more)" not in r.reason + + +def test_more_directories_than_the_cap_are_counted_not_dropped(repo: Path): + """Above the cap the hint stays readable and stays honest: five prefixes and + the exact number withheld. Truncating without the count would understate the + widening the operator was being told about.""" + later = {f"d{i:02d}/f.md": "one\n" for i in range(8)} + s, claim, sealed = _out_of_scope_case(repo, later) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + + assert r.grade == "stale" + assert r.reason == ( + "8 change(s) outside declared pathspecs — add --path d00/, --path d01/, " + "--path d02/, --path d03/, --path d04/ (+3 more) to make this scope-exact" + ) + assert r.reason.count("--path ") == WIDENING_HINT_CAP + + +def test_a_top_level_file_is_named_by_itself_not_by_the_repository_root(repo: Path): + """A path with no parent directory is the one case where "use the parent" + would be a lie with consequences: the parent is the repository root, and + declaring the root puts EVERY path in scope, so the hint would be advising a + declaration that stops meaning anything. It names the file instead — the + smallest prefix that covers it and nothing else.""" + s, claim, sealed = _out_of_scope_case(repo, {"TOPLEVEL.md": "one\n"}) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + assert r.reason == ( + "1 change(s) outside declared pathspecs" + " — add --path TOPLEVEL.md to make this scope-exact" + ) + assert "--path ." not in r.reason and "--path /" not in r.reason + + +def test_the_hint_is_deterministic_and_order_independent(repo: Path): + """Determinism is a hard invariant of the trust path, and the delta's order + is git's. Sorting is what makes the same scope miss produce the same reason + twice.""" + paths = ["b/x.md", "a/y.md", "b/z.md", "a/y.md"] + first = _widening_hint(paths) + assert first == _widening_hint(reversed(paths)) + assert first == "add --path a/, --path b/ to make this scope-exact" + + +def test_a_scope_exact_reason_is_byte_identical_to_before(repo: Path): + """No hint where nothing missed. The in-scope sentence is the one archived + notes carry for their 163 inert pathspec claims.""" + s, claim, sealed = _case_scope_exact(repo) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + assert r.grade == "scope-exact" + assert r.reason == "all 1 change(s) within declared pathspecs" + assert "add --path" not in r.reason + + +def test_a_stale_claim_with_no_pathspecs_gains_no_hint(repo: Path): + """The hint belongs to the branch that measured an out-of-scope SET. A claim + that declared no scope has no widening arithmetic to report, and inventing + one would be the relevance inference the module forbids.""" + s, claim, sealed = _case_stale(repo) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + assert r.grade == "stale" + assert r.reason == "tree moved since evidence: 1 path(s) differ" + assert "--path" not in r.reason diff --git a/tests/test_interrupt_capture.py b/tests/test_interrupt_capture.py new file mode 100644 index 0000000..bc50791 --- /dev/null +++ b/tests/test_interrupt_capture.py @@ -0,0 +1,409 @@ +"""P2.3 — the recorder records the flight when the flight is interrupted. + +Two halves, and they meet at the same sentence: a lost event must cost one +event, never the ledger. + + - capture/CLI: SIGINT or SIGTERM during a wrapped command appends an event + that says what was NOT witnessed (no exit code, coverage "unobserved") + instead of guessing, keeps the output drained so far, and exits on the + conventional code with no traceback. + - ledger: an append is fsynced, a torn FINAL record is readable as a lost + event rather than as tamper, and `append` refuses to write past the tear + rather than burying it mid-log where it would break the chain forever. + +The signal legs drive the real CLI in a real process. Calling a handler +directly would test the handler, not the delivery; the delivery is the part +v0.1 got wrong. +""" + +from __future__ import annotations + +import json +import os +import signal +import subprocess +import sys +import threading +import time +from pathlib import Path + +import pytest + +from didrun import gitplumbing +from didrun.capture import CaptureInterrupted, run_wrapped +from didrun.claims import Claim, grade +from didrun.ledger import Event, LedgerError, Session, canonical_json + +# POSIX signal semantics. Windows has no SIGKILL and no process-group Ctrl-C +# with these mechanics; docs/COMPAT.md already scopes Tier-0 capture there. +pytestmark = pytest.mark.skipif( + os.name == "nt", reason="POSIX signal delivery semantics" +) + + +def _session(repo: Path, readonly: bool = False) -> Session: + return Session(repo / ".didrun", readonly=readonly) + + +def _wrapper_event(argv=("echo", "hi"), exit_code=0, **kw) -> Event: + return Event( + argv=tuple(argv), + cwd="/tmp", + env_fingerprint="abc123", + observed_via="wrapper", + coverage="complete", + exit_code=exit_code, + **kw, + ) + + +_CLI_BOOT = "import sys; from didrun.cli import main; sys.exit(main(sys.argv[1:]))" + +# Prints a marker, flushes, publishes its pid, then blocks. The pid file is +# written LAST and via a rename, so its existence proves both that the child is +# running and that the marker has already been handed to the pipe: the test +# never sends a signal into a race with the child's startup. +_STARTED_THEN_SLEEP = ( + "import os, sys, time\n" + "sys.stdout.write('MARKER-STARTED\\n')\n" + "sys.stdout.flush()\n" + "tmp = sys.argv[1] + '.tmp'\n" + "open(tmp, 'w').write(str(os.getpid()))\n" + "os.replace(tmp, sys.argv[1])\n" + "time.sleep(120)\n" +) + + +def _wait_for(predicate, timeout: float, interval: float = 0.02) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(interval) + return predicate() + + +def _start_wrapped_sleeper(repo: Path, tmp_path: Path): + """Spawn `didrun run -- ` and wait until it is running. + + Returns ``(proc, pid_file, child_argv)``. The child script lives outside the + repo so it cannot move a tree digest, and its source carries the marker text + so no assertion about the marker can pass off didrun's own summary line. + """ + script = tmp_path / "started_then_sleep.py" + script.write_text(_STARTED_THEN_SLEEP, encoding="utf-8") + pid_file = tmp_path / "child.pid" + child_argv = [sys.executable, str(script), str(pid_file)] + proc = subprocess.Popen( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "run", "--", *child_argv], + cwd=str(repo), + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + if not _wait_for(pid_file.exists, 60): + proc.kill() + proc.communicate(timeout=30) + pytest.fail("the wrapped child never started") + return proc, pid_file, child_argv + + +def _interrupt(repo: Path, tmp_path: Path, signum: int): + """Run a blocking child under didrun, signal DIDRUN, collect the outcome.""" + proc, _pid_file, child_argv = _start_wrapped_sleeper(repo, tmp_path) + proc.send_signal(signum) + out, err = proc.communicate(timeout=120) + return proc.returncode, out, err, child_argv + + +# --- 1 + 4: the event is recorded, and the exit code is conventional ---------- + + +@pytest.mark.parametrize( + "signame,expected_code", + [("SIGINT", 130), ("SIGTERM", 143)], +) +def test_an_interrupt_records_an_unobserved_event_and_exits_cleanly( + repo: Path, tmp_path: Path, signame: str, expected_code: int +): + signum = getattr(signal, signame) + code, out, err, child_argv = _interrupt(repo, tmp_path, signum) + + assert code == expected_code, err.decode("utf-8", "replace") + # v0.1's whole report was a traceback. The line that replaces it has to be + # both present and machine-greppable. + assert b"Traceback" not in err + assert b"KeyboardInterrupt" not in err + assert f"interrupted by {signame}".encode("ascii") in err + + entries = list(_session(repo, readonly=True).entries()) + assert len(entries) == 1, "the flight recorder lost the flight" + ev = entries[0].event + assert ev.observed_via == "wrapper" + assert ev.coverage == "unobserved" + assert ev.exit_code is None, "nobody witnessed an exit; recording one is a lie" + assert ev.argv == tuple(child_argv), "the in-flight argv is what was recorded" + # The recorded index is what the operator was told to look at. + assert f"recorded event {entries[0].index}".encode("ascii") in err + + ok, broken = _session(repo, readonly=True).verify_chain() + assert (ok, broken) == (True, None) + + +# --- 2: partial output survives (this is what P2.2 unlocked) ------------------ + + +def test_partial_output_survives_the_interrupt(repo: Path, tmp_path: Path): + """subprocess.run structurally could not do this. + + Its bare `except:` kills the child and re-raises with the buffers still + bound inside communicate(), where nothing can reach them. The pump drains + into a caller-owned accumulator, so the bytes the child had already written + are still there after the signal. + """ + _code, _out, _err, _argv = _interrupt(repo, tmp_path, signal.SIGINT) + + s = _session(repo, readonly=True) + ev = list(s.entries())[0].event + assert ev.stdout_blob is not None + assert b"MARKER-STARTED" in s.blobs.get(ev.stdout_blob) + + +# --- 3: it grades `unknown`, with no ladder change --------------------------- + + +def test_a_claim_bound_to_an_interrupted_event_grades_unknown( + repo: Path, tmp_path: Path +): + """The design criterion for the schema-free shape. + + `_supporting_event` skips the event (exit_code != 0) and `_witnessed_failure` + skips it (exit_code is None), so the correct grade falls out of the existing + ladder. The other half of this test — that claims.py was not edited — is a + `git diff --name-only` check in the unit's verification, because a comment + is not a check. + """ + _code, _out, _err, _argv = _interrupt(repo, tmp_path, signal.SIGINT) + + s = _session(repo, readonly=True) + events = [e.event for e in s.entries()] + claim = Claim( + ctype="command-succeeded", + label="the interrupted command", + event_indices=(0,), + declared_at_index=0, + ) + result = grade( + claim, + gitplumbing.commit_tree(repo, "HEAD"), + events, + repo, + s.blobs.root, + ) + assert result.grade == "unknown" + assert result.exit_code is None + # Never `failed`: an interrupted command was not caught lying. + assert "no witnessed successful command" in result.reason + + +# --- 5: the child dies, the parent lives — the normal path is untouched ------ + + +def test_killing_the_child_still_takes_the_normal_path(repo: Path, tmp_path: Path): + """A SIGKILLed child is a witnessed (negative) exit, not an interrupt. + + The interrupt path must not widen to swallow it: `-9` is evidence about the + command, and downgrading it to `unobserved` would lose a real observation. + """ + proc, pid_file, _argv = _start_wrapped_sleeper(repo, tmp_path) + os.kill(int(pid_file.read_text()), signal.SIGKILL) + _out, err = proc.communicate(timeout=120) + + assert b"interrupted by" not in err + ev = list(_session(repo, readonly=True).entries())[0].event + assert ev.coverage == "complete" + assert ev.exit_code == -signal.SIGKILL + + +# --- 6: a torn tail costs one event, not the ledger -------------------------- + + +def _torn_log(tmp_path: Path, records: int = 3, keep: int = 2) -> tuple: + """A log of ``records`` events truncated mid-way through record ``keep``. + + Returns ``(session, prefix_len, surviving_hashes)``. + """ + s = Session(tmp_path / ".didrun") + for i in range(records): + s.append(_wrapper_event(argv=("cmd", str(i)))) + lines = s.log_path.read_bytes().splitlines(keepends=True) + prefix = b"".join(lines[:keep]) + hashes = [json.loads(line)["entry_hash"] for line in lines[:keep]] + # Cut at a byte offset inside the record, not at a line boundary, or the + # log is merely SHORT and parses cleanly. + s.log_path.write_bytes(prefix + lines[keep][:37]) + return Session(tmp_path / ".didrun"), len(prefix), hashes + + +def test_a_torn_final_record_is_readable_and_is_not_tamper(tmp_path: Path): + s, prefix_len, hashes = _torn_log(tmp_path) + + entries = list(s.entries()) + assert [e.index for e in entries] == [0, 1] + assert [e.entry_hash for e in entries] == hashes + assert s.tail_truncated is True + assert s.tail_truncated_offset == prefix_len + + verdict = s.verify_chain_detail() + assert verdict.status == "intact" + assert verdict.first_broken_index is None + assert "torn at byte" in verdict.reason + assert str(prefix_len) in verdict.reason + assert s.verify_chain() == (True, None) + + +def test_a_malformed_middle_record_is_still_a_hard_error(tmp_path: Path): + """The tolerance is for the LAST line only. + + A hole with records after it is corruption, not an interrupted write, and + quietly skipping it would let a reader present a log with a missing middle + as a whole one. + """ + s = Session(tmp_path / ".didrun") + for i in range(3): + s.append(_wrapper_event(argv=("cmd", str(i)))) + lines = s.log_path.read_bytes().splitlines(keepends=True) + s.log_path.write_bytes(lines[0] + b"{not json at all}\n" + lines[2]) + + reader = Session(tmp_path / ".didrun") + with pytest.raises(LedgerError) as exc: + list(reader.entries()) + assert "corruption inside the log" in str(exc.value) + assert reader.tail_truncated is False + + +# --- 6b: the recorder survives the tear, and never writes past it ------------ + + +def test_a_torn_tail_does_not_kill_the_recorder_and_is_never_written_past( + tmp_path: Path, +): + """Change 7 without change 8 fixes the reader and leaves the writer dead. + + `_last()` must read past the tear (or every subsequent append raises, and + one lost event still kills the ledger); `append()` must refuse to WRITE + past it (or the partial bytes end up mid-log, unparseable forever, and one + lost event becomes a permanently broken chain). + """ + s, prefix_len, hashes = _torn_log(tmp_path) + before = s.log_path.read_bytes() + + assert s._last() == (1, hashes[1]) + + with pytest.raises(LedgerError) as exc: + s.append(_wrapper_event(argv=("cmd", "after"))) + assert "torn final record" in str(exc.value) + assert str(prefix_len) in str(exc.value), "the operator needs the byte offset" + assert s.log_path.read_bytes() == before, "a refusal must not mutate evidence" + + # The operator's recovery, at the offset the refusal named. + with s.log_path.open("r+b") as fh: + fh.truncate(prefix_len) + + fresh = Session(tmp_path / ".didrun") + entry = fresh.append(_wrapper_event(argv=("cmd", "after"))) + assert entry.index == 2 + assert fresh.tail_truncated is False + # No unparseable line anywhere, in the middle or at the end. + for line in fresh.log_path.read_bytes().splitlines(): + json.loads(line) + assert fresh.verify_chain() == (True, None) + assert [e.index for e in fresh.entries()] == [0, 1, 2] + + +# --- 7: fsync changes durability, not bytes ---------------------------------- + + +def test_append_fsyncs_the_record_before_returning(tmp_path: Path, monkeypatch): + """The durability half of the change, pinned by the call. + + The byte-identity test below is a regression guard: it passes on the + unfixed code by construction, because the point of fsync is that it changes + nothing on disk. Only this one fails when the fsync goes away, so both are + here and they are not the same test. + """ + real_fsync = os.fsync + fsynced: list = [] + + def spy(fd): + fsynced.append(fd) + return real_fsync(fd) + + monkeypatch.setattr(os, "fsync", spy) + s = Session(tmp_path / ".didrun") + s.append(_wrapper_event()) + + assert fsynced, "append returned before the record was durable" + assert list(s.entries())[0].event.argv == ("echo", "hi") + + +def test_fsync_adds_no_bytes_to_the_log(tmp_path: Path): + """The on-disk line is exactly what every previous release wrote. + + Pinned against a reconstruction rather than a golden file so it fails if the + record's SHAPE moves, not only if fsync appended something. + """ + s = Session(tmp_path / ".didrun") + events = [_wrapper_event(argv=("cmd", str(i))) for i in range(3)] + entries = [s.append(ev) for ev in events] + + expected = b"".join( + canonical_json( + { + "index": entry.index, + "prev_hash": entry.prev_hash, + "entry_hash": entry.entry_hash, + "event": ev.to_dict(), + } + ) + + b"\n" + for entry, ev in zip(entries, events) + ) + assert s.log_path.read_bytes() == expected + + +# --- 8: handlers are borrowed, never kept ------------------------------------ + + +def test_signal_handlers_are_restored_after_a_normal_run(repo: Path): + before = { + sig: signal.getsignal(sig) for sig in (signal.SIGINT, signal.SIGTERM) + } + ev = run_wrapped([sys.executable, "-c", "pass"], _session(repo), repo) + assert ev.exit_code == 0 + after = {sig: signal.getsignal(sig) for sig in (signal.SIGINT, signal.SIGTERM)} + assert after == before + + +def test_run_wrapped_from_a_worker_thread_records_normally(repo: Path): + """`signal.signal` raises off the main thread, and run_wrapped is a library + entry point an embedder may call from a worker. The guard makes that a + silent skip, not a ValueError — and the recorded event is unchanged.""" + outcome: dict = {} + + def work() -> None: + try: + outcome["event"] = run_wrapped( + [sys.executable, "-c", "print('hi')"], _session(repo), repo + ) + except BaseException as exc: # recorded, never swallowed + outcome["error"] = exc + + worker = threading.Thread(target=work) + worker.start() + worker.join(120) + assert not worker.is_alive() + assert "error" not in outcome, repr(outcome.get("error")) + + ev = outcome["event"] + assert (ev.observed_via, ev.coverage, ev.exit_code) == ("wrapper", "complete", 0) + assert not isinstance(outcome.get("error"), CaptureInterrupted) diff --git a/tests/test_interrupt_records_the_flight.py b/tests/test_interrupt_records_the_flight.py new file mode 100644 index 0000000..1b2aea1 --- /dev/null +++ b/tests/test_interrupt_records_the_flight.py @@ -0,0 +1,121 @@ +"""A signal must never cost the event for a command that actually ran. + +Regression cover for the lost-flight window between the drain and the append. +Two distinct mechanisms put an event at risk once the child is reaped: + +1. A second interrupt landing while didrun is digesting the tree and appending + (the recorder's own Python signal handling). +2. A terminal Ctrl-C reaching the whole foreground process group, which kills + the ``git`` the after-digest shells out to -- even when the child had + already exited cleanly and the drain saw no signal at all. + +Both used to end the process with no record of a command whose side effects +are on disk. The event is worth more than the after-digest: `None` grades +`unknown`, which is honest, while no event at all says the command never ran. +""" + +import os +import signal +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from didrun import gitplumbing +from didrun.capture import run_wrapped +from didrun.ledger import Session + + +def _git(repo, *args): + subprocess.run(["git", *args], cwd=repo, check=True, capture_output=True) + + +@pytest.fixture() +def repo(tmp_path): + _git(tmp_path, "init", "-q", ".") + _git(tmp_path, "config", "user.email", "t@t.test") + _git(tmp_path, "config", "user.name", "t") + (tmp_path / "f.txt").write_text("x\n") + _git(tmp_path, "add", "-A") + _git(tmp_path, "commit", "-qm", "init") + return tmp_path + + +def test_a_failing_after_digest_still_records_the_event(repo, monkeypatch): + """The mechanism behind the process-group kill, made deterministic. + + Rather than racing a real signal, fail the digest the way a killed git + fails it. The event must still land, tree_after must be None, and coverage + must say the tree was not witnessed instead of claiming "complete". + """ + session = Session(repo / ".didrun") + + real = gitplumbing.tree_digest + calls = {"n": 0} + + def flaky(*a, **kw): + calls["n"] += 1 + if calls["n"] > 1: # tree_before succeeds, tree_after dies + raise gitplumbing.GitError("git rev-parse --git-path index failed (-2): ") + return real(*a, **kw) + + monkeypatch.setattr(gitplumbing, "tree_digest", flaky) + + event = run_wrapped([sys.executable, "-c", "print('ran')"], session, repo=repo) + + entries = list(session.entries()) + assert len(entries) == 1, "a witnessed command left no event" + assert event.exit_code == 0, "the command completed; its exit code is known" + assert event.tree_after is None, "an unwitnessed tree must be None, not a guess" + assert event.coverage == "observed-text-only", ( + "claiming 'complete' would assert the tree was witnessed when it was not" + ) + assert session.verify_chain()[0], "the chain must stay intact" + + +def test_second_interrupt_during_the_record_phase_does_not_lose_the_flight(repo): + """End to end, through the CLI, against the real double-Ctrl-C window. + + The first signal ends the child; the second lands while didrun is + digesting and appending. Before the fix this window ran ~30-140ms wide and + lost the event outright. + """ + didrun = Path(sys.executable).parent / "didrun" + if not didrun.exists(): # pragma: no cover - depends on install layout + pytest.skip("didrun console script not present in this environment") + + (repo / "child.py").write_text( + "import time\n" + "for i in range(40):\n" + " print(i, flush=True)\n" + " time.sleep(0.05)\n" + ) + + lost = [] + for gap_ms in (30, 60, 90, 120): + for stale in repo.glob(".didrun/session.log"): + stale.unlink() + proc = subprocess.Popen( + [str(didrun), "run", "--", sys.executable, "child.py"], + cwd=repo, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + time.sleep(0.6) + for delay in (0.0, gap_ms / 1000.0): + time.sleep(delay) + try: + os.killpg(proc.pid, signal.SIGINT) + except (ProcessLookupError, PermissionError): # pragma: no cover + pass + proc.wait(timeout=60) + + log = repo / ".didrun" / "session.log" + lines = [l for l in log.read_text().splitlines() if l.strip()] if log.exists() else [] + if not lines: + lost.append(gap_ms) + + assert not lost, f"interrupt lost the flight at gaps (ms): {lost}" diff --git a/tests/test_ledger.py b/tests/test_ledger.py index e50ce04..b6194c0 100644 --- a/tests/test_ledger.py +++ b/tests/test_ledger.py @@ -2,6 +2,8 @@ from __future__ import annotations +import json +import os import subprocess import sys import textwrap @@ -10,12 +12,14 @@ import pytest from didrun.ledger import ( + GENESIS_HASH, Event, Session, BlobStore, LedgerError, canonical_json, sha256_hex, + unknown_event_keys_of, ) @@ -39,6 +43,42 @@ def test_blob_roundtrip_and_dedup(tmp_path: Path): assert store.get(d1) == b"hello" +def test_two_writers_of_one_blob_do_not_collide_on_a_temp_name( + tmp_path: Path, monkeypatch +): + """P2.4 — concurrent recorders store the same blob constantly. + + An empty stderr is one digest every event shares, so two `didrun run` + processes writing at once is the normal case, not the exotic one. This is + the crash, driven deterministically instead of hoped for: writer A is + suspended at its rename, writer B completes the SAME blob inside that + window, and A resumes. With a shared ``.tmp`` name B's rename has + already consumed A's temp and A dies with FileNotFoundError. + + Measured before the fix on an eight-way concurrent `didrun run`: 2 of 6 + attempts lost a process to exactly this traceback. + """ + root = tmp_path / "obj" + store = BlobStore(root) + data = b"identical bytes from both writers" + real_replace = os.replace + interleaved = [] + + def replace_with_a_second_writer_inside(src, dst): + if not interleaved: + interleaved.append(True) + BlobStore(root).put(data) + return real_replace(src, dst) + + monkeypatch.setattr(os, "replace", replace_with_a_second_writer_inside) + + assert store.put(data) == sha256_hex(data) + assert interleaved == [True], "the second writer never ran; the test is vacuous" + assert store.get(sha256_hex(data)) == data + # No temp residue survives a completed pair of writes. + assert [p.name for p in root.iterdir()] == [sha256_hex(data)] + + def test_blob_corruption_detected(tmp_path: Path): store = BlobStore(tmp_path / "obj") d = store.put(b"payload") @@ -124,3 +164,145 @@ def test_self_stability_precondition(): assert ev.self_stable() ev2 = _wrapper_event(tree_before="aaa", tree_after="bbb") assert not ev2.self_stable() + + +# --- P0.2 — the frozen v1 preimage, unknown keys, unknown versions ----------- + + +def _records(session: Session) -> list[dict]: + return [json.loads(l) for l in session.log_path.read_text().splitlines() if l.strip()] + + +def _rewrite(session: Session, records: list[dict]) -> None: + session.log_path.write_bytes(b"".join(canonical_json(r) + b"\n" for r in records)) + + +def test_v1_record_on_disk_carries_no_preimage_version_key(tmp_path: Path): + """A v1 record must stay byte-shaped exactly as every previous release + wrote it, so an older binary keeps reading a ledger this one produced.""" + s = Session(tmp_path / ".didrun") + s.append(_wrapper_event()) + rec = _records(s)[0] + assert set(rec) == {"index", "prev_hash", "entry_hash", "event"} + + +def test_archived_shape_record_round_trips(tmp_path: Path): + """A record with no preimage_version key — i.e. every archived record — + reads back, recomputes, and grades intact. + + The expected hash is spelled out here with the pre-P0.2 preimage rather + than borrowed from compute_hash(), so this fails if the projection ever + stops being byte-identical to the whole event body at version 1. + """ + s = Session(tmp_path / ".didrun") + body = _wrapper_event(argv=("cmd", "0")).to_dict() + stored = sha256_hex( + GENESIS_HASH.encode("ascii") + + canonical_json({"index": 0, "prev": GENESIS_HASH, "event": body}) + ) + _rewrite( + s, + [{"index": 0, "prev_hash": GENESIS_HASH, "entry_hash": stored, "event": body}], + ) + + entries = list(s.entries()) + assert len(entries) == 1 + assert entries[0].preimage_version == 1 + assert entries[0].unknown_event_keys == () + assert entries[0].event.argv == ("cmd", "0") # argv tuple-ization survives + assert entries[0].compute_hash() == stored + assert s.verify_chain_detail().status == "intact" + assert s.verify_chain() == (True, None) + + +def test_unknown_event_key_is_readable_but_unverifiable(tmp_path: Path): + """A ledger written by a newer binary must stay readable, and must not be + reported as tamper — but must never be reported as intact either.""" + s = Session(tmp_path / ".didrun") + s.append(_wrapper_event()) + recs = _records(s) + recs[0]["event"]["umask"] = "0022" + _rewrite(s, recs) + + # Readable, not fatal: no TypeError out of cls(**d). + ev = Event.from_dict(recs[0]["event"]) + assert ev.argv == ("echo", "hi") + assert unknown_event_keys_of(recs[0]["event"]) == ("umask",) + + entries = list(s.entries()) + assert entries[0].unknown_event_keys == ("umask",) + # This is exactly why unverifiable has to exist: the projection cannot see + # the extra key, so the recomputed hash still matches and a hash check + # alone would call this chain intact. + assert entries[0].compute_hash() == entries[0].entry_hash + + verdict = s.verify_chain_detail() + assert verdict.status == "unverifiable" + assert verdict.unverifiable_index == 0 + assert verdict.first_broken_index is None + assert "umask" in verdict.reason + + ok, idx = s.verify_chain() + assert ok is False + assert idx == 0 + + +def test_unknown_preimage_version_fails_closed(tmp_path: Path): + s = Session(tmp_path / ".didrun") + for i in range(3): + s.append(_wrapper_event(argv=("cmd", str(i)))) + recs = _records(s) + recs[1]["preimage_version"] = 2 + _rewrite(s, recs) + + entries = list(s.entries()) + assert entries[1].preimage_version == 2 + with pytest.raises(LedgerError): + entries[1].compute_hash() + + verdict = s.verify_chain_detail() + assert verdict.status == "unverifiable" + assert verdict.unverifiable_index == 1 + assert verdict.first_broken_index is None + assert "2" in verdict.reason + + +def test_verify_chain_reports_rather_than_raises_on_unknown_version(tmp_path: Path): + """compute_hash now raises on an unknown version. A reader that crashes is + worse than one that reports, so the boolean wrapper must still return.""" + s = Session(tmp_path / ".didrun") + for i in range(3): + s.append(_wrapper_event(argv=("cmd", str(i)))) + recs = _records(s) + recs[1]["preimage_version"] = 2 + _rewrite(s, recs) + + ok, idx = s.verify_chain() # must not raise + assert ok is False + assert idx == 1 + + +def test_mutated_event_body_is_broken_not_unverifiable(tmp_path: Path): + """Tamper inside a known field is still tamper — unverifiable must not + become a laundering route for a mutated body.""" + s = Session(tmp_path / ".didrun") + for i in range(5): + s.append(_wrapper_event(argv=("cmd", str(i)))) + recs = _records(s) + recs[1]["event"]["argv"] = ["cmd", "tampered"] + _rewrite(s, recs) + + verdict = s.verify_chain_detail() + assert verdict.status == "broken" + assert verdict.first_broken_index == 1 + assert verdict.unverifiable_index is None + + ok, idx = s.verify_chain() + assert ok is False + assert idx == 1 + + +def test_empty_chain_verdict(tmp_path: Path): + s = Session(tmp_path / ".didrun") + assert s.verify_chain_detail().status == "empty" + assert s.verify_chain() == (True, None) diff --git a/tests/test_ledger_permissions.py b/tests/test_ledger_permissions.py new file mode 100644 index 0000000..d886fea --- /dev/null +++ b/tests/test_ledger_permissions.py @@ -0,0 +1,266 @@ +"""P2.1 — the ledger root is private, and read-only commands do not create one. + +The criterion here is deliberately about the ROOT, not about files. The ledger's +``objects/`` directory is populated by git, twice per ``didrun run``, through the +``GIT_OBJECT_DIRECTORY`` redirect, at git's own modes — so "every file is 0600" +is not a property this package can hold. 0700 on the root is, and it is what +makes another uid unable to reach the files through the directory. +""" + +from __future__ import annotations + +import os +import stat +from pathlib import Path + +import pytest + +from didrun import gitplumbing +from didrun.cli import main +from didrun.ledger import Event, LedgerError, Session + + +def _mode(path: Path) -> int: + return stat.S_IMODE(path.stat().st_mode) + + +def _wrapper_event(argv=("echo", "hi"), exit_code=0, **kw) -> Event: + return Event( + argv=tuple(argv), + cwd="/tmp", + env_fingerprint="abc123", + observed_via="wrapper", + coverage="complete", + exit_code=exit_code, + **kw, + ) + + +@pytest.fixture +def umask(request): + """Run a test under a specific umask and restore it afterwards.""" + previous = os.umask(request.param) + try: + yield request.param + finally: + os.umask(previous) + + +# --- 1. the criterion: an existing 0755 ledger is re-tightened --------------- + +def test_existing_0755_ledger_is_retightened_on_open(tmp_path: Path): + # A ledger a v0.1 binary created: the directory already exists at 0755. + root = tmp_path / ".didrun" + root.mkdir() + os.chmod(root, 0o755) + assert _mode(root) == 0o755 + + Session(root) + + # mkdir(mode=0o700, exist_ok=True) does NOT re-tighten an existing + # directory, so an implementation that only passes mode= fails here. + assert _mode(root) == 0o700 + + +def test_fresh_ledger_root_is_private(tmp_path: Path): + root = tmp_path / ".didrun" + Session(root) + assert _mode(root) == 0o700 + + +# --- 2. umask independence, and the files this version creates --------------- + +@pytest.mark.parametrize("umask", [0o022, 0o002, 0o077], indirect=True) +def test_root_and_own_files_are_private_under_any_umask( + tmp_path: Path, umask: int, monkeypatch +): + from didrun import manifest as _manifest + from didrun.claims import Claim + + root = tmp_path / f".didrun-{umask:03o}" + session = Session(root) + session.append(_wrapper_event()) + _manifest.declare_claim( + session, + Claim(ctype="tests-pass", label="t", event_indices=(0,), declared_at_index=0), + ) + _manifest._record_seal(session, "c" * 40, "t" * 40, 1) + + assert _mode(root) == 0o700 + for name in ("session.log", "claims.jsonl", "seals.jsonl", ".gitignore"): + assert _mode(root / name) == 0o600, name + + # Deliberately NOT asserted: modes under objects/. git writes into that + # directory itself via GIT_OBJECT_DIRECTORY (gitplumbing.tree_digest), at + # git's modes, on every recorded command — so any assertion here would be + # about git's umask handling, not didrun's, and would be repaired-then-undone + # by the next `didrun run`. The root's 0700 is the control (red-team A2.4). + + +# --- 2b. the inherited-file bound, pinned rather than assumed ---------------- + +def test_inherited_v01_files_keep_their_modes_only_the_root_is_repaired( + tmp_path: Path, +): + root = tmp_path / ".didrun" + root.mkdir() + log = root / "session.log" + gitignore = root / ".gitignore" + log.write_bytes(b"") + gitignore.write_text("*\n", encoding="ascii") + os.chmod(log, 0o644) + os.chmod(gitignore, 0o644) + os.chmod(root, 0o755) + gitignore_before = gitignore.read_bytes() + mtime_before = gitignore.stat().st_mtime_ns + + Session(root) + + # This is the DELIBERATE BOUND of Session.harden, not an oversight: the + # root is the control, the files are not retro-fixed. O_CREAT's mode + # applies at creation only, and .gitignore is written only when absent, so + # a ledger an older binary created keeps 0644 on those files forever. + # Nobody may read the 0600 assertions in the test above as a general + # "the files are 0600" guarantee. + assert _mode(root) == 0o700 + assert _mode(log) == 0o644 + assert _mode(gitignore) == 0o644 + assert gitignore.read_bytes() == gitignore_before + assert gitignore.stat().st_mtime_ns == mtime_before + + +# --- the two guards on the chmod -------------------------------------------- + +def test_a_ledger_owned_by_another_uid_is_warned_about_not_chmodded( + tmp_path: Path, monkeypatch, capsys +): + root = tmp_path / ".didrun" + root.mkdir() + os.chmod(root, 0o755) + # Cannot create a foreign-owned directory without root, so move the OTHER + # side of the comparison: this process reports a euid the directory is not + # owned by. The branch under test is identical. + monkeypatch.setattr(os, "geteuid", lambda: root.stat().st_uid + 1) + + Session(root) + + assert _mode(root) == 0o755, "chmodded a directory this process does not own" + err = capsys.readouterr().err + assert "not tightening" in err and "0700" in err + + +def test_missing_geteuid_does_not_crash_the_windows_path(tmp_path: Path, monkeypatch): + # os.geteuid does not exist on Windows. The ownership check is skipped + # there rather than raising AttributeError on every Session open. + root = tmp_path / ".didrun" + root.mkdir() + os.chmod(root, 0o755) + monkeypatch.delattr(os, "geteuid", raising=False) + + Session(root) + + # POSIX chmod still works here, so the mode moves; the point of the test is + # that the absent attribute is a skipped check, not an exception. + assert _mode(root) == 0o700 + + +# --- 3. a read-only open creates nothing ------------------------------------ + +def test_readonly_session_creates_no_ledger(tmp_path: Path): + root = tmp_path / ".didrun" + session = Session(root, readonly=True) + assert not root.exists() + # And it still answers read questions about the ledger that is not there. + assert list(session.entries()) == [] + assert session.events() == [] + assert session.verify_chain() == (True, None) + assert not root.exists() + + +def test_readonly_session_refuses_to_write(tmp_path: Path): + session = Session(tmp_path / ".didrun", readonly=True) + with pytest.raises(LedgerError): + session.append(_wrapper_event()) + with pytest.raises(LedgerError): + session.blobs.put(b"payload") + assert not (tmp_path / ".didrun").exists() + + +def test_cmd_verify_on_a_repo_with_no_ledger_creates_nothing(repo: Path, capsys): + ledger = repo / ".didrun" + assert not ledger.exists() + + rc = main(["--repo", str(repo), "verify"]) + + assert rc == 0 + assert not ledger.exists(), "a read-only verify manufactured a ledger" + assert sorted(p.name for p in repo.iterdir()) == [".git", "calc.py"] + + +def test_cmd_show_session_on_a_repo_with_no_ledger_creates_nothing(repo: Path): + rc = main(["--repo", str(repo), "show", "--session"]) + assert rc == 0 + assert not (repo / ".didrun").exists() + + +# --- 4. a read-only open re-modes nothing ----------------------------------- + +def test_readonly_verify_does_not_remode_an_existing_ledger(repo: Path): + ledger = repo / ".didrun" + ledger.mkdir() + os.chmod(ledger, 0o755) + + rc = main(["--repo", str(repo), "verify"]) + + assert rc == 0 + # Pointing a read-only command at someone else's archive must not touch it. + assert _mode(ledger) == 0o755 + + +# --- 5. the chmod moves no tree digest -------------------------------------- + +def test_chmod_is_digest_neutral(repo: Path): + ledger = repo / ".didrun" + ledger.mkdir() + os.chmod(ledger, 0o755) + before = gitplumbing.tree_digest(repo) + assert before is not None + + Session(ledger) + + assert _mode(ledger) == 0o700 + after = gitplumbing.tree_digest(repo) + assert after == before + + +# --- 6. the new file-open path still appends a verifiable chain -------------- + +def test_append_and_chain_survive_the_private_open(tmp_path: Path): + root = tmp_path / ".didrun" + session = Session(root) + for i in range(4): + session.append(_wrapper_event(argv=("cmd", str(i)))) + ok, broke = session.verify_chain() + assert ok and broke is None + assert [e.index for e in session.entries()] == [0, 1, 2, 3] + + # Re-opening an existing ledger appends, never truncates. + reopened = Session(root) + reopened.append(_wrapper_event(argv=("cmd", "4"))) + assert len(list(reopened.entries())) == 5 + assert reopened.verify_chain() == (True, None) + assert _mode(root / "session.log") == 0o600 + + +def test_blob_temp_mode_does_not_leak_through_the_rename(tmp_path: Path): + # os.replace preserves the SOURCE mode, so a loose 0644 temp would become a + # 0644 blob. Under a permissive umask that is exactly what a plain + # write_bytes produced. + previous = os.umask(0o000) + try: + session = Session(tmp_path / ".didrun") + digest = session.blobs.put(b"secret output") + finally: + os.umask(previous) + assert _mode(session.blobs.root / digest) == 0o600 + assert session.blobs.get(digest) == b"secret output" diff --git a/tests/test_operator_authority.py b/tests/test_operator_authority.py new file mode 100644 index 0000000..ef91b23 --- /dev/null +++ b/tests/test_operator_authority.py @@ -0,0 +1,939 @@ +"""P5.1 — operator authority, and the exact size of what it buys. + +Every test here is written against one sentence, which is also the constant the +source prints on every surface: a `cited` authority records that a file with this +digest was cited for this scope. It is NOT a signature and NOT approval, because +**an agent can author its own authorization artifact in one line** — nothing in +this file, and nothing in didrun, distinguishes that from a human writing one. + +So the tests are deliberately asymmetric. The mechanism tests (bind, scope, +expiry, refusal) pin plumbing. The two that matter are the honesty ones: +``test_the_limitation_string_is_in_the_output``, which fails if any surface shows +a citation without the sentence that bounds it or reaches for a stronger word, +and ``test_retroactive_fabrication_is_detectable_and_only_that``, whose docstring +exists to stop a later reader from mistaking this feature for tamper-proofing. +""" + +from __future__ import annotations + +import hashlib +import json +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun import redact +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + +# Neutral, low-entropy, and spaced so the tokenizer cannot see one long run: this +# string exists to be searched for in the published note, not to trip a detector. +_ARTIFACT_BODY = "body of the cited artifact\n" + +# Structurally a github token, and synthetic: the shape the detectors match, never +# a real credential. Same construction as tests/test_detector_tiers.py. +_LABEL_TOKEN = "ghp_" + "F" * 36 + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _one_event(session: Session, repo: Path): + return run_wrapped([sys.executable, "-c", "print('ok')"], session, repo) + + +def _claim(session: Session, label: str = "suite") -> None: + M.declare_claim( + session, + Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0), + ) + + +def _artifact(repo: Path, name: str = "APPROVAL.md", body: str = _ARTIFACT_BODY) -> Path: + path = repo / name + path.write_text(body, encoding="utf-8") + return path + + +def _authority_lines(repo: Path) -> list: + path = repo / ".didrun" / "authority.jsonl" + if not path.exists(): + return [] + return [ln for ln in path.read_text(encoding="ascii").splitlines() if ln.strip()] + + +def _commit(repo: Path, name: str, body: str) -> None: + (repo / name).write_text(body, encoding="utf-8") + for args in (["add", name], ["commit", "-qm", f"add {name}"]): + subprocess.run(["git", *args], cwd=str(repo), check=True, capture_output=True) + + +def _ready(repo: Path) -> Session: + """A session with one recorded event and one claim bound to it.""" + s = _session(repo) + _one_event(s, repo) + _claim(s) + return s + + +# --- test 1: record and bind ------------------------------------------------- + +def test_authorize_records_one_line_and_the_seal_binds_it(repo: Path): + """The digest travels into the note. The artifact's bytes never do. + + An authorization artifact is exactly the kind of file that carries a + credential in passing, and the note is the one artifact that leaves the + machine — so the absence of the body is asserted directly, not inferred from + the fact that nothing in the code writes it. + """ + s = _ready(repo) + artifact = _artifact(repo) + record = M.declare_authority( + s, artifact, scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + + assert len(_authority_lines(repo)) == 1 + stored = json.loads(_authority_lines(repo)[0]) + assert stored == record + assert "artifact_bytes" in stored and stored["artifact_bytes"] == len( + _ARTIFACT_BODY.encode("utf-8") + ) + + m = M.seal(s, repo, allow_secrets=True) + assert len(m.authority) == 1 + bound = m.authority[0] + real = hashlib.sha256(artifact.read_bytes()).hexdigest() + assert bound["artifact_sha256"] == real + assert bound["grade"] == M.AUTHORITY_GRADE == "cited" + assert bound["principal_basis"] == "self-asserted" + + note = m.to_json().decode("ascii") + # The body, and every word of it long enough to be recognisable, is absent. + assert _ARTIFACT_BODY.strip() not in note + assert "cited artifact" not in note + assert real in note + + +def test_an_ordinary_seal_cites_nothing_even_when_an_authority_exists(repo: Path): + """The regression case, and the anti-rubber-stamp invariant in one. + + An authority on disk must not attach itself to a seal that performs no + exceptional transition. If it did, `cited` would appear on every note, which + is the routine-isation this design is shaped to avoid. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + m = M.seal(s, repo) + assert m.authority == [] + report = M.verify(s, repo) + assert report.authority == [] + assert render.authority_lines(report) == [] + assert M.AUTHORITY_LIMITATION not in render.render_verdict(report) + + +# --- test 2: the limitation string is in the output -------------------------- + +_STRONGER_WORDS = ("signed", "approved", "authorized by", "verified") + + +def _stronger_word_in(surface: str) -> list: + """Which upgrade words appear in a surface, matched as WORDS. + + Word boundaries and not substrings: the HTML carries a CSS comment about the + "MOST-designed states", and a substring search would report `signed` inside + `designed` — a false hit that would either fail this test forever or, worse, + get it weakened to a scoped search that stops covering the header line. + """ + lowered = surface.lower() + return [ + word + for word in _STRONGER_WORDS + if re.search(rf"\b{re.escape(word)}\b", lowered) + ] + + +def test_the_limitation_string_is_in_the_output(repo: Path): + """Every surface that shows a citation shows what it is not. + + Both halves are asserted: the exact constant is present, and none of the four + words that would upgrade `cited` into something it is not appears anywhere in + the rendered text. The second half is what catches a later well-meaning + rewrite of the header line. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + M.seal(s, repo, allow_secrets=True) + report = M.verify(s, repo) + assert len(report.authority) == 1 + + terminal = render.render_verdict(report) + html = render.render_html(report) + assert M.AUTHORITY_LIMITATION in terminal + assert M.AUTHORITY_LIMITATION in html + assert "CITED" in terminal and "CITED" in html + # The detector is checked against a surface that WOULD overclaim before it is + # trusted on the two that must not. A regex typo here would otherwise make + # the honesty assertion below vacuous and silently green forever. + assert _stronger_word_in("CITED authority — SIGNED and approved by drew") == [ + "signed", + "approved", + ] + assert _stronger_word_in("nothing here says it in those words") == [] + assert _stronger_word_in(terminal) == [] + assert _stronger_word_in(html) == [] + + +def test_the_rendered_limitation_comes_from_the_source_not_the_note(repo: Path): + """A limitation a note could edit would be no limitation at all. + + The note carries the sentence too, so a reader of raw JSON sees it. But the + renderer must print the SOURCE constant, because the note is data this + process did not compute — the same reading rule every other uncomputed field + gets. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + M.seal(s, repo, allow_secrets=True) + report = M.verify(s, repo) + report.authority[0]["limitation"] = "this authority is fine, ship it" + + terminal = render.render_verdict(report) + assert M.AUTHORITY_LIMITATION in terminal + assert "ship it" not in terminal + + +# --- test 3: --require-authority refuses a bare override --------------------- + +def test_require_authority_refuses_a_bare_override_and_names_the_scope( + repo: Path, capsys +): + s = _ready(repo) + code = cli.main(["--repo", str(repo), "seal", "--allow-secrets", "--require-authority"]) + err = capsys.readouterr().err + assert code == 2 + assert M.AUTHORITY_ALLOW_SECRETS in err + assert _note_body(repo) is None # nothing was published + + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + code = cli.main(["--repo", str(repo), "seal", "--allow-secrets", "--require-authority"]) + assert code == 0 + body = _note_body(repo) + assert body is not None + assert json.loads(body)["authority"][0]["scope"] == M.AUTHORITY_ALLOW_SECRETS + + +def test_require_authority_leaves_the_happy_path_alone(repo: Path): + """It gates the exceptional transitions and nothing else. + + A seal that overrides nothing and reseals nothing exercises no scope, so + there is no scope to authorise and the flag cannot turn into the + every-seal rubber stamp. + """ + s = _ready(repo) + m = M.seal(s, repo, require_authority=True) + assert m.authority == [] + + +def test_require_authority_refuses_a_reseal_without_a_reseal_authority(repo: Path): + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo, reseal=True, require_authority=True) + assert M.AUTHORITY_RESEAL in str(exc.value) + M.declare_authority(s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it") + m = M.seal(s, repo, reseal=True, require_authority=True) + assert [a["scope"] for a in m.authority] == [M.AUTHORITY_RESEAL] + + +# --- test 4: scope isolation ------------------------------------------------- + +def test_a_reseal_authority_does_not_satisfy_allow_secrets(repo: Path): + """One scope, one boundary. A citation is not a general permission.""" + s = _ready(repo) + M.declare_authority(s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it") + + m = M.seal(s, repo, allow_secrets=True) + assert m.authority == [] # the wrong scope binds to nothing + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo, allow_secrets=True, require_authority=True) + assert M.AUTHORITY_ALLOW_SECRETS in str(exc.value) + assert M.AUTHORITY_RESEAL not in str(exc.value).split("scope", 1)[1].split(".", 1)[0] + + +# --- test 5: expiry ---------------------------------------------------------- + +def test_an_authority_expires_after_the_seals_it_declared(repo: Path): + """`--expires-after-seals 1` covers the next seal and not the one after. + + The clock is seals, not wall time: a wall clock in the trust path is not + allowed here, and it would make a citation's liveness depend on how long a + build took. + """ + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_ALLOW_SECRETS, + label="drew ok", + expires_after_seals=1, + ) + + first = M.seal(s, repo, allow_secrets=True) + assert len(first.authority) == 1 + + # A second commit, so the second seal gets its own note and the narrowing + # guard is not what this test is measuring. + _commit(repo, "next.py", "x = 1\n") + _claim(s, label="second unit") + second = M.seal(s, repo, allow_secrets=True) + assert second.authority == [] + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo, allow_secrets=True, require_authority=True, reseal=True) + assert M.AUTHORITY_ALLOW_SECRETS in str(exc.value) + + +def test_a_wider_ttl_covers_the_seals_it_says_it_does(repo: Path): + """The complement of the test above: expiry counts, it does not just fire.""" + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_ALLOW_SECRETS, + label="drew ok", + expires_after_seals=2, + ) + assert len(M.seal(s, repo, allow_secrets=True).authority) == 1 + _commit(repo, "next.py", "x = 1\n") + _claim(s, label="second unit") + assert len(M.seal(s, repo, allow_secrets=True).authority) == 1 + _commit(repo, "third.py", "y = 2\n") + _claim(s, label="third unit") + assert M.seal(s, repo, allow_secrets=True).authority == [] + + +# --- test 6: never standing -------------------------------------------------- + +@pytest.mark.parametrize("scope", ["", "everything", "seal", "allow_secrets", None]) +def test_an_unscoped_or_unknown_authority_is_refused_at_authorize_time(repo, scope): + """There is no standing grant and no scope for an ordinary seal. + + `allow_secrets` (underscored) is in the parameter list on purpose: a + near-miss of a real scope must be refused rather than helpfully coerced, + because coercing it would silently widen what was authorised. + """ + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.declare_authority(s, _artifact(repo), scope=scope, label="drew ok") + assert "scope" in str(exc.value) + assert _authority_lines(repo) == [] + + +def test_the_cli_refuses_an_unknown_scope_with_exit_2(repo: Path): + """And it refuses it BECAUSE the scope is unknown, not because it cannot parse. + + The good invocation is asserted first on purpose. argparse exits 2 for any + unrecognised argument, including an `authorize` subcommand that does not + exist at all — so a bare "unknown scope exits 2" assertion passes on a + binary with no operator authority in it, which measured green on exactly + that during this unit's falsification pass. Pinning the accepted case first + is what makes the refusal below a statement about the SCOPE. + """ + _ready(repo) + artifact = _artifact(repo) + assert ( + cli.main( + [ + "--repo", str(repo), "authorize", + "--artifact", str(artifact), + "--scope", M.AUTHORITY_ALLOW_SECRETS, + "--label", "drew ok", + ] + ) + == 0 + ) + assert len(_authority_lines(repo)) == 1 + + with pytest.raises(SystemExit) as exc: + cli.main( + [ + "--repo", str(repo), "authorize", + "--artifact", str(artifact), + "--scope", "everything", + "--label", "drew ok", + ] + ) + assert exc.value.code == 2 + assert len(_authority_lines(repo)) == 1 # the refusal recorded nothing + + +def test_the_authorize_command_prints_the_digest_and_the_limitation(repo, capsys): + """`authorize`'s own output is where an operator forms their idea of this.""" + _ready(repo) + artifact = _artifact(repo) + assert ( + cli.main( + [ + "--repo", str(repo), "authorize", + "--artifact", str(artifact), + "--scope", M.AUTHORITY_ALLOW_SECRETS, + "--label", "drew ok", + ] + ) + == 0 + ) + out = capsys.readouterr().out + assert hashlib.sha256(artifact.read_bytes()).hexdigest() in out + assert M.AUTHORITY_LIMITATION in out + assert "self-asserted" in out + assert _stronger_word_in(out) == [] + + +def test_a_missing_or_empty_artifact_is_refused(repo: Path): + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.declare_authority( + s, repo / "nope.md", scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + assert "no such file" in str(exc.value) + + empty = repo / "EMPTY.md" + empty.write_text("", encoding="utf-8") + with pytest.raises(M.ManifestError) as exc: + M.declare_authority(s, empty, scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok") + assert "empty" in str(exc.value) + assert _authority_lines(repo) == [] + + +@pytest.mark.parametrize("ttl", [0, -1, True, "2", 1.5]) +def test_a_ttl_that_authorises_nothing_or_everything_is_refused(repo, ttl): + """No value means "never", and zero seals authorises nothing. + + `True` is in the list because `bool` is an `int` subclass: without the + `_is_index` reading it would be accepted as a TTL of 1, which is a real + number arriving from a place that never meant one. + """ + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_ALLOW_SECRETS, + label="drew ok", + expires_after_seals=ttl, + ) + assert "expires-after-seals" in str(exc.value) + + +def test_an_unlabelled_authority_is_refused(repo: Path): + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label=" " + ) + assert "--label" in str(exc.value) + + +# --- test 7: what the feature actually buys ---------------------------------- + +def test_retroactive_fabrication_is_detectable_and_only_that(repo: Path): + """Editing the cited artifact after the seal is detectable. That is ALL. + + This is the ONE property operator authority buys. It does NOT establish that + a human wrote the artifact, that anyone read it, or that anyone approved + anything: an agent can author its own authorization artifact in one line, + cite it, and seal. It is also NOT tamper-proofing — whoever can rewrite the + note can rewrite the digest inside it just as easily as they can rewrite a + grade. What is detectable is exactly this: a digest recorded in an EARLIER + sealed note no longer matches the file the note names. + + Do not read this test as evidence of anything beyond that sentence. + """ + s = _ready(repo) + artifact = _artifact(repo) + M.declare_authority( + s, artifact, scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + sealed = M.seal(s, repo, allow_secrets=True) + sealed_digest = sealed.authority[0]["artifact_sha256"] + assert sealed_digest == hashlib.sha256(artifact.read_bytes()).hexdigest() + + artifact.write_text(_ARTIFACT_BODY + "and one more line\n", encoding="utf-8") + assert hashlib.sha256(artifact.read_bytes()).hexdigest() != sealed_digest + + # The earlier note is unchanged and still names the pre-edit digest, which is + # what makes the mismatch visible at all. + published = json.loads(_note_body(repo)) + assert published["authority"][0]["artifact_sha256"] == sealed_digest + + # And the mismatch is not a verdict: the claims still grade exactly as they + # did. An authority is record, never verdict. + report = M.verify(s, repo) + assert report.all_verified is True + + +# --- test 8: compat ---------------------------------------------------------- + +def test_a_note_without_an_authority_key_reads_back_additively(): + """Every note published before P5.1 lacks the key. [] is what that means.""" + for version in (1, 2, 3): + body = json.dumps( + { + "version": version, + "commit": "a" * 40, + "tree": "b" * 40, + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + }, + sort_keys=True, + separators=(",", ":"), + ).encode("ascii") + m = M.Manifest.from_json(body) + assert m.authority == [] + assert json.loads(m.to_json())["authority"] == [] + + +def test_a_malformed_authority_block_in_a_note_is_read_as_no_authority(): + """A block that is not a list, or holds things that are not records. + + Read down to nothing rather than raised on: an authority block is record and + never verdict, so it cannot be the reason a readable note stops verifying. + """ + base = { + "version": M.MANIFEST_VERSION, + "commit": "a" * 40, + "tree": "b" * 40, + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + } + for bad in ("standing grant", 7, {"scope": "allow-secrets"}): + body = json.dumps( + dict(base, authority=bad), sort_keys=True, separators=(",", ":") + ).encode("ascii") + assert M.Manifest.from_json(body).authority == [] + + +def test_the_authority_version_contract_did_not_bump(repo: Path): + """Additive field, so no MANIFEST_VERSION bump (Locked Decision 9).""" + assert M.MANIFEST_VERSION == 3 + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + m = M.seal(s, repo, allow_secrets=True) + assert m.version == 3 + + +def test_an_unreadable_authority_record_is_dropped_not_trusted(repo: Path, capsys): + """A torn or hand-edited record costs a citation; it never grants one. + + Dropping is the refusing direction, which is why a scope that needed the + record is refused rather than quietly granted. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + path = repo / ".didrun" / "authority.jsonl" + good = path.read_text(encoding="ascii") + record = json.loads(good.strip()) + record["expires_after_seals"] = "forever" + path.write_text( + json.dumps(record, sort_keys=True, separators=(",", ":")) + "\n{tor\n", + encoding="ascii", + ) + with pytest.raises(M.ManifestError): + M.seal(s, repo, allow_secrets=True, require_authority=True) + assert "dropped 2 unreadable authority record(s)" in capsys.readouterr().err + + +def test_an_authority_from_the_future_binds_to_nothing(repo: Path): + """A record may not name a seal generation that has not happened. + + `authority.jsonl` is a local unhashed file, so this is validate-rather-than- + trust, not a forgery barrier: whoever can write the file can write a record + that passes. It closes the accident and the crude backdate, which is the same + bound every other uncomputed field in this package carries. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + path = repo / ".didrun" / "authority.jsonl" + record = json.loads(path.read_text(encoding="ascii").strip()) + record["declared_at_index"] = 9 + path.write_text( + json.dumps(record, sort_keys=True, separators=(",", ":")) + "\n", + encoding="ascii", + ) + assert M.seal(s, repo, allow_secrets=True).authority == [] + + +# --- the published bytes are the scanned bytes ------------------------------- + +def test_a_secret_in_an_authority_label_blocks_and_is_never_published(repo: Path): + """The authority block is published, so it is scanned AND redacted. + + Adding a published field that the export gate blocks on but the redactor + never touches would re-create the defect P3.2 closed: a token in a claim + label refused the seal and was then published verbatim on the override, + under a refusal message that had already promised a redacted artifact. + + The scope here is `reseal` rather than `allow-secrets`, and that is not + incidental: a bound authority is only published when the seal exercises its + scope, so `allow-secrets` is the one scope whose citation can never be + blocked — the flag that binds it is the same flag that overrides the block. + `reseal` is the case where the gate has something to say. + """ + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_RESEAL, + label=f"replace it {_LABEL_TOKEN}", + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo, reseal=True) + assert "github-token" in str(exc.value) + assert _note_body(repo) is None # the blocked seal published nothing + + m = M.seal(s, repo, reseal=True, allow_secrets=True) + note = m.to_json().decode("ascii") + assert _LABEL_TOKEN not in note + assert "redacted" in m.authority[0]["label"] + applied = m.authority[0]["redaction"]["fields"] + assert [f["field"] for f in applied] == ["authority[0].label"] + + +def _note_body(repo: Path): + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", "HEAD"], + cwd=str(repo), + capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +# --- the principal is outside text too --------------------------------------- + +# A high-entropy value that trips the notice tier rather than a named detector. +# Notice tier is the dangerous one here: it does not block, so a seal carrying it +# succeeds with `secrets_override` false and nothing asking the operator to look. +_NOTICE_TOKEN = "XMP7t2n3AGm5EhSc-sTMAMpLouiqyBfxD-__oQkcEbg" + + +def test_a_secret_in_the_principal_blocks_and_is_never_published(repo: Path): + """`principal` is `--principal` or `$USER` — outside text, and published. + + This is the same defect as the label case above, one field over: the export + gate scanned `principal` through the whole-manifest backstop and blocked on + it, the refusal said "re-run with --allow-secrets to export a REDACTED + artifact anyway", and the override then published the token verbatim into + the git note with `redaction.fields` empty. A refusal that promises + redaction has to deliver it for the field it refused over. + """ + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_RESEAL, + label="replace it", + principal=_LABEL_TOKEN, + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo, reseal=True) + assert "github-token" in str(exc.value) + assert _note_body(repo) is None + + m = M.seal(s, repo, reseal=True, allow_secrets=True) + note = m.to_json().decode("ascii") + assert _LABEL_TOKEN not in note + assert "redacted" in m.authority[0]["principal"] + applied = m.authority[0]["redaction"]["fields"] + assert [f["field"] for f in applied] == ["authority[0].principal"] + + +def test_the_same_secret_is_scrubbed_in_the_label_and_the_principal(repo: Path): + """One string in two adjacent fields must not come out two different ways. + + The notice tier made this the quiet case: the seal is not refused, nothing + is overridden, `secrets_override` stays false — and the value was scrubbed + in `label` and published raw in `principal`, in the same JSON object, in the + one artifact designed to leave the machine. + """ + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_RESEAL, + label=f"svc {_NOTICE_TOKEN}", + principal=_NOTICE_TOKEN, + ) + m = M.seal(s, repo, reseal=True) + + assert m.secrets_override is False # nothing was overridden + note = m.to_json().decode("ascii") + assert _NOTICE_TOKEN not in note + entry = m.authority[0] + assert "redacted" in entry["label"] and "redacted" in entry["principal"] + assert sorted(f["field"] for f in entry["redaction"]["fields"]) == [ + "authority[0].label", + "authority[0].principal", + ] + + +def test_every_authority_field_is_classified(repo: Path): + """The enumeration is not allowed to be silently incomplete. + + `_redact_authorities` names the fields it redacts by hand. Hand-written + enumerations go stale — `principal` was published verbatim for exactly that + reason — so this test fails the moment `declare_authority` or + `_bind_authorities` grows a field that neither table classifies, instead of + waiting for a reviewer to notice a new key in a note. + """ + s = _ready(repo) + record = M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it" + ) + produced = set(record) | {"grade", "limitation"} + classified = {name for name, _kind in M._AUTHORITY_TEXT_FIELDS} | set( + M._AUTHORITY_PASSTHROUGH_FIELDS + ) + assert produced <= classified, f"unclassified authority field(s): {produced - classified}" + + # And the published entry carries exactly what was classified, plus the + # redaction declaration — no key reaches a note unaccounted for. + m = M.seal(s, repo, reseal=True) + assert set(m.authority[0]) - {"redaction"} <= classified + + +def test_a_published_authority_field_the_redactor_forgot_still_blocks( + repo: Path, monkeypatch +): + """The whole-manifest backstop, exercised rather than asserted. + + `_scan_for_secrets` documents `export_bytes` as the backstop: "a field a + later version adds and forgets to redact is still in these bytes, so it + still blocks". Nothing tested that — removing the authority block from + `export_bytes` broke no test — so the sentence was a comment, not a + property. Here a field is added, vouched for as safe, and NOT redacted; the + seal must still refuse. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it" + ) + real_bind = M._bind_authorities + + def bind_with_a_new_field(session, scopes, seal_index): + bound = real_bind(session, scopes, seal_index) + for entry in bound: + entry["future_field"] = _LABEL_TOKEN + return bound + + monkeypatch.setattr(M, "_bind_authorities", bind_with_a_new_field) + monkeypatch.setattr( + M, + "_AUTHORITY_PASSTHROUGH_FIELDS", + M._AUTHORITY_PASSTHROUGH_FIELDS | {"future_field"}, + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo, reseal=True) + assert "github-token" in str(exc.value) + + +def test_an_unclassified_authority_field_is_dropped_not_published( + repo: Path, monkeypatch +): + """For a field nobody classified, the failing direction is to leave it out. + + The counterpart to the test above: that one covers a field a later version + vouches for, this one covers a field it never thought about. Dropping loses + a line from a note; publishing loses a credential. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it" + ) + real_bind = M._bind_authorities + + def bind_with_a_new_field(session, scopes, seal_index): + bound = real_bind(session, scopes, seal_index) + for entry in bound: + entry["unclassified_field"] = _LABEL_TOKEN + return bound + + monkeypatch.setattr(M, "_bind_authorities", bind_with_a_new_field) + # It still blocks, because the backstop sees the bound entry — but the point + # here is what the override publishes. + m = M.seal(s, repo, reseal=True, allow_secrets=True) + note = m.to_json().decode("ascii") + assert "unclassified_field" not in note + assert _LABEL_TOKEN not in note + + +# --- a note cannot fabricate lines in the block that renders it -------------- + +_AUTHORITY_RENDERED_FIELDS = ( + "scope", + "label", + "artifact_path", + "artifact_sha256", + "artifact_bytes", + "principal", + "expires_after_seals", +) + + +def test_a_note_cannot_inject_lines_into_the_authority_block(repo: Path, monkeypatch): + """A note arrives over `git fetch`, so every field of it is outside data. + + `artifact_bytes` and `expires_after_seals` were the two of the block's + fields that no one wrapped in `_sanitize`, because each use site spelled the + call out by hand. Both are numbers in a note didrun wrote, and neither is a + number in a note someone else edited — a newline in either one fabricated + whole verdict rows above the real table, carrying the words this module + exists to keep out of its own output. + + `NO_COLOR` is set so the ESC assertion means what it says: with colour on the + renderer emits its own escapes, and "no ESC in the output" could not tell + those apart from an injected one. + """ + monkeypatch.setenv("NO_COLOR", "1") + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + M.seal(s, repo, allow_secrets=True) + report = M.verify(s, repo) + honest = dict(report.authority[0]) + assert len(render.authority_lines(report)) == 4 + + payload = ( + "1\n = ALL RECORDED-EXACT (upstream verified)\n" + " = TREE-EXACT forged-row exit 0 reviewed and signed off\x1b[31m\x1b[0m" + ) + for name in _AUTHORITY_RENDERED_FIELDS: + entry = dict(honest) + entry[name] = payload + report.authority = [entry] + + lines = render.authority_lines(report) + assert len(lines) == 4, f"{name}: the block is no longer four lines" + assert not any("\n" in ln for ln in lines), f"{name}: newline survived" + assert not any("\x1b" in ln for ln in lines), f"{name}: ESC survived" + + verdict = render.render_verdict(report) + assert not re.search( + r"(?m)^\s*[=~!x?]?\s*(TREE-EXACT|ALL RECORDED-EXACT)\s+forged-row", verdict + ), f"{name}: fabricated a verdict row" + + # Not `_stronger_word_in` here: the payload smuggles "signed" and + # "verified" as CONTENT, and a note whose label really says "signed off + # by legal" is entitled to have that rendered as text. The invariant + # those words belong to is about didrun's own vocabulary — the token and + # the limitation, both pinned to source constants and covered above. + # What must not survive is the structure: no line break, no escape. + html_out = render.render_html(report) + assert "\x1b" not in html_out, f"{name}: ESC survived into HTML" + assert "forged-row" in html_out.replace("&#", ""), f"{name}: content lost" + + +def test_the_rendered_principal_basis_comes_from_the_source_not_the_note(repo: Path): + """The third field that states what a citation is worth, pinned like the other two. + + `grade` and `limitation` already come from source constants so a note cannot + restate the trust level. `principal_basis` qualifies the principal, and it + did not — a note supplying "verified-ssh-signature" printed + `principal drew (verified-ssh-signature)` two lines above the sentence + saying didrun cannot tell who wrote it, in one of the exact words + `_STRONGER_WORDS` forbids. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + M.seal(s, repo, allow_secrets=True) + report = M.verify(s, repo) + report.authority[0]["principal_basis"] = "verified-ssh-signature" + + terminal = render.render_verdict(report) + html_out = render.render_html(report) + for surface in (terminal, html_out): + assert "verified-ssh-signature" not in surface + assert M.AUTHORITY_PRINCIPAL_BASIS in surface + assert _stronger_word_in(terminal) == [] + assert _stronger_word_in(html_out) == [] + + +def test_the_published_principal_basis_comes_from_the_source_not_the_ledger(repo: Path): + """`authority.jsonl` is local, gitignored and unhashed, so it is outside data too. + + Whoever can write that file can write any basis into it. The note must carry + the basis this version can actually establish, not the one the file claims. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it" + ) + path = repo / ".didrun" / "authority.jsonl" + record = json.loads(_authority_lines(repo)[0]) + record["principal_basis"] = "verified-ssh-signature" + path.write_text(json.dumps(record) + "\n", encoding="ascii") + + m = M.seal(s, repo, reseal=True) + assert m.authority[0]["principal_basis"] == M.AUTHORITY_PRINCIPAL_BASIS + assert "verified-ssh-signature" not in m.to_json().decode("ascii") + + +def test_the_command_surfaces_cannot_be_line_injected_by_an_authority( + repo: Path, capsys, monkeypatch +): + """`didrun authorize` and `didrun seal` print these strings too. + + The verify block was the reported case, but it is not the only surface that + prints an authority's label: `seal` echoes the citation it just published, + reading the label out of `authority.jsonl`. Redaction is not sanitization — + a scrubbed label still carries whatever control characters surrounded the + secret — so the seal's own output could grow a verdict row it never + computed, from a local file that gets validate-rather-than-trust treatment. + """ + monkeypatch.setenv("NO_COLOR", "1") + forged = r"(?m)^\s*[=~!]?\s*TREE-EXACT\s+forged-row" + payload = "ok\n = TREE-EXACT forged-row exit 0 reviewed and signed off" + + s = _ready(repo) + artifact = _artifact(repo) + code = cli.main( + [ + "--repo", str(repo), "authorize", + "--artifact", str(artifact), + "--scope", M.AUTHORITY_RESEAL, + "--label", payload, + ] + ) + out = capsys.readouterr().out + assert code == 0 + assert not re.search(forged, out), "authorize fabricated a verdict row" + assert "^J" in out, "the newline was not caret-encoded" + + assert cli.main(["--repo", str(repo), "seal", "--reseal"]) == 0 + out = capsys.readouterr().out + assert not re.search(forged, out), "seal fabricated a verdict row" + assert "^J" in out, "the newline was not caret-encoded" diff --git a/tests/test_preimage_golden.py b/tests/test_preimage_golden.py new file mode 100644 index 0000000..4c90bdd --- /dev/null +++ b/tests/test_preimage_golden.py @@ -0,0 +1,125 @@ +"""P0.2 — the v1 chain preimage is frozen. Three assertions guard it. + +Every entry_hash ever written by didrun was computed over the v1 preimage. If +the preimage changes, every archived ledger reports tamper at index 0. The +literals below pin it. + +DO NOT REGENERATE THE LITERALS IN THIS FILE. If an assertion here fails, the +v1 chain preimage changed and every previously recorded entry_hash stopped +reproducing. That is the defect this file exists to catch — fix the code, or +define a *new* preimage version and leave version 1 alone. Recomputing a +literal to make the test pass destroys the only evidence that the freeze held. + +Three assertions, because one is not enough: + + (i) the golden entry_hash literal — catches any change to the hashed bytes; + (ii) _PREIMAGE_FIELDS[1] equals the frozen name list below — catches a field + added to the preimage list; + (iii) the frozen name list equals Event's dataclass fields — catches a field + added to Event. + +(ii) and (iii) together are what close the hole that set-equality alone leaves: +adding a field to *both* Event and _PREIMAGE_FIELDS[1] satisfies a check that +only compares those two to each other, while breaking every archived hash. + +Assertion (i) is deliberately written against the pre-P0.2 API surface +(``ChainEntry(index=..., prev_hash=..., event=...)`` and ``compute_hash()``), so +this file runs unchanged on the code as it stood *before* the preimage freeze +landed. That is what makes the golden literal falsifiable: it was generated +from pre-change code and must reproduce on post-change code. +""" + +from __future__ import annotations + +import dataclasses + +import pytest + +from didrun import ledger +from didrun.ledger import ChainEntry, Event, Session + +# --- the golden preimage ----------------------------------------------------- +# Generated from the code as it stood before the P0.2 edit. Synthetic values +# only: no real path, host, or digest appears here. + +GOLDEN_EVENT = { + "argv": ["pytest", "-q", "tests/"], + "cwd": "/w/repo", + "env_fingerprint": "0123456789abcdef", + "observed_via": "wrapper", + "coverage": "complete", + "exit_code": 0, + "started_at": 1000000.5, + "ended_at": 1000002.25, + "stdout_blob": "1" * 64, + "stderr_blob": "2" * 64, + "transcript_blob": None, + "tree_before": "3" * 40, + "tree_after": "3" * 40, + "submodule_dirty": False, +} +GOLDEN_INDEX = 7 +GOLDEN_PREV_HASH = "4" * 64 +GOLDEN_ENTRY_HASH = "99715b0cb8a7d3bef413a9d5c3692518b75ec5a5fddcf9bc7a7f6da333ee7d3c" + +# The v1 preimage field names, frozen as a literal. Not derived from anything. +V1_PREIMAGE_FIELD_NAMES = [ + "argv", + "coverage", + "cwd", + "ended_at", + "env_fingerprint", + "exit_code", + "observed_via", + "started_at", + "stderr_blob", + "stdout_blob", + "submodule_dirty", + "transcript_blob", + "tree_after", + "tree_before", +] + +# _PREIMAGE_FIELDS does not exist before P0.2. Assertion (i) is the evidence +# that crosses that boundary; (ii) can only run once the constant is there. +_HAS_PREIMAGE_FIELDS = hasattr(ledger, "_PREIMAGE_FIELDS") + + +def test_v1_chain_preimage_golden_hash(): + """(i) The hashed bytes of a v1 entry have not moved.""" + entry = ChainEntry( + index=GOLDEN_INDEX, + prev_hash=GOLDEN_PREV_HASH, + event=Event.from_dict(dict(GOLDEN_EVENT)), + ) + assert entry.compute_hash() == GOLDEN_ENTRY_HASH + + +@pytest.mark.skipif( + not _HAS_PREIMAGE_FIELDS, + reason="pre-P0.2 code has no _PREIMAGE_FIELDS; assertion (i) is the cross-boundary evidence", +) +def test_v1_preimage_field_list_matches_frozen_literal(): + """(ii) The declared v1 field list is exactly the frozen 14 names.""" + assert sorted(ledger._PREIMAGE_FIELDS[1]) == V1_PREIMAGE_FIELD_NAMES + + +def test_event_dataclass_matches_frozen_literal(): + """(iii) Event's field set is exactly the frozen 14 names. + + Runs on pre- and post-P0.2 code alike. Adding a field to Event fails here + even if the same field is also added to _PREIMAGE_FIELDS[1]. + """ + assert sorted(f.name for f in dataclasses.fields(Event)) == V1_PREIMAGE_FIELD_NAMES + + +def test_preimage_api_is_all_or_nothing(): + """The skip above must not be re-armable by deleting the constant. + + If this binary carries the P0.2 chain-verdict API, it must also carry both + preimage constants — so assertion (ii) cannot be silenced by removing + _PREIMAGE_FIELDS while keeping the rest of the freeze. + """ + if hasattr(Session, "verify_chain_detail"): + assert hasattr(ledger, "PREIMAGE_VERSION") + assert _HAS_PREIMAGE_FIELDS diff --git a/tests/test_projection_contract.py b/tests/test_projection_contract.py new file mode 100644 index 0000000..f04660f --- /dev/null +++ b/tests/test_projection_contract.py @@ -0,0 +1,349 @@ +"""P3.1 — the exported redaction projection is a reconstructible span partition. + +The criterion here is EXACT RECONSTRUCTION, not "the output's non-marker +characters are a subsequence of the input". The subsequence version was proposed +first and it does not detect the bug it targets: deletion always preserves the +subsequence property, so a redactor that deletes arbitrary text satisfies it. +`test_subsequence_criterion_is_insufficient` pins that, so the weaker check +cannot be reintroduced as an improvement. + +Every credential-shaped string below is synthetic and structural — a shape the +detectors match, never a real token. +""" + +from __future__ import annotations + +import random +import subprocess +import sys +from pathlib import Path + +from didrun import manifest as M +from didrun import redact +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + + +# The marker format is written out here rather than imported: `_REDACTION_MARKER` +# is a frozen part of the exported format, so the test should fail if it moves. +def _marker(kind: str) -> str: + return "«redacted:" + kind + "»" + + +def _reconstruct(source: str, result) -> str: + """Independent reimplementation of the reconstruction contract. + + Deliberately NOT `redact.reconstruct`: a test that calls the same helper the + source uses can only check that the helper agrees with itself. Kept slices + come from the OUTPUT, replaced slices from the source, and markers are + located by arithmetic — never searched for, because an argv is allowed to + contain the literal marker text. + """ + rebuilt = [] + out_i = 0 + src_i = 0 + for span in result.applied: + keep = span.start - src_i + assert keep >= 0, f"applied spans overlap or go backwards: {result.applied}" + rebuilt.append(result.text[out_i:out_i + keep]) + out_i += keep + marker = _marker(span.kind) + assert result.text[out_i:out_i + len(marker)] == marker, ( + f"no marker at offset {out_i} for span {span}" + ) + out_i += len(marker) + rebuilt.append(source[span.start:span.end]) + src_i = span.end + rebuilt.append(result.text[out_i:]) + return "".join(rebuilt) + + +# Both red-team repros verbatim, plus the nested case from the overlap finding. +# Synthetic token bodies; nothing here is a credential. +_GHP = "ghp_" + "A" * 36 +_SK = "sk-AbCdEf0123456789XyZwVuTsQq" + +CASES = [ + "run --key MY_TOKEN=" + _GHP + " && echo done", + 'sh -c "OPENAI_API_KEY=' + _SK + ' pytest -q" && rm -rf /tmp/x', + "MY_SECRET=sk-AAAAAAAAAAAAAAAAAAAA tail words here", + "plain text with nothing to hide", + "AKIAIOSFODNN7EXAMPLE at the start", + "trailing secret AKIAIOSFODNN7EXAMPLE", + "two MY_TOKEN=" + _GHP + " secrets API_SECRET=" + _SK + " in one string", + # A hostile argv containing the literal marker text. Reconstruction must + # still be exact: markers are located by offset, not by search. + "echo " + _marker("openai-key") + " MY_TOKEN=" + _GHP + " done", +] + + +def test_exact_reconstruction(): + """The criterion (red-team HIGH-7): substituting each marker back with the + source span the projection records reproduces the input byte for byte.""" + for source in CASES: + result = redact.redact(source) + assert _reconstruct(source, result) == source, f"lost content in: {source!r}" + + +def test_applied_spans_are_disjoint_and_ascending(): + for source in CASES: + spans = redact.redact(source).applied + for prev, nxt in zip(spans, spans[1:]): + assert prev.end <= nxt.start, f"overlapping partition in {source!r}: {spans}" + for span in spans: + assert 0 <= span.start < span.end <= len(source) + + +def test_length_identity(): + for source in CASES: + result = redact.redact(source) + widths = sum(s.end - s.start for s in result.applied) + markers = sum(len(_marker(s.kind)) for s in result.applied) + assert len(result.text) == len(source) - widths + markers, source + + +_BENIGN_WORDS = [ + "run", "make", "test", "--verbose", "-q", "build", "src/app.py", + "&&", "echo", "done", "pytest", "lint", "config.toml", "the", "unit", +] +_SECRET_SHAPES = [ + "AKIAIOSFODNN7EXAMPLE", + _GHP, + _SK, + "MY_TOKEN=" + _GHP, + "API_SECRET=hunter2taffylongvalue", + "xoxb-1234567890-abcdefghijkl", + "sk-proj-AbCdEf0123456789XyZwVuTsQq", +] + + +def test_property_reconstruction_and_length_over_seeds(): + """Random benign text with k injected secrets, over 200 FIXED seeds. + + Seeded per case and never from the global `random` state: determinism is a + hard invariant, and a property test that cannot be replayed is not evidence. + """ + for seed in range(200): + rng = random.Random(seed) + words = [rng.choice(_BENIGN_WORDS) for _ in range(rng.randint(4, 20))] + for _ in range(rng.randint(1, 4)): + words.insert(rng.randint(0, len(words)), rng.choice(_SECRET_SHAPES)) + source = " ".join(words) + result = redact.redact(source) + assert _reconstruct(source, result) == source, f"seed {seed}: {source!r}" + widths = sum(s.end - s.start for s in result.applied) + markers = sum(len(_marker(s.kind)) for s in result.applied) + assert len(result.text) == len(source) - widths + markers, f"seed {seed}" + + +def _is_subsequence(needle: str, haystack: str) -> bool: + it = iter(haystack) + return all(ch in it for ch in needle) + + +def test_subsequence_criterion_is_insufficient(): + """Why the criterion is exact reconstruction and not a subsequence check. + + v0.1 replaced overlapping spans end-first, so the outer span's stale end + offset ran past the shortened string and ate whatever followed. Below is + that measured output. The proposed subsequence assertion PASSES on it, + because deletion always preserves the subsequence property — which is why + it was replaced. Do not reintroduce it as a simplification. + """ + source = CASES[0] + v01_buggy_output = "run --key " + _marker("env-secret-assignment") + non_marker = v01_buggy_output.replace(_marker("env-secret-assignment"), "") + + assert _is_subsequence(non_marker, source) # the weaker criterion passes + assert "&& echo done" not in v01_buggy_output # on output missing 12 bytes + + # The criterion that actually catches it. + fake = redact.RedactionResult( + text=v01_buggy_output, + findings=[], + applied=[redact.AppliedSpan(10, 59, "env-secret-assignment", + "env-secret-assignment")], + ) + assert _reconstruct(source, fake) != source + + # And the current redactor does not produce it. + assert redact.redact(source).text != v01_buggy_output + + +def test_merge_labels_the_widest_finding_not_the_first(): + """Merge policy over a chain that no single finding covers. + + A(0,10) then B(5,30) then C(6,15): the merged extent is (0,30), which + belongs to no finding, and the label must track the widest contributor + (B) rather than whichever one opened the run. Hand-built findings, because + a three-way straddle like this is not reachable from the current detector + set — the policy still has to be right when it becomes reachable. + """ + findings = [ + redact.Finding("high-entropy", (0, 10), ""), + redact.Finding("high-entropy", (5, 30), ""), + redact.Finding("openai-key", (6, 15), ""), + ] + (span,) = redact.merge_findings(findings) + assert (span.start, span.end) == (0, 30) + assert span.kind == "high-entropy" + + # Same extents, but now the widest is the structured one. + findings[1] = redact.Finding("github-token", (5, 30), "") + (span,) = redact.merge_findings(findings) + assert span.kind == "github-token" + + # Exact-width tie: the structured detector wins over the entropy sweep. + (span,) = redact.merge_findings([ + redact.Finding("high-entropy", (0, 20), ""), + redact.Finding("aws-access-key", (0, 20), ""), + ]) + assert span.kind == "aws-access-key" + + +def test_nested_spans_merge_to_one_outermost(): + source = "MY_SECRET=sk-AAAAAAAAAAAAAAAAAAAA tail words here" + result = redact.redact(source) + assert len(result.applied) == 1, result.applied + span = result.applied[0] + assert (span.start, span.end) == (0, 33) + # The wider structured detector labels the merged span, not the inner one. + assert span.kind == "env-secret-assignment" + assert span.detector_id == "env-secret-assignment" + assert not span.withheld + assert result.text == _marker("env-secret-assignment") + " tail words here" + + +def test_sk_proj_keys_are_recalled(): + """Finding 18: `\\bsk-[A-Za-z0-9]{20,}\\b` missed every `sk-proj-` key.""" + kinds = {f.kind for f in redact.scan("sk-proj-AbCdEf0123456789XyZwVuTsQq")} + assert "openai-key" in kinds + + # The leading \b still holds: an `sk-` inside a word is not a key. + benign = "risk-assessment-framework-rollout-plan" + assert not any(f.kind == "openai-key" for f in redact.scan(benign)) + + +def test_projection_is_stable_for_the_same_input(): + """No wall clock, no randomness: the same argv projects identically.""" + argv = ["run", "--key", "MY_TOKEN=" + _GHP, "&&", "echo", "done"] + assert redact.redact_argv(argv) == redact.redact_argv(argv) + + +def test_withheld_element_is_never_an_empty_string(monkeypatch): + """Redact-or-withhold, never mangle — and never silently. + + The post-condition cannot fail once findings are merged, so this forces it. + The honesty risk is a withheld element exported as an empty preview, which + reads as "nothing ran here" instead of "this was suppressed". + """ + def broken(source, result): + raise ValueError("forced post-condition failure") + + monkeypatch.setattr(redact, "reconstruct", broken) + out = redact.redact_argv(["safe", "MY_TOKEN=" + _GHP]) + assert out.argv[0] == "«withheld:unknown»" # no findings -> unknown + assert out.argv[1] == "«withheld:env-secret-assignment»" + assert "" not in out.argv + assert all(e["withheld"] is True for e in out.applied) + assert [e["argv_index"] for e in out.applied] == [0, 1] + + +# --- the README guarantee ---------------------------------------------------- + +_README = Path(__file__).resolve().parents[1] / "README.md" + +# Locked Decision 10: every guarantee in README.md / TRUST_MODEL.md must have a +# test that fails when it stops being true. The removed sentence claimed the +# HTML report was safe to attach to a PR because "secrets are redacted from +# every export" — the HTML path applies no redaction at all. Two assertions, +# not one: a bare negative is satisfied by replacing one false sentence with a +# different false sentence. +# +# The sentence widened once the claim FIELDS started being redacted too, and +# the narrowing that replaced it has to stay narrow: the HTML report's file +# list for a stale claim is recomputed against the working tree at verify time, +# so it is NOT the note's redacted copy. Both halves are asserted, so a future +# edit cannot quietly drop the caveat while keeping the reassuring half. +# +# The enumeration widened again at v3: a conjunction's conjunct names and every +# claim's `reason` joined the redacted set, because a conjunction's reason NAMES a +# conjunct and a conjunct name is operator-authored. The sentence has to keep +# enumerating what is actually covered — a list that goes stale in the +# understating direction is the same defect in a quieter key, and the assertion +# below is what forces the edit. +_REMOVED_CLAIM = "secrets are redacted from every export" +_REPLACEMENT = ( + "Redaction covers the sealed note: every exported claim string — the " + "`argv_preview`, the label, the pathspecs, the changed paths, the conjuncts " + "and the reason — is redacted, and the projection is declared in the manifest." +) +_CAVEAT = ( + "a stale claim's file list is recomputed against your working tree at " + "verify time and is shown as it is on disk" +) + + +def test_readme_does_not_claim_every_export_is_redacted(): + text = " ".join(_README.read_text(encoding="utf-8").split()) + assert _REMOVED_CLAIM not in text + + +def test_readme_states_what_is_actually_redacted(): + text = " ".join(_README.read_text(encoding="utf-8").split()) + assert _REPLACEMENT in text + + +def test_readme_keeps_the_caveat_the_widened_claim_needs(): + """The widened sentence is only true because of what follows it.""" + text = " ".join(_README.read_text(encoding="utf-8").split()) + assert _CAVEAT in text + + +# --- the projection reaches the note ---------------------------------------- + +def _git(repo: Path, *a: str) -> None: + subprocess.run(["git", *a], cwd=str(repo), capture_output=True, text=True, check=True) + + +def test_sealed_note_declares_the_projection(repo: Path): + """Test 8: after a seal, `redaction.applied` names exactly the spans that + were replaced, and a consumer can tell which argv elements were touched.""" + session = Session(repo / ".didrun") + secret_arg = "MY_TOKEN=" + _GHP + run_wrapped([sys.executable, "-c", "print(1)", secret_arg], session, repo) + M.declare_claim( + session, + Claim(ctype="tests-pass", label="t", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(session, repo, allow_secrets=True) + + entry = m.claims[0] + block = entry["redaction"] + assert block["projection_version"] == redact.PROJECTION_VERSION + assert block["detector_set_version"] == redact.DETECTOR_SET_VERSION + + recorded = list(session.events()[0].argv) + preview = entry["claim"]["argv_preview"] + assert len(preview) == len(recorded) + + touched = {e["argv_index"] for e in block["applied"]} + assert touched == {3} + for i, element in enumerate(recorded): + if i not in touched: + assert preview[i] == element # untouched elements travel verbatim + + (applied,) = block["applied"] + assert applied["detector_id"] == "env-secret-assignment" + assert applied["withheld"] is False + assert (applied["start"], applied["end"]) == (0, len(secret_arg)) + + # A consumer reconstructs the redacted element from the declared span. + rebuilt = ( + recorded[3][:applied["start"]] + + _marker(applied["detector_id"]) + + recorded[3][applied["end"]:] + ) + assert preview[3] == rebuilt + assert _GHP not in " ".join(preview) diff --git a/tests/test_redact_render.py b/tests/test_redact_render.py index d288d67..72e31ac 100644 --- a/tests/test_redact_render.py +++ b/tests/test_redact_render.py @@ -2,11 +2,18 @@ from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field from didrun import redact from didrun import render -from didrun.claims import GradeResult, Claim +from didrun.claims import ( + ENV_DRIFT, + ENV_INCOMPARABLE, + ENV_MATCH, + ENV_NOT_RECORDED, + GradeResult, + Claim, +) from didrun.gitplumbing import PathChange @@ -35,15 +42,20 @@ def test_benign_git_hash_not_flagged(): def test_redact_replaces_with_marker(): text = "token sk-ABCDEFGHIJKLMNOPQRSTUVWX here" - out, findings = redact.redact(text) - assert findings - assert "sk-ABCDEFGHIJKLMNOPQRSTUVWX" not in out - assert "redacted" in out + result = redact.redact(text) + assert result.findings + assert "sk-ABCDEFGHIJKLMNOPQRSTUVWX" not in result.text + assert "redacted" in result.text # --- render helpers ---------------------------------------------------------- -def _report(results, all_verified=False, worst="stale"): +def _report(results, all_verified=False, worst="stale", chain="intact", + chain_index=None, chain_why="", require_env_match=False): + # Bound outside the class body on purpose: a class-body default may not read + # a name the class body itself also assigns (LOAD_NAME, not LOAD_CLASSDEREF). + gating = require_env_match + @dataclass class R: commit: str = "abcdef123456" @@ -52,15 +64,60 @@ class R: results: list = None coverage: dict = None secrets_override: bool = False + notes_skipped: int = 0 + # Mirrors VerifyReport's chain fields. Carried as real attributes rather + # than left off: render reads them directly, so a surface that starts + # depending on a chain state this fake cannot express fails here loudly + # instead of silently rendering the default. + chain_status: str = chain + chain_broken_index: object = chain_index + chain_reason: str = chain_why + # Mirrors VerifyReport's environment fields, for the same reason. + require_env_match: bool = gating + # Mirrors VerifyReport's authority block, for the same reason. Empty on + # purpose: every surface test in this file renders a report that cited + # nothing, which is what pins "an ordinary verdict gains no authority + # line". The cited case lives in tests/test_operator_authority.py. + authority: list = field(default_factory=list) @property def worst_status(self): return worst + @property + def env_counts(self): + counts = {ENV_MATCH: 0, ENV_DRIFT: 0, ENV_INCOMPARABLE: 0, + ENV_NOT_RECORDED: 0} + for r in self.results: + counts[r.env_status] = counts.get(r.env_status, 0) + 1 + return counts + @property def all_verified(self): return all_verified + # Mirrors VerifyReport: a superseded result is record, not verdict, so + # both surfaces count the live set and report the retained count + # separately. Real GradeResults carry the flag, so the fake derives + # these the same way the real report does rather than declaring them. + @property + def live_results(self): + return [r for r in self.results if not r.is_superseded] + + @property + def superseded_count(self): + return len(self.results) - len(self.live_results) + + # Mirrors VerifyReport: both human surfaces report how many verdicts + # were checked against the recorded entry the seal named. + @property + def evidence_bound_count(self): + return sum(1 for r in self.results if r.evidence_bound) + + @property + def total(self): + return len(self.results) + return R(results=results, coverage={"by_coverage": {"complete": 1}}) @@ -92,6 +149,23 @@ def test_cli_no_color_has_text_tokens(monkeypatch): assert "VERIFIED" not in out +def test_unparseable_notes_are_reported_on_both_verdict_paths(monkeypatch): + """"No manifest" and "there were notes and none could be read" are + different facts, and the second must reach the operator on the no-claims + path too — that is the path a repo with only foreign notes lands on.""" + monkeypatch.setenv("NO_COLOR", "1") + quiet = render.render_verdict(_report([])) + assert "skipped" not in quiet # never invented when the count is zero + + empty = _report([], worst="empty") + empty.notes_skipped = 2 + assert "2 notes skipped (unparseable)" in render.render_verdict(empty) + + graded = _report([_gr("tree-exact", "t")]) + graded.notes_skipped = 1 + assert "1 note skipped (unparseable)" in render.render_verdict(graded) + + def test_no_surface_says_verified(): """Honesty invariant: no human surface may label a grade 'VERIFIED'/'PROVEN'.""" results = [_gr("tree-exact", "t"), _gr("failed", "f")] diff --git a/tests/test_seal_publication.py b/tests/test_seal_publication.py new file mode 100644 index 0000000..f680dd1 --- /dev/null +++ b/tests/test_seal_publication.py @@ -0,0 +1,338 @@ +"""P0.1 — seal publishes fail-closed, on stdin, atomically with the watermark. + +The defect these pin: `_attach_note` discarded git's exit status, so a denied +write to refs/notes/didrun produced a silent half-seal — no note, watermark +advanced, exit 0. The pair (note published, watermark advanced) is now atomic +in both directions, and the rollback RESTORES a note it overwrote rather than +removing it. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + +# Captured at import so a selective patch can delegate without recursing. +_REAL_RUN = subprocess.run + +# Keys `to_json()` emits that a note of this manifest version never carried. +# Additive fields land here as a DELIBERATE one-line diff, never silently. +# v2 adds no TOP-LEVEL key: the evidence binding lives inside each claim entry, +# so the manifest's own key set is unchanged from v1. +# `claims_from`/`claims_to` are P4.1's: the claim window the seal covered. Both +# are additive with a reading default (0 / len(claims)), so a stored note of +# EITHER version legitimately lacks them, and both rows carry them. +# v3 is P4.2's `conjunction` claim type, which adds no top-level key either: the +# conjunct list and the per-conjunct grades live inside a claim entry. The row +# exists so the version this build writes has a DECLARED allowlist instead of a +# KeyError, and so a v3-only top-level field would still have to land here on +# purpose. +_ADDITIVE_KEYS_BY_MANIFEST_VERSION = { + 1: frozenset({"claims_from", "claims_to"}), + 2: frozenset({"claims_from", "claims_to"}), + 3: frozenset({"claims_from", "claims_to"}), +} + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _seeded(repo: Path, labels=("t",)) -> Session: + """One recorded event plus one claim per label, ready to seal.""" + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + for label in labels: + M.declare_claim( + s, Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0) + ) + return s + + +def _seal_lines(repo: Path) -> int: + path = repo / ".didrun" / "seals.jsonl" + if not path.exists(): + return 0 + return len([ln for ln in path.read_text(encoding="ascii").splitlines() if ln.strip()]) + + +def _note_bytes(repo: Path, commit: str = "HEAD"): + """The raw note body, or None when the commit carries no note.""" + proc = _REAL_RUN( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +def _is_notes_add(argv) -> bool: + argv = list(argv) + return argv[:2] == ["git", "notes"] and "add" in argv + + +def _patch_notes_add(monkeypatch, returncode: int, seen: list): + """Fail (or merely observe) ONLY `git notes … add`. + + A blanket patch of `manifest.subprocess.run` is wrong: the same symbol + serves `_rev` and `_resolve_manifest`, so a blanket failure aborts the seal + before it ever reaches `_attach_note` and the test passes for the wrong + reason. A failing `git` shim on PATH has the same defect. + """ + + def fake(argv, *args, **kwargs): + if _is_notes_add(argv): + seen.append((list(argv), kwargs.get("input"))) + if returncode != 0: + return subprocess.CompletedProcess( + list(argv), returncode, b"", b"fatal: could not write note object" + ) + return _REAL_RUN(argv, *args, **kwargs) + + monkeypatch.setattr(M.subprocess, "run", fake) + + +def _round_trip(body: bytes) -> M.Manifest: + """The note round-trip criterion: newline discipline, key-superset semantic + equality, serialization idempotence. Deliberately NOT byte equality — + `git notes` appends a trailing newline to every body, so that assertion is + false on every real note. + """ + # (a) strip at most one trailing newline, and prove at most one was there. + assert body.endswith(b"\n"), "note body does not end in a newline" + stripped = body[:-1] + assert not stripped.endswith(b"\n"), "note body carries more than one trailing newline" + + manifest = M.Manifest.from_json(stripped) + source = json.loads(stripped) + emitted = json.loads(manifest.to_json()) + + # (b) every source key survives with an equal value; extras are allowlisted. + for key, value in source.items(): + assert key in emitted, f"round-trip dropped manifest key {key!r}" + assert emitted[key] == value, f"round-trip changed manifest key {key!r}" + # The allowlist is the BOUND on what may be missing, not an exact set: every + # body this helper sees was written by the seal under test, so it already + # carries every key `to_json` emits and the extra set is empty. Exact + # equality here asserted that a note this binary just published must LACK a + # key it declares, which stopped being true the moment an additive field + # landed. The leg that actually exercises a key-lacking note is + # tests/compat's leg 2 (over stored v1 bodies), plus the byte pin below. + extra = frozenset(emitted) - frozenset(source) + allowed = _ADDITIVE_KEYS_BY_MANIFEST_VERSION[source["version"]] + assert extra <= allowed, ( + f"unexpected extra manifest keys {sorted(extra - allowed)} — a field was " + "added without updating the per-version allowlist" + ) + + # (c) serialization is idempotent; this is where canonical_json is pinned. + once = manifest.to_json() + assert M.Manifest.from_json(once).to_json() == once + return manifest + + +def test_note_publication_failure_is_fatal(repo: Path, monkeypatch): + """git refuses the note -> seal raises, nothing is recorded, no note exists.""" + s = _seeded(repo) + seen: list = [] + _patch_notes_add(monkeypatch, 1, seen) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + + assert "could not publish note" in str(exc.value) + assert "fatal: could not write note object" in str(exc.value) + assert seen, "the selective patch never saw a `git notes add` — wrong seam" + assert _seal_lines(repo) == 0 + assert _note_bytes(repo) is None + + +def test_watermark_failure_rolls_back_and_leaves_no_note(repo: Path, monkeypatch): + """No prior note: a failed watermark write must leave the commit noteless.""" + s = _seeded(repo) + + def boom(*args, **kwargs): + raise OSError("read-only file system: seals.jsonl") + + monkeypatch.setattr(M, "_record_seal", boom) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + + message = str(exc.value) + assert "no watermark was recorded" in message + assert "rolled back" in message + assert "noteless" in message + assert "read-only file system" in message + assert _seal_lines(repo) == 0 + assert _note_bytes(repo) is None + + +def test_watermark_failure_restores_a_prior_note(repo: Path, monkeypatch): + """The rollback may never destroy evidence the commit already carried. + + This is the leg a `git notes remove` rollback fails: `_attach_note` forces, + so the seal overwrote a real prior note. + """ + s = _seeded(repo) + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-m", "prior evidence body"], + cwd=str(repo), + capture_output=True, + check=True, + ) + prior = _note_bytes(repo) + assert prior == b"prior evidence body\n" + + def boom(*args, **kwargs): + raise OSError("read-only file system: seals.jsonl") + + monkeypatch.setattr(M, "_record_seal", boom) + + # `reseal=True` because the prior note here is deliberately NOT a didrun + # manifest, so it records no claim window and P4.1 refuses to replace it + # blind. This test's subject is the rollback restoring the overwritten body + # byte-for-byte, which needs the seal to reach `_record_seal` at all. + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo, reseal=True) + + message = str(exc.value) + assert "no watermark was recorded" in message + assert "prior note was restored" in message + assert _seal_lines(repo) == 0 + assert _note_bytes(repo) == prior # byte-restored, not removed + + +def test_body_travels_on_stdin_not_argv(repo: Path, monkeypatch): + """The real criterion: `-F -` with the body on `input=`, and no `-m`.""" + s = _seeded(repo) + seen: list = [] + _patch_notes_add(monkeypatch, 0, seen) + + m = M.seal(s, repo) + + assert len(seen) == 1 + argv, stdin = seen[0] + assert "-F" in argv + assert "-" in argv + assert "-m" not in argv + body = m.to_json() + assert body not in [el.encode("ascii") if isinstance(el, str) else el for el in argv] + assert all(body.decode("ascii") != el for el in argv) + assert stdin == body + + +def test_note_larger_than_arg_max_publishes_and_resolves(repo: Path): + """A body over 1.5 MB — past Linux's 131,072-byte MAX_ARG_STRLEN per argument + and past macOS's 1,048,576-byte kern.argmax — publishes and re-resolves. + + 256 KB is the wrong number: it succeeds under `-m` on macOS and fails under + `-m` on Linux, so it proves nothing and diverges by platform. + """ + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label="x" * 1_600_000, + event_indices=(0,), + declared_at_index=0, + ), + ) + + m = M.seal(s, repo) + body = _note_bytes(repo) + assert body is not None + assert len(body) >= 1_572_864, f"body is only {len(body)} bytes — too small to bind" + + resolved = _round_trip(body) + assert resolved == m + assert M.verify(s, repo).all_verified + + +def test_happy_path_publishes_exactly_what_it_returns(repo: Path): + s = _seeded(repo) + + m = M.seal(s, repo) + + assert _seal_lines(repo) == 1 + body = _note_bytes(repo) + assert body is not None + assert _round_trip(body) == m + + +def test_happy_path_cli_still_exits_zero(repo: Path): + _seeded(repo) + + assert cli.main(["--repo", str(repo), "seal"]) == 0 + assert _seal_lines(repo) == 1 + assert _note_bytes(repo) is not None + + +def test_manifest_json_bytes_are_pinned(): + """The manifest's on-the-wire bytes are compat surface across every sealed + note. A `to_json` tidy-up would move them silently; this pin makes any move + a deliberate diff. + + P3.2 adds exactly one key, `secrets`, whose default is `{}`. Every key a + stored note carries still comes back with an equal value, so the addition + is additive with a v1-reproducing default — which is why this is a one-line + diff here plus a one-line diff in the compat allowlist, and not a + MANIFEST_VERSION bump. + + P4.1 adds `claims_from`/`claims_to`, the claim window the seal covered, on + the same terms: reading defaults of 0 and `len(claims)`, so no stored note + changes meaning and this stays a deliberate diff rather than a version bump. + + P5.1 adds `authority`, the cited authorization artifacts, with a reading + default of `[]` — and it sorts FIRST, so the pinned bytes move at the front. + Same terms again: additive, no version bump, one deliberate diff here and one + in the compat allowlist. It is a manifest key and not an `Event` field + precisely so it can be exactly this: a change no stored note notices. + """ + m = M.Manifest( + version=1, + commit="0" * 40, + tree="1" * 40, + claims=[ + { + "claim": { + "argv_preview": ["pytest", "-q"], + "ctype": "tests-pass", + "declared_at_index": 0, + "event_indices": [0], + "label": "t", + "pathspecs": [], + }, + "delta": [], + "exit_code": 0, + "grade": "tree-exact", + "reason": "r", + "supporting_event_index": 0, + } + ], + coverage={"by_coverage": {"complete": 1}, "total_events": 1}, + ) + assert m.to_json() == ( + b'{"authority":[],' + b'"claims":[{"claim":{"argv_preview":["pytest","-q"],"ctype":"tests-pass",' + b'"declared_at_index":0,"event_indices":[0],"label":"t","pathspecs":[]},' + b'"delta":[],"exit_code":0,"grade":"tree-exact","reason":"r",' + b'"supporting_event_index":0}],' + b'"claims_from":0,"claims_to":1,' + b'"commit":"0000000000000000000000000000000000000000",' + b'"coverage":{"by_coverage":{"complete":1},"total_events":1},' + b'"secrets":{},"secrets_override":false,' + b'"tree":"1111111111111111111111111111111111111111","version":1}' + ) diff --git a/tests/test_streaming_capture.py b/tests/test_streaming_capture.py new file mode 100644 index 0000000..7898331 --- /dev/null +++ b/tests/test_streaming_capture.py @@ -0,0 +1,349 @@ +"""P2.2 — the Tier-0 capture pump streams instead of buffering to completion. + +The criterion is byte-identity of what is RECORDED: the drained blobs must equal +what ``subprocess.run(capture_output=True)`` would have returned, for every shape +of output. Everything else here (tee, heartbeat, ``show --output``) is display, +and the tests pin that display never becomes the default and never carries the +child's bytes into a place the buffered path did not. +""" + +from __future__ import annotations + +import re +import subprocess +import sys +import threading +import time +from pathlib import Path + +import pytest + +from didrun.capture import run_wrapped +from didrun.ledger import Session, sha256_hex + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +# Spawning the CLI as a real process is the only way to see what a caller's +# terminal sees; `-c` is used rather than a path so no absolute path is embedded. +_CLI_BOOT = "import sys; from didrun.cli import main; sys.exit(main(sys.argv[1:]))" + + +def _cli_argv(repo: Path, *args: str) -> list: + return [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), *args] + + +def _run_cli(repo: Path, *args: str, timeout: float = 120) -> subprocess.CompletedProcess: + return subprocess.run( + _cli_argv(repo, *args), cwd=str(repo), capture_output=True, timeout=timeout + ) + + +def _child_script(tmp_path: Path, name: str, source: str) -> list: + """A child whose ARGV carries none of the text it prints. + + ``didrun run``'s summary line echoes argv, so a `-c` child whose source + contains the marker would let a marker-absence assertion pass (or fail) for + the wrong reason. The script lives outside the repo so it cannot move a tree + digest either. + """ + path = tmp_path / name + path.write_text(source, encoding="utf-8") + return [sys.executable, str(path)] + + +def _wait_for(predicate, timeout: float, interval: float = 0.05) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(interval) + return predicate() + + +# --- 1. the criterion: recorded blobs are byte-identical ---------------------- + +_NOTHING = "pass" +_LARGE = ( + "import sys\n" + "for _ in range(1024):\n" + " sys.stdout.buffer.write(b'y' * 1024)\n" + "sys.stdout.buffer.flush()\n" +) +_INTERLEAVED = ( + "import sys\n" + "for i in range(200):\n" + " sys.stdout.buffer.write(b'out-%d\\n' % i)\n" + " sys.stdout.buffer.flush()\n" + " sys.stderr.buffer.write(b'err-%d\\n' % i)\n" + " sys.stderr.buffer.flush()\n" +) +# NULs and lone 0xfe/0xff: bytes no text decoder round-trips. +_BINARY = ( + "import sys\n" + "sys.stdout.buffer.write(bytes(range(256)) * 8)\n" + "sys.stdout.buffer.flush()\n" +) +_STDERR_ONLY = ( + "import sys\n" + "sys.stderr.buffer.write(b'only stderr, repeatedly\\n' * 500)\n" + "sys.stderr.buffer.flush()\n" +) + + +@pytest.mark.parametrize( + "name,script", + [ + ("empty", _NOTHING), + ("large", _LARGE), + ("interleaved", _INTERLEAVED), + ("binary", _BINARY), + ("stderr-only", _STDERR_ONLY), + ], +) +def test_streamed_blobs_are_byte_identical_to_the_buffered_path( + repo: Path, name: str, script: str +): + """The pump changes WHEN bytes are seen, never WHICH bytes are recorded.""" + argv = [sys.executable, "-c", script] + reference = subprocess.run(argv, cwd=str(repo), capture_output=True) + + s = _session(repo) + ev = run_wrapped(argv, s, repo) + + assert ev.stdout_blob == sha256_hex(reference.stdout), f"{name}: stdout digest" + assert ev.stderr_blob == sha256_hex(reference.stderr), f"{name}: stderr digest" + assert s.blobs.get(ev.stdout_blob) == reference.stdout + assert s.blobs.get(ev.stderr_blob) == reference.stderr + assert ev.exit_code == reference.returncode + # The persisted shape is unchanged: this is still a complete wrapper event. + assert (ev.observed_via, ev.coverage) == ("wrapper", "complete") + + +# --- 2. liveness -------------------------------------------------------------- + +_EARLY_THEN_SLEEP = ( + "import sys, time\n" + "sys.stdout.write('MARKER-EARLY\\n')\n" + "sys.stdout.flush()\n" + "time.sleep(5)\n" +) + + +def test_tee_surfaces_output_before_the_child_exits(repo: Path, tmp_path: Path): + """--tee is progressive, not a replay at the end: the marker is readable + while the child is still running.""" + sink = tmp_path / "tee.out" + child = _child_script(tmp_path, "early_then_sleep.py", _EARLY_THEN_SLEEP) + with sink.open("wb") as fh: + proc = subprocess.Popen( + _cli_argv(repo, "run", "--tee", "--", *child), + cwd=str(repo), + stdout=fh, + stderr=subprocess.DEVNULL, + ) + try: + seen = _wait_for(lambda: b"MARKER-EARLY" in sink.read_bytes(), timeout=3.0) + # Read the liveness fact at the same instant, not after the wait. + still_running = proc.poll() is None + finally: + rc = proc.wait(timeout=60) + + assert seen, "the child's output never reached the terminal while it ran" + assert still_running, "the marker only appeared after the child had exited" + assert rc == 0 + + +def test_default_run_tees_nothing_and_prints_only_the_summary(repo: Path, tmp_path: Path): + """No flags: stdout carries the summary line and not one child byte. + External gates parse this stream.""" + child = _child_script( + tmp_path, + "noisy.py", + "import sys\nprint('MARKER-CHILD')\nprint('MARKER-ERR', file=sys.stderr)\n", + ) + proc = _run_cli(repo, "run", "--", *child) + + assert proc.returncode == 0 + lines = [ln for ln in proc.stdout.decode().splitlines() if ln.strip()] + assert len(lines) == 1, f"expected only the summary line, got {lines!r}" + assert lines[0].startswith("recorded wrapper event") + assert b"MARKER-CHILD" not in proc.stdout + assert b"MARKER-CHILD" not in proc.stderr + assert b"MARKER-ERR" not in proc.stdout + assert b"MARKER-ERR" not in proc.stderr + + +# --- 3. no deadlock ----------------------------------------------------------- + +_BOTH_PIPES_BIG = ( + "import sys\n" + "for _ in range(16):\n" + " sys.stdout.buffer.write(b'o' * 65536)\n" + " sys.stderr.buffer.write(b'e' * 65536)\n" + "sys.stdout.buffer.flush()\n" + "sys.stderr.buffer.flush()\n" +) + + +def test_pump_does_not_deadlock_when_both_pipes_fill(repo: Path): + """1 MiB down each pipe before the child exits — far past the OS pipe buffer. + A pump that drained one stream to EOF first would hang here forever, so the + join timeout is the assertion.""" + s = _session(repo) + argv = [sys.executable, "-c", _BOTH_PIPES_BIG] + box: dict = {} + + def go() -> None: + box["event"] = run_wrapped(argv, s, repo) + + worker = threading.Thread(target=go, daemon=True) + worker.start() + worker.join(120) + + assert not worker.is_alive(), "the capture pump deadlocked on a full pipe" + ev = box["event"] + assert s.blobs.get(ev.stdout_blob) == b"o" * 65536 * 16 + assert s.blobs.get(ev.stderr_blob) == b"e" * 65536 * 16 + + +# --- 4. heartbeat ------------------------------------------------------------- + +_HEARTBEAT_RE = re.compile( + r"^didrun: \d+s \S+ \(\d+ args\) stdout=\d+B stderr=\d+B$" +) + +_NOISY_THEN_SLEEP = ( + "import sys, time\n" + "sys.stdout.write('MARKER-CONTENT-OUT\\n')\n" + "sys.stderr.write('MARKER-CONTENT-ERR\\n')\n" + "sys.stdout.flush(); sys.stderr.flush()\n" + "time.sleep(3)\n" +) + + +def test_heartbeat_is_bounded_and_carries_no_content(repo: Path, tmp_path: Path): + child = _child_script(tmp_path, "noisy_then_sleep.py", _NOISY_THEN_SLEEP) + proc = _run_cli(repo, "run", "--heartbeat", "1", "--", *child) + + assert proc.returncode == 0 + err = proc.stderr.decode() + beats = [ln for ln in err.splitlines() if ln.startswith("didrun: ")] + assert len(beats) >= 2, f"expected repeated heartbeats over a 3s child, got {beats!r}" + for line in beats: + assert _HEARTBEAT_RE.match(line), f"heartbeat off-shape: {line!r}" + # Not one byte of what the child printed may ride out on the progress line. + assert "MARKER-CONTENT-OUT" not in err + assert "MARKER-CONTENT-ERR" not in err + assert b"MARKER-CONTENT-OUT" not in proc.stdout + # And the heartbeat stays off stdout, which external gates parse. + assert "didrun: " not in proc.stdout.decode() + + +def test_heartbeat_is_off_by_default(repo: Path): + child = [sys.executable, "-c", "import time; time.sleep(2)"] + proc = _run_cli(repo, "run", "--", *child) + assert proc.returncode == 0 + assert proc.stderr == b"", f"unrequested output on stderr: {proc.stderr!r}" + + +# --- 5. show --event N --output ---------------------------------------------- + +# Synthetic, structurally identical to a real AWS key; matches redact's +# aws-access-key pattern. Never a live credential. +_FAKE_KEY = "AKIAZZ7EXAMPLE9QTEST" + + +def test_show_output_round_trips_the_recorded_bytes(repo: Path): + s = _session(repo) + ev = run_wrapped([sys.executable, "-c", _BINARY], s, repo) + recorded = s.blobs.get(ev.stdout_blob) + + proc = _run_cli(repo, "show", "--event", "0", "--output") + assert proc.returncode == 0 + assert proc.stdout == recorded + + +def test_show_output_reads_the_stderr_stream(repo: Path): + s = _session(repo) + ev = run_wrapped([sys.executable, "-c", _STDERR_ONLY], s, repo) + + proc = _run_cli(repo, "show", "--event", "0", "--output", "--stream", "stderr") + assert proc.returncode == 0 + assert proc.stdout == s.blobs.get(ev.stderr_blob) + + +def test_show_output_redacted_applies_the_redactor(repo: Path): + s = _session(repo) + run_wrapped([sys.executable, "-c", f"print({_FAKE_KEY!r})"], s, repo) + + raw = _run_cli(repo, "show", "--event", "0", "--output") + assert _FAKE_KEY.encode() in raw.stdout + + red = _run_cli(repo, "show", "--event", "0", "--output", "--redacted") + assert red.returncode == 0 + assert _FAKE_KEY.encode() not in red.stdout + assert "«redacted:aws-access-key»" in red.stdout.decode("utf-8") + + +def test_show_output_refuses_a_missing_blob_without_a_traceback(repo: Path): + s = _session(repo) + ev = run_wrapped([sys.executable, "-c", "print('gone')"], s, repo) + (s.blobs.root / ev.stdout_blob).unlink() + + proc = _run_cli(repo, "show", "--event", "0", "--output") + assert proc.returncode == 2 + assert b"Traceback" not in proc.stderr + assert b"didrun show:" in proc.stderr + + +def test_show_output_refuses_an_out_of_range_event(repo: Path): + s = _session(repo) + run_wrapped([sys.executable, "-c", "print('one')"], s, repo) + + proc = _run_cli(repo, "show", "--event", "7", "--output") + assert proc.returncode == 2 + assert b"Traceback" not in proc.stderr + + proc = _run_cli(repo, "show", "--output") + assert proc.returncode == 2 + assert b"--event" in proc.stderr + + +def test_show_output_does_not_create_a_ledger(repo: Path, tmp_path: Path): + """--output is a read-only command and stays one (P2.1's seam).""" + empty = tmp_path / "empty" + empty.mkdir() + proc = _run_cli(empty, "show", "--event", "0", "--output") + assert proc.returncode == 2 + # didrun's own refusal, not argparse's: a 2 from an unrecognised flag would + # satisfy the returncode and the missing directory for the wrong reason. + assert b"didrun show:" in proc.stderr + assert not (empty / ".didrun").exists() + + +# --- 6. exit-code fidelity ---------------------------------------------------- + + +@pytest.mark.parametrize("code", [0, 1, 42]) +def test_exit_codes_match_the_buffered_path(repo: Path, code: int): + argv = [sys.executable, "-c", f"import sys; sys.exit({code})"] + reference = subprocess.run(argv, cwd=str(repo), capture_output=True) + s = _session(repo) + ev = run_wrapped(argv, s, repo) + assert ev.exit_code == reference.returncode == code + + +def test_signal_death_records_the_same_code_as_the_buffered_path(repo: Path): + argv = [ + sys.executable, + "-c", + "import os, signal; os.kill(os.getpid(), signal.SIGTERM)", + ] + reference = subprocess.run(argv, cwd=str(repo), capture_output=True) + s = _session(repo) + ev = run_wrapped(argv, s, repo) + assert ev.exit_code == reference.returncode + assert ev.exit_code < 0, "a signal death must not be recorded as a clean exit" diff --git a/tests/test_unreadable_note_refuses.py b/tests/test_unreadable_note_refuses.py new file mode 100644 index 0000000..ed9b80c --- /dev/null +++ b/tests/test_unreadable_note_refuses.py @@ -0,0 +1,305 @@ +"""A note this binary cannot read is a graded refusal — never a crash, never a pass. + +The invariant is stated in `cli.cmd_verify` itself: "Evidence this binary cannot +read is a graded refusal, not a crash. 2, not --strict's 1: 'could not read the +manifest' is a different fact from 'the manifest graded badly'." It was not +holding. Measured on the pre-fix tree, over 22 malformed note shapes: + + - 11 shapes left an uncaught traceback (`KeyError`, `TypeError`, + `JSONDecodeError`, `ClaimError`) and exited **1** — colliding with the exit + code `--strict` uses for a note that graded badly, so a corrupt note was + indistinguishable from an honest failure by exit code alone; + - 4 shapes were **ACCEPTED**. `{"version": true}` verified green at exit 0, + because `bool` is an `int` subclass and `True > 3` is False. So did `2.5`, + `0` and `-1`. + +The accepted set is the one that matters. docs/COMPAT.md's "Version 3" section +rests the whole forward-compatibility story on the version guard — the +MANIFEST_VERSION bump exists so an older reader refuses a v3 note *on its +version* before it reaches the claim vocabulary it cannot parse. A guard a +malformed field walks past is not a guard. + +Test 1 is the gate: over every malformed shape, `didrun verify --strict` must +exit 2 with no traceback. Not "must not crash" — must not PASS either, and the +two failures need one test that cannot be satisfied by fixing only one of them. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + + +# --- helpers ----------------------------------------------------------------- + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _seal_one(repo: Path) -> M.Manifest: + """A repo carrying one honest sealed claim, so every mutation below starts + from a note that verifies green.""" + session = _session(repo) + run_wrapped([sys.executable, "-c", "print('ok')"], session, repo) + M.declare_claim( + session, + Claim( + ctype="tests-pass", + label="unit", + event_indices=(0,), + declared_at_index=0, + ), + ) + return M.seal(session, repo) + + +def _note_text(repo: Path, commit: str = "HEAD") -> str: + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + text=True, + ) + assert proc.returncode == 0, "the commit carries no note" + return proc.stdout + + +def _put_note(repo: Path, raw: str, commit: str = "HEAD") -> None: + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-m", raw, commit], + cwd=str(repo), + capture_output=True, + text=True, + check=True, + ) + + +def _mutated(raw: str, fn) -> str: + d = json.loads(raw) + fn(d) + return json.dumps(d, sort_keys=True, separators=(",", ":")) + "\n" + + +def _set_ctype(d, value) -> None: + d["claims"][0]["claim"]["ctype"] = value + + +# Every shape measured as broken on the pre-fix tree, plus the four that were +# accepted. Keyed by name so a failure says which shape regressed. +# +# The `version` cases are split into two groups on purpose. A NON-INTEGER +# version is a malformed field (ManifestFormatError). An integer version above +# MANIFEST_VERSION is the documented forward-compat refusal and keeps its own +# message and its own exception — the distinction is load-bearing on the +# tree-fallback scan, where one is skipped and the other propagates. +MALFORMED = { + # -- the fail-OPEN set: these verified green before the fix --------------- + "version true": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", True)), + "version 2.5": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", 2.5)), + "version 0": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", 0)), + "version -1": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", -1)), + # -- the crash set -------------------------------------------------------- + "version string": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", "3")), + "version null": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", None)), + "version list": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", [3])), + "version absent": lambda raw: _mutated(raw, lambda d: d.pop("version")), + "not json": lambda raw: "this is not json\n", + "json list": lambda raw: "[1,2,3]\n", + "json string": lambda raw: '"hello"\n', + "no commit": lambda raw: _mutated(raw, lambda d: d.pop("commit")), + "no tree": lambda raw: _mutated(raw, lambda d: d.pop("tree")), + "no claims": lambda raw: _mutated(raw, lambda d: d.pop("claims")), + "no coverage": lambda raw: _mutated(raw, lambda d: d.pop("coverage")), + "claims not a list": lambda raw: _mutated( + raw, lambda d: d.__setitem__("claims", "x") + ), + "claim entry not a dict": lambda raw: _mutated( + raw, lambda d: d["claims"].__setitem__(0, 7) + ), + "claim object missing": lambda raw: _mutated( + raw, lambda d: d["claims"][0].pop("claim") + ), + "ctype unknown": lambda raw: _mutated(raw, lambda d: _set_ctype(d, "nonsense")), + "ctype killed": lambda raw: _mutated(raw, lambda d: _set_ctype(d, "diff-exercised")), + "ctype null": lambda raw: _mutated(raw, lambda d: _set_ctype(d, None)), + "event_indices not iterable": lambda raw: _mutated( + raw, lambda d: d["claims"][0]["claim"].__setitem__("event_indices", 7) + ), +} + + +# --- test 1: the gate -------------------------------------------------------- + + +@pytest.mark.parametrize("shape", sorted(MALFORMED)) +def test_an_unreadable_note_refuses_with_exit_2_and_no_traceback( + repo: Path, capsys, shape: str +): + """THE criterion, over every malformed shape, through the real CLI. + + Three assertions, because the two pre-fix failure modes are opposites and a + one-sided test would pass on half a fix: + + 1. exit 2 — not 0 (accepted), and not 1 (which is `--strict`'s "graded + badly" and would make a corrupt note look like an honest failure); + 2. the refusal is on stderr, prefixed like every other refusal; + 3. nothing raised out of the command. + """ + _seal_one(repo) + _put_note(repo, MALFORMED[shape](_note_text(repo))) + capsys.readouterr() + + # Any exception escaping cli.main IS the traceback this test is about, so it + # is not caught here — pytest reporting it as an error is the correct + # outcome for a regression. + rc = cli.main(["--repo", str(repo), "verify", "--strict"]) + captured = capsys.readouterr() + + assert rc == 2, ( + f"{shape!r}: expected the graded refusal (2), got {rc}. " + f"0 means the malformed note was ACCEPTED; 1 means it was reported as a " + f"grading failure rather than an unreadable one." + ) + assert "didrun verify:" in captured.err + assert "Traceback" not in captured.err + + +def test_the_malformed_set_covers_both_pre_fix_failure_modes(): + """A guard on test 1's fixture: it must keep testing both directions. + + If the fail-open cases were ever dropped from MALFORMED, test 1 would still + pass over a build that only stopped crashing — and the accepted-green shapes + are the ones that could put a false verdict in front of a reviewer. + """ + assert {"version true", "version 2.5", "version 0", "version -1"} <= set(MALFORMED) + assert {"not json", "ctype unknown", "no commit"} <= set(MALFORMED) + + +# --- test 2: the fail-open shape, stated on its own --------------------------- + + +def test_a_boolean_version_is_not_version_one(repo: Path): + """The measured false green, isolated. + + `bool` is an `int` subclass: `True > 3` is False, so the pre-fix guard let + `{"version": true}` through and graded the note under v1 semantics. It + verified at exit 0. This is the same reading rule `_superseded_by` already + applied via `_is_index`, and the reason is identical — a field that arrived + from a file is not a field this process computed. + """ + _seal_one(repo) + body = json.loads(_note_text(repo)) + body["version"] = True + raw = json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii") + + with pytest.raises(M.ManifestFormatError) as exc: + M.Manifest.from_json(raw) + assert "True" in str(exc.value) + # And it must NOT be reported as the forward-compat refusal, which would + # tell an operator to upgrade over a note that is simply corrupt. + assert "upgrade didrun" not in str(exc.value) + + +# --- test 3: the two refusals stay distinguishable --------------------------- + + +def test_a_format_refusal_is_a_manifest_error_but_a_version_refusal_is_not_a_format_one( + repo: Path, +): + """The type split the tree-fallback scan depends on. + + `ManifestFormatError` subclasses `ManifestError` so the CLI's existing + handler catches it and exits 2. But a version this binary cannot read must + NOT be a format error: on the scan path a format error is skipped and + counted, and a too-new version propagates. Collapsing the two would make a + v4 note skippable, and the scan would report some older note's verdict as if + it were current. + """ + assert issubclass(M.ManifestFormatError, M.ManifestError) + + _seal_one(repo) + body = json.loads(_note_text(repo)) + body["version"] = M.MANIFEST_VERSION + 1 + too_new = json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii") + + with pytest.raises(M.ManifestError) as exc: + M.Manifest.from_json(too_new) + assert not isinstance(exc.value, M.ManifestFormatError) + assert "upgrade didrun" in str(exc.value) + + +# --- test 4: the scan path still skips rather than dying --------------------- + + +def test_the_tree_fallback_still_skips_a_corrupt_note_and_counts_it( + repo: Path, git +): + """Preserved behaviour, pinned because the fix could plausibly break it. + + A foreign or corrupt note under the same ref must not hide every good one — + it is skipped and COUNTED. Before the fix that happened by accident, because + the scan caught bare `Exception` and `json` raised something that was not a + `ManifestError`. Now `ManifestFormatError` IS a `ManifestError`, so the skip + clause has to come first by type. If it did not, this verify would raise + instead of resolving the good note. + """ + _seal_one(repo) + good_tree = M.gitplumbing.commit_tree(repo, M.gitplumbing.head_commit(repo)) + + # A second commit carrying an unreadable note under the same ref. + (repo / "other.py").write_text("x = 1\n") + git(repo, "add", "other.py") + git(repo, "commit", "-qm", "other") + _put_note(repo, "not a manifest at all\n", "HEAD") + + # Ask for a tree no note carries, so the scan is forced to walk PAST the + # corrupt note instead of stopping at the good one. `git notes list` order is + # by object id, so a test that relied on the corrupt note being reached first + # would pass or fail on a hash — this reaches it every time. + m, resolved_by, skipped = M._resolve_manifest(repo, None, "f" * 40) + assert m is None + assert resolved_by == "none" + assert skipped == 1, "the corrupt note was not skipped and counted" + + # And the good note is still resolvable with the corrupt one in the ref. + m, resolved_by, _ = M._resolve_manifest(repo, None, good_tree) + assert m is not None, "the corrupt note hid the good one" + assert resolved_by == "tree-fallback" + + +# --- test 5: no false refusal of an honest note ------------------------------ + + +@pytest.mark.parametrize("version", [1, 2, M.MANIFEST_VERSION]) +def test_every_version_this_binary_writes_or_must_read_still_parses( + repo: Path, version: int +): + """The fix must refuse nothing that was legitimately published. + + Every note in the world carries an integer version of 1, 2 or 3, and the + compat corpus replays all three. A guard that tightened onto them would be a + worse defect than the one it closed. + """ + _seal_one(repo) + body = json.loads(_note_text(repo)) + body["version"] = version + raw = json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii") + assert M.Manifest.from_json(raw).version == version + + +def test_an_honest_note_still_verifies_green(repo: Path, capsys): + """The end-to-end control for test 1: the same path, unmutated, exits 0.""" + _seal_one(repo) + capsys.readouterr() + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0