From c9871070934bd08c835595c825c39156c3d11ec0 Mon Sep 17 00:00:00 2001 From: Drew Date: Tue, 28 Jul 2026 22:31:22 -0700 Subject: [PATCH 1/8] compat: freeze the v1 format and make seal publication fail closed P0 of the v0.2 hardening pack -- the foundation every later phase needs, so that adding fields and grades later cannot invalidate evidence already sealed. seal: publication and the watermark are now one atomic pair. _attach_note checks git's exit code instead of discarding it (a seal could report success when publication had failed), the body travels on stdin via -F - rather than -m (a real note body hit 1.6 MB, past ARG_MAX), and everything after publication runs inside a rollback that restores the commit's prior note. ledger: freeze the v1 chain preimage behind an explicit per-version field list, so future Event fields cannot silently change the hash of records already written. from_dict now filters unknown keys instead of cls(**d), so a newer ledger no longer crashes an older reader. Verified by recomputing a v1 entry hash under pre-change code: byte-identical. manifest: refuse unknown manifest versions rather than parsing them optimistically; carry the fingerprint version in band. tests: add the preimage golden, the seal-publication suite, and a content-blind compat replay harness that reads ledger structure and git metadata only, gated on DIDRUN_COMPAT_CORPUS so it skips cleanly when absent. 53 -> 96 passing, 4 env-gated skips; harness.recall still PASS at 100%. --- .gitignore | 6 + docs/COMPAT.md | 99 ++++ src/didrun/capture.py | 26 +- src/didrun/cli.py | 15 +- src/didrun/gitplumbing.py | 11 +- src/didrun/ledger.py | 174 +++++- src/didrun/manifest.py | 156 +++++- src/didrun/render.py | 13 + tests/compat/__init__.py | 15 + tests/compat/conftest.py | 154 ++++++ tests/compat/synthetic.py | 383 ++++++++++++++ tests/compat/test_corpus_replay.py | 732 ++++++++++++++++++++++++++ tests/test_capture_claims_manifest.py | 141 +++++ tests/test_gitplumbing.py | 52 ++ tests/test_ledger.py | 145 +++++ tests/test_preimage_golden.py | 125 +++++ tests/test_redact_render.py | 18 + tests/test_seal_publication.py | 293 +++++++++++ 18 files changed, 2523 insertions(+), 35 deletions(-) create mode 100644 docs/COMPAT.md create mode 100644 tests/compat/__init__.py create mode 100644 tests/compat/conftest.py create mode 100644 tests/compat/synthetic.py create mode 100644 tests/compat/test_corpus_replay.py create mode 100644 tests/test_preimage_golden.py create mode 100644 tests/test_seal_publication.py diff --git a/.gitignore b/.gitignore index ded1a21..a89422d 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,9 @@ build/ # The ledger is secret-bearing by construction — never commit it. .didrun/ *.didrun-bundle + +# Local-only working material (private paths, internal project references). +# Not for publication -- this repo ships docs/ to PyPI and GitHub. +research/ +docs/DIDRUN_V02_PACK.md +.didrun-history/ diff --git a/docs/COMPAT.md b/docs/COMPAT.md new file mode 100644 index 0000000..06cffdf --- /dev/null +++ b/docs/COMPAT.md @@ -0,0 +1,99 @@ +# Format compatibility + +didrun writes evidence that is meant to outlive the version that wrote it: a chain of +recorded events on disk, and a manifest published as a git note on a commit. This +document states what is frozen, what may change additively, and what v0.2 changed in a +way that is visible from outside. + +The rule behind all of it: **when didrun meets a format it does not understand, it +refuses and says so.** It never reads unknown evidence under the semantics it happens to +know — a confident verdict about a format the binary has never seen is exactly the +failure mode this tool exists to avoid. + +## The chain preimage is versioned by an explicit field list + +`ChainEntry.compute_hash` does not hash "the event". It hashes the fields named in +`ledger._PREIMAGE_FIELDS` for the entry's `preimage_version`. Version 1 is **frozen**: +its field list is a literal in the source and a golden test recomputes a known entry +hash from a fixed event, so any change to the field set fails that test rather than +silently invalidating every `entry_hash` ever written. + +Adding a field to `Event` therefore does not change existing hashes. Putting a field +into the *preimage* does, which is why a new preimage version is a deliberate, tested +step and not a side effect of adding an attribute. + +An entry whose `preimage_version` this binary does not know is not verified against a +guess: the chain check reports it as unverifiable and names the index. + +## Manifest versioning + +`MANIFEST_VERSION` is the note format's version. + +- **Additive fields do not bump it.** A new key that older notes simply lack is + backward-compatible in both directions. +- **A change to an existing field's type or meaning bumps it**, as does a change to the + claim-type vocabulary — those are the changes that make an old reader wrong rather + than merely incomplete. +- **A version newer than the reader is refused, not coerced.** `Manifest.from_json` + raises on a version above the one it knows and names both numbers. Older versions are + read. +- **Every new manifest key is read with `.get()` and a default that reproduces v1 + semantics.** A required key would break reading every note already published. + +When didrun falls back to resolving a manifest by tree id it scans the notes ref. A note +it cannot parse at all — a foreign note under the same ref, a corrupt body — is skipped +and counted, and the count is reported, so one bad note cannot hide every good one. A +note it *can* parse but whose version is too new is not skipped: it aborts the +resolution. The fallback returns the first note whose sealed tree matches; it does not +currently detect a second match. + +## The environment fingerprint is versioned in band + +`env_fingerprint()` returns `v:<16 hex chars>`. The prefix exists because the digest's +*shape* does not change when its key set does — two bare digests over different key sets +look comparable and are not. + +A value with **no prefix was produced by v0.1** and is **incomparable** with a versioned +one. It is not evidence that the environment drifted, and nothing may report it as such; +`capture.fingerprint_version()` returns `None` for it so a consumer can tell the two +apart. + +## Tree-digest semantics changed in v0.2 (release-note level) + +v0.1 excluded only `.didrun/` from a tree digest. v0.2 also excludes the archive root +`.didrun-history/`, so rotating a ledger into the archive no longer moves the digest of +the tree that just archived it. + +This is a **semantics change, not hygiene**: for a repository that carries a +`.didrun-history/` directory which is **not** gitignored, a tree digest taken by v0.2 +differs from one taken by v0.1, and claims recorded against the older digest grade +`stale`. For every other repository the digest is unchanged: with no such directory there +is nothing to exclude, and with a gitignored one `git add -A` already skipped it before +the exclusion pass ever ran. + +The one repository where this was measured before release gitignores `.didrun-history/`, +so no archived tree id there is affected. That is a measurement of one repository, not a +proof about all repositories. Both configurations are pinned by tests. + +`.didrun/` itself remains excluded unconditionally, regardless of gitignore state. + +## Two things v0.2 does not close + +Stated here because a trust tool's silence about its own limits is the same defect as an +overclaim. + +1. **A wrapped command inherits the caller's environment.** didrun records *that* the + command ran and *what* it exited with; it does not control what the environment made + the command do. `GOFLAGS=-exec=/usr/bin/true` yields an exit-0 event and a + `tree-exact` `tests-pass` claim for a command that ran no tests. No allowlist closes + this — `NODE_OPTIONS`, `PYTHONPATH`, `LD_PRELOAD`, `DYLD_INSERT_LIBRARIES`, + `PYTEST_ADDOPTS`, a user gitconfig, a `sitecustomize.py`, and a `PATH` shim all + produce it — and putting the effective environment into the event body would put the + richest secret carrier on a developer machine into the chain preimage. A declared + clean-environment mode is the honest shape for this, and it is not built. + +2. **The ledger has no retention or purge.** Captured stdout and stderr are stored as + content-addressed blobs under `.didrun/objects`. A credential that was scrubbed out of + git history survives there as a loose object until the ledger directory is removed by + hand. Treat the ledger as secret-bearing: it is gitignored by default, and it should + not be committed, shared, or attached to an issue. diff --git a/src/didrun/capture.py b/src/didrun/capture.py index b120206..2826344 100644 --- a/src/didrun/capture.py +++ b/src/didrun/capture.py @@ -35,6 +35,13 @@ # leaks a secret (redaction of captured *output* is a separate pass; see redact.py). _ENV_FINGERPRINT_KEYS = ("PATH", "SHELL", "LANG", "PWD", "VIRTUAL_ENV") +# The fingerprint's preimage is versioned IN BAND, because the digest's shape +# does not change when its key set does: two 16-hex strings over different key +# sets are silently incomparable, which reads as "the environment drifted" when +# the truth is "these were produced by different didrun versions". A bare digest +# with no prefix was produced by v0.1 and is incomparable, not drifted. +FINGERPRINT_VERSION = 1 + # The env var that gates Tier-2 traps. The trap does nothing unless this is set, # and it is set ONLY by `didrun run`, so an installed trap never records an # unrelated shell session on the machine (the privacy gate). @@ -47,7 +54,24 @@ def env_fingerprint(env: Optional[dict] = None) -> str: for key in _ENV_FINGERPRINT_KEYS: val = env.get(key, "") parts.append(f"{key}={val}") - return hashlib.sha256("\n".join(parts).encode("utf-8", "replace")).hexdigest()[:16] + digest = hashlib.sha256("\n".join(parts).encode("utf-8", "replace")).hexdigest()[:16] + return f"v{FINGERPRINT_VERSION}:{digest}" + + +def fingerprint_version(value: str) -> Optional[int]: + """The in-band version of a fingerprint, or None if it carries no prefix. + + ``None`` means the value came from a didrun that predates the prefix, so it + is INCOMPARABLE with a versioned one — a consumer must not read a mismatch + between the two as environment drift. + """ + prefix, sep, _digest = value.partition(":") + if not sep or not prefix.startswith("v"): + return None + digits = prefix[1:] + if not (digits.isascii() and digits.isdigit()): + return None + return int(digits) def run_wrapped(argv: list[str], session: Session, repo: Optional[Path] = None) -> Event: diff --git a/src/didrun/cli.py b/src/didrun/cli.py index d02f135..fc2ac4e 100644 --- a/src/didrun/cli.py +++ b/src/didrun/cli.py @@ -118,7 +118,14 @@ def cmd_seal(args) -> int: def cmd_verify(args) -> int: repo = Path(args.repo or os.getcwd()) session = _session(repo) - report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") + try: + report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") + except _manifest.ManifestError as exc: + # Evidence this binary cannot read is a graded refusal, not a crash. 2, + # not --strict's 1: "could not read the manifest" is a different fact + # from "the manifest graded badly". + print(f"didrun verify: {exc}", file=sys.stderr) + return 2 if args.html: Path(args.html).write_text(render.render_html(report), encoding="utf-8") print(f"wrote HTML report to {args.html}") @@ -152,7 +159,11 @@ def cmd_show(args) -> int: ) return 0 if ok else 1 # Default: the verdict view for a commit. - report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") + try: + report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") + except _manifest.ManifestError as exc: + print(f"didrun show: {exc}", file=sys.stderr) + return 2 if args.html: Path(args.html).write_text(render.render_html(report), encoding="utf-8") print(f"wrote HTML report to {args.html}") diff --git a/src/didrun/gitplumbing.py b/src/didrun/gitplumbing.py index a087d16..7bcbfab 100644 --- a/src/didrun/gitplumbing.py +++ b/src/didrun/gitplumbing.py @@ -142,11 +142,18 @@ def submodule_dirty(repo: Path) -> bool: # is stable across repos with different ignore rules. LEDGER_DIRNAME = ".didrun" +# Where rotated ledgers are kept. Same argument as the ledger dir, one step +# later: archiving a ledger is tooling housekeeping, so if the archive root +# counted, the act of archiving would move the digest and stale the very unit +# that just archived. This is a digest-SEMANTICS change from v0.1 and only for +# a repo that carries this directory WITHOUT gitignoring it — see docs/COMPAT.md. +ARCHIVE_DIRNAME = ".didrun-history" + def tree_digest( repo: Path, ledger_objects: Optional[Path] = None, - exclude: tuple[str, ...] = (LEDGER_DIRNAME,), + exclude: tuple[str, ...] = (LEDGER_DIRNAME, ARCHIVE_DIRNAME), ) -> Optional[str]: """Digest the current working tree (staged + unstaged + untracked). @@ -155,7 +162,7 @@ def tree_digest( are written into ``ledger_objects`` if given (zero pollution of the user's object DB) with the repo's real objects available as an alternate so the write-tree can reference existing blobs. ``exclude`` pathspecs are kept out - of the digest (the ledger dir by default). + of the digest (the ledger dir and the archive root by default). """ if not is_git_repo(repo): return None diff --git a/src/didrun/ledger.py b/src/didrun/ledger.py index 065d72a..080516a 100644 --- a/src/didrun/ledger.py +++ b/src/didrun/ledger.py @@ -21,7 +21,7 @@ import hashlib import json import os -from dataclasses import dataclass, field, asdict +from dataclasses import dataclass, field, asdict, fields as dataclass_fields from pathlib import Path from typing import Any, Iterator, Optional @@ -29,6 +29,33 @@ # an empty vs tampered-truncated log are distinguishable. GENESIS_HASH = "0" * 64 +# The chain preimage this binary writes. Every entry_hash ever recorded was +# computed over version 1, so version 1's field list is frozen forever: a name +# added to or removed from it stops every archived entry_hash reproducing. +# A future schema adds a *new* version here and leaves 1 untouched. +PREIMAGE_VERSION = 1 + +# canonical_json sorts keys, so the order within each tuple does not affect the +# digest — only membership does. +_PREIMAGE_FIELDS: dict[int, tuple[str, ...]] = { + 1: ( + "argv", + "cwd", + "env_fingerprint", + "observed_via", + "coverage", + "exit_code", + "started_at", + "ended_at", + "stdout_blob", + "stderr_blob", + "transcript_blob", + "tree_before", + "tree_after", + "submodule_dirty", + ), +} + # Observation provenance. Only wrapper/shim events witnessed an actual exit code; # a transcript only saw bytes on a terminal and must never claim an exit code. OBSERVED_VIA = ("wrapper", "shim", "trap", "transcript") @@ -123,9 +150,21 @@ def to_dict(self) -> dict: @classmethod def from_dict(cls, d: dict) -> "Event": - d = dict(d) - d["argv"] = tuple(d.get("argv", ())) - return cls(**d) + # Unknown keys are dropped rather than fatal: a ledger written by a + # newer binary must stay *readable* by this one. What is dropped is not + # discarded silently — unknown_event_keys_of() reports it, and an entry + # carrying any is graded unverifiable rather than recomputed. + known = {k: v for k, v in d.items() if k in _EVENT_FIELD_NAMES} + known["argv"] = tuple(known.get("argv", ())) + return cls(**known) + + +_EVENT_FIELD_NAMES = frozenset(f.name for f in dataclass_fields(Event)) + + +def unknown_event_keys_of(d: dict) -> tuple[str, ...]: + """Keys in a stored event body that this binary's Event schema lacks.""" + return tuple(sorted(k for k in d if k not in _EVENT_FIELD_NAMES)) @dataclass @@ -136,16 +175,48 @@ class ChainEntry: prev_hash: str event: Event entry_hash: str = "" + # Neither of these is serialized into the event body or hashed; ChainEntry + # is built by hand, never via asdict, so they are free. + preimage_version: int = PREIMAGE_VERSION + unknown_event_keys: tuple[str, ...] = () def compute_hash(self) -> str: # The hash binds the previous link, the position, and the canonical - # event body. Reordering, truncation, or mutation all break it. + # event body, projected onto the field list declared for this preimage + # version. Reordering, truncation, or mutation all break it. At version + # 1 the projection is byte-identical to the whole to_dict(), which is + # what keeps every archived entry_hash reproducing. + field_names = _PREIMAGE_FIELDS.get(self.preimage_version) + if field_names is None: + raise LedgerError( + f"unknown chain preimage version: {self.preimage_version}" + ) + d = self.event.to_dict() body = canonical_json( - {"index": self.index, "prev": self.prev_hash, "event": self.event.to_dict()} + { + "index": self.index, + "prev": self.prev_hash, + "event": {k: d[k] for k in field_names}, + } ) return sha256_hex(self.prev_hash.encode("ascii") + body) +@dataclass(frozen=True) +class ChainVerdict: + """The honest three-way answer about a chain, not a boolean. + + ``unverifiable`` is the state the boolean could not express: the entry was + written by something this binary does not fully understand, so its hash + cannot be recomputed. That is never the same statement as tamper. + """ + + status: str # "intact" | "broken" | "unverifiable" | "empty" + first_broken_index: Optional[int] = None + unverifiable_index: Optional[int] = None + reason: str = "" + + class BlobStore: """Content-addressed byte store. @@ -231,6 +302,10 @@ def append(self, event: Event) -> ChainEntry: "entry_hash": entry.entry_hash, "event": event.to_dict(), } + # Omitted at version 1 so a v1 record on disk stays byte-identical to + # what every previous release wrote, and an older binary keeps reading it. + if entry.preimage_version != 1: + rec["preimage_version"] = entry.preimage_version # Append one canonical line. Newline-delimited so the tail read is cheap. with self.log_path.open("ab") as fh: fh.write(canonical_json(rec) + b"\n") @@ -247,33 +322,98 @@ def entries(self) -> Iterator[ChainEntry]: if not line: continue rec = json.loads(line) + body = rec["event"] entry = ChainEntry( index=rec["index"], prev_hash=rec["prev_hash"], - event=Event.from_dict(rec["event"]), + event=Event.from_dict(body), entry_hash=rec["entry_hash"], + # Absent means version 1: that is what every archived + # record looks like and it must keep meaning v1. + preimage_version=rec.get("preimage_version", 1), + unknown_event_keys=unknown_event_keys_of(body), ) yield entry def events(self) -> list[Event]: return [e.event for e in self.entries()] - def verify_chain(self) -> tuple[bool, Optional[int]]: - """Recompute the chain. Returns (ok, first_broken_index). + def verify_chain_detail(self) -> ChainVerdict: + """Recompute the chain and report which of the four states it is in. - ``first_broken_index`` is the index whose stored hash or linkage does - not recompute — the earliest point tamper is detectable. ``None`` when - the chain is intact. + Stops at the first entry that is broken or that cannot be recomputed. + An entry this binary cannot recompute — an unknown preimage version, or + event keys outside the version's declared-exhaustive field list — is + ``unverifiable``: the projection would quietly hash a subset and + reproduce the stored digest, so reporting ``intact`` there would be a + lie and reporting ``broken`` would be a different one. """ prev = GENESIS_HASH expected_index = 0 + seen = False for entry in self.entries(): + seen = True if entry.index != expected_index: - return False, entry.index + return ChainVerdict( + "broken", + first_broken_index=entry.index, + reason=( + f"entry index {entry.index} out of sequence " + f"(expected {expected_index})" + ), + ) if entry.prev_hash != prev: - return False, entry.index - if entry.compute_hash() != entry.entry_hash: - return False, entry.index + return ChainVerdict( + "broken", + first_broken_index=entry.index, + reason=f"entry {entry.index} does not link to the previous entry", + ) + if entry.unknown_event_keys: + return ChainVerdict( + "unverifiable", + unverifiable_index=entry.index, + reason=( + f"entry {entry.index} carries event keys this binary does " + f"not know ({', '.join(entry.unknown_event_keys)}); its " + f"preimage cannot be recomputed" + ), + ) + try: + recomputed = entry.compute_hash() + except LedgerError as exc: + return ChainVerdict( + "unverifiable", + unverifiable_index=entry.index, + reason=f"entry {entry.index}: {exc}", + ) + if recomputed != entry.entry_hash: + return ChainVerdict( + "broken", + first_broken_index=entry.index, + reason=f"entry {entry.index} does not recompute to its stored hash", + ) prev = entry.entry_hash expected_index += 1 - return True, None + if not seen: + return ChainVerdict("empty") + return ChainVerdict("intact") + + def verify_chain(self) -> tuple[bool, Optional[int]]: + """Recompute the chain. Returns (ok, first_broken_index). + + ``first_broken_index`` is the index whose stored hash or linkage does + not recompute — the earliest point tamper is detectable. ``None`` when + the chain is intact. + + A thin wrapper over ``verify_chain_detail``; anything other than intact + or empty is ``ok=False``, including ``unverifiable``, which collapses to + the same boolean but is never the same fact. Callers that need to tell + "cannot recompute" from "tamper" must use ``verify_chain_detail``. This + signature is depended on and does not change. + """ + verdict = self.verify_chain_detail() + if verdict.status in ("intact", "empty"): + return True, None + if verdict.status == "broken": + return False, verdict.first_broken_index + return False, verdict.unverifiable_index diff --git a/src/didrun/manifest.py b/src/didrun/manifest.py index 451965d..2fce0a6 100644 --- a/src/didrun/manifest.py +++ b/src/didrun/manifest.py @@ -56,9 +56,23 @@ def to_json(self) -> bytes: @classmethod def from_json(cls, data: bytes) -> "Manifest": + """Parse a note body. Refuses a version it does not understand. + + Refuse, never coerce: a newer manifest may have changed what an + existing field MEANS, and grading it under this version's semantics + would produce a confident verdict about a format this binary has never + seen. Older versions are accepted — new fields are additive and are + read with a v1-reproducing default (see docs/COMPAT.md). + """ d = json.loads(data) + version = d["version"] + if version > MANIFEST_VERSION: + raise ManifestError( + f"manifest version {version} is newer than this didrun understands " + f"(max {MANIFEST_VERSION}); upgrade didrun" + ) return cls( - version=d["version"], + version=version, commit=d["commit"], tree=d["tree"], claims=d["claims"], @@ -155,13 +169,29 @@ def seal( secrets_override=bool(findings) and allow_secrets, ) + # Publication and the watermark are one atomic pair. A published note with + # no watermark re-seals every earlier claim forever (the watermark is what + # scopes a seal); a watermark with no published note reports evidence that + # was never written. Nothing is recorded until the note is on the commit, + # and if recording fails the note goes back to what it was. + prior_note = _read_note(repo, commit) if write_notes else None if write_notes: _attach_note(repo, commit, manifest) - if bundle_path is not None: - Path(bundle_path).write_bytes(manifest.to_json()) - # Advance the seal watermark (append-only, like everything in the ledger). - _record_seal(session, commit, tree, len(all_claims)) + # Everything after publication runs inside the rollback. A caller-supplied + # bundle path is the likeliest failure in this function, and an exception + # escaping here would leave the note published with no watermark -- exactly + # the state the pair above exists to prevent. + try: + if bundle_path is not None: + Path(bundle_path).write_bytes(manifest.to_json()) + # Advance the seal watermark (append-only, like everything in the ledger). + _record_seal(session, commit, tree, len(all_claims)) + except Exception as exc: + raise ManifestError( + f"seal aborted: no watermark was recorded; " + f"{_rollback_note(repo, commit, prior_note, write_notes)}; cause: {exc}" + ) return manifest @@ -174,6 +204,7 @@ class VerifyReport: results: list[GradeResult] coverage: dict secrets_override: bool = False + notes_skipped: int = 0 # notes the tree fallback could not parse at all @property def worst_status(self) -> str: @@ -204,7 +235,7 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor commit = _rev(repo, commitish) tree = gitplumbing.commit_tree(repo, commitish) if commit else None - manifest, resolved_by = _resolve_manifest(repo, commit, tree) + manifest, resolved_by, notes_skipped = _resolve_manifest(repo, commit, tree) if manifest is None: return VerifyReport( commit=commit or "", @@ -212,6 +243,7 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor resolved_by="none", results=[], coverage={}, + notes_skipped=notes_skipped, ) # Re-grade the MANIFEST'S OWN claims against its sealed tree (deterministic @@ -236,6 +268,7 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor results=results, coverage=manifest.coverage, secrets_override=manifest.secrets_override, + notes_skipped=notes_skipped, ) @@ -298,18 +331,107 @@ def _rev(repo: Path, commitish: str) -> Optional[str]: return proc.stdout.strip() if proc.returncode == 0 else None +def _git_stderr(proc) -> str: + err = proc.stderr or b"" + if isinstance(err, bytes): + err = err.decode("utf-8", "replace") + return err.strip()[:500] + + +def _read_note(repo: Path, commit: str) -> Optional[bytes]: + """The note body currently attached to ``commit``, or None if there is none. + + Read before publishing so a failed seal can put back what it overwrote. + """ + proc = subprocess.run( + ["git", "notes", f"--ref={NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + def _attach_note(repo: Path, commit: str, manifest: Manifest) -> None: - subprocess.run( - ["git", "notes", f"--ref={NOTES_REF}", "add", "-f", "-m", manifest.to_json().decode("ascii"), commit], + """Publish the manifest as ``commit``'s note. Fails closed. + + The body travels on stdin (``-F -``), not as an argv element: notes grow + with claim count and the command line does not. ``-F -`` and ``-m`` produce + the identical note object, so this moves no stored byte. + """ + proc = subprocess.run( + ["git", "notes", f"--ref={NOTES_REF}", "add", "-f", "-F", "-", commit], cwd=str(repo), + input=manifest.to_json(), capture_output=True, - text=True, + ) + if proc.returncode != 0: + raise ManifestError( + f"could not publish note to {NOTES_REF} for {commit}: " + f"git exited {proc.returncode}: {_git_stderr(proc)}" + ) + + +def _restore_note(repo: Path, commit: str, prior: Optional[bytes]) -> bool: + """Put ``commit``'s note back the way it was. Best-effort; never raises. + + ``_attach_note`` forces (-f), so a seal can overwrite an existing note. + Rolling back by REMOVING the note would turn "the watermark write failed" + into "the commit's earlier evidence is gone", so the prior body is + re-attached; removal happens only when there was nothing there before. + Returns whether the restore is known to have succeeded. + """ + try: + if prior is None: + proc = subprocess.run( + ["git", "notes", f"--ref={NOTES_REF}", "remove", "--ignore-missing", commit], + cwd=str(repo), + capture_output=True, + ) + else: + proc = subprocess.run( + ["git", "notes", f"--ref={NOTES_REF}", "add", "-f", "-F", "-", commit], + cwd=str(repo), + input=prior, + capture_output=True, + ) + return proc.returncode == 0 + except Exception: + return False + + +def _rollback_note( + repo: Path, commit: str, prior: Optional[bytes], published: bool +) -> str: + """Undo this seal's publication and say, exactly, what state the commit is in.""" + if not published: + return "no note was published, so none was rolled back" + restored = _restore_note(repo, commit, prior) + if prior is None: + return ( + "the note was rolled back and the commit was left noteless (it carried none before)" + if restored + else "the note could NOT be rolled back and this seal's note is STILL ATTACHED" + ) + return ( + "the note was rolled back and the commit's prior note was restored" + if restored + else "the note was NOT rolled back and the commit's prior note was NOT restored" ) def _resolve_manifest( repo: Path, commit: Optional[str], tree: Optional[str] -) -> tuple[Optional[Manifest], str]: +) -> tuple[Optional[Manifest], str, int]: + """Find the manifest for a commit. Returns (manifest, resolved_by, skipped). + + ``skipped`` counts notes on the tree-fallback path that could not be + parsed at all — a foreign note under the same ref, or a corrupt body. Those + are skipped so one bad note cannot hide every good one, but they are + counted so the caller can say so rather than reporting a silent "none". + A *version* refusal is not a skip: it means didrun met evidence it cannot + read, and it propagates. + """ + skipped = 0 # Try the note on the exact commit. if commit: proc = subprocess.run( @@ -319,7 +441,7 @@ def _resolve_manifest( text=True, ) if proc.returncode == 0 and proc.stdout.strip(): - return Manifest.from_json(proc.stdout.encode("ascii")), "commit" + return Manifest.from_json(proc.stdout.encode("ascii")), "commit", skipped # Tree fallback: scan notes for one whose sealed tree matches (survives amend). if tree: listing = subprocess.run( @@ -343,11 +465,19 @@ def _resolve_manifest( if show.returncode == 0 and show.stdout.strip(): try: m = Manifest.from_json(show.stdout.encode("ascii")) + except ManifestError: + # A version this binary cannot read. Fail closed: the + # alternative is scanning past it and reporting the + # verdict of some older note as if it were current. + raise except Exception: + skipped += 1 continue + # First match wins (unchanged). Ambiguity policy is not + # this unit's; see docs/COMPAT.md. if m.tree == tree: - return m, "tree-fallback" - return None, "none" + return m, "tree-fallback", skipped + return None, "none", skipped def _scan_for_secrets(session: Session, results: list[GradeResult]) -> list[redact.Finding]: diff --git a/src/didrun/render.py b/src/didrun/render.py index 564b980..50501ce 100644 --- a/src/didrun/render.py +++ b/src/didrun/render.py @@ -66,6 +66,15 @@ def _sanitize(s: str) -> str: return "".join(out) +def _skipped_notes(n: int) -> str: + """Notes under the didrun ref that could not be parsed at all. + + Reported rather than swallowed: "no manifest" and "there were notes here + and none of them could be read" are different facts. + """ + return f"{n} note{'' if n == 1 else 's'} skipped (unparseable)" + + def render_verdict(report, width: int = 80) -> str: """Render a VerifyReport as the CLI instrument panel.""" lines: list[str] = [] @@ -75,6 +84,8 @@ def render_verdict(report, width: int = 80) -> str: if worst == "empty": lines.append(_c("○ NO CLAIMS", "33") + " nothing sealed for this commit") lines.append(f" commit {report.commit[:12] or '(none)'} · resolved-by {report.resolved_by}") + if report.notes_skipped: + lines.append(_c(f" ! {_skipped_notes(report.notes_skipped)}", "33")) return "\n".join(lines) token, color, marker, _gloss = _GRADE_DISPLAY.get(worst, ("UNKNOWN", "33", "?", "")) @@ -92,6 +103,8 @@ def render_verdict(report, width: int = 80) -> str: ) if report.secrets_override: lines.append(_c(" ! sealed with --allow-secrets (redacted export)", "33")) + if report.notes_skipped: + lines.append(_c(f" ! {_skipped_notes(report.notes_skipped)}", "33")) lines.append("") # Per-claim table. Failing/stale first (already sorted). diff --git a/tests/compat/__init__.py b/tests/compat/__init__.py new file mode 100644 index 0000000..5ed974b --- /dev/null +++ b/tests/compat/__init__.py @@ -0,0 +1,15 @@ +"""Compat corpus replay harness — CONTENT-BLIND by construction. + +THE CONTENT RULE (non-negotiable, and the reason this package exists at all): +this harness may READ ledger bytes; it may never print, log, echo, or assert on +them. Every assertion in here compares digests, booleans, counts, or path +shapes. ``assert computed_hash == stored_hash`` is fine — a hash is not +content. ``assert line == ...`` is forbidden, because pytest prints both sides +of a failed comparison, and a ledger line is a verbatim record of everything a +session ran and printed: argv, stdout, stderr, and any credential that passed +through them. + +The tree this harness is aimed at is 174 MB of secret-bearing evidence for a +build that is still running. A harness that echoes one line of it into a CI log +or a chat transcript has done more damage than any bug it could have found. +""" diff --git a/tests/compat/conftest.py b/tests/compat/conftest.py new file mode 100644 index 0000000..fe4e7bc --- /dev/null +++ b/tests/compat/conftest.py @@ -0,0 +1,154 @@ +"""Fixtures and guardrails for the compat replay harness. + +The corpus leg is OFF unless ``DIDRUN_COMPAT_CORPUS`` points at a copy of an +archive tree, so CI is green by construction and no unattended run can ever +touch a live ledger. +""" + +from __future__ import annotations + +import hashlib +import os +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +import pytest + +from . import synthetic + +ENV_VAR = "DIDRUN_COMPAT_CORPUS" + +SKIP_UNSET = ( + "set DIDRUN_COMPAT_CORPUS to a COPY of an archive tree to run the corpus replay" +) +SKIP_MISSING = ( + "DIDRUN_COMPAT_CORPUS does not point at a directory; " + "point it at a COPY of an archive tree to run the corpus replay" +) +SKIP_LIVE = ( + "DIDRUN_COMPAT_CORPUS points at a live repo (it has both a .didrun ledger " + "and a .git dir); this harness must never read a ledger something else is " + "writing. Copy the tree WITHOUT its live .didrun directory (the archives " + "under .didrun-history and the git dir are what the legs need) and point " + "the variable at the copy" +) +SKIP_REFUSED = ( + "DIDRUN_COMPAT_CORPUS points at (or inside) a tree this harness refuses by " + "name: it is live, secret-bearing evidence for a running build. Make a copy " + "on a separate volume and point the variable at the copy" +) + +# The refused trees, stored as sha256 digests of their absolute paths rather +# than as literals. Two rules meet here and a digest is the only thing that +# satisfies both: the pack requires these exact paths refused BY NAME, and +# RULES 17 forbids any private absolute path, repo name, or home directory in a +# shipped file. A one-way digest names nothing and still matches exactly. +# A heuristic is not a guardrail when the thing it guards is a running build's +# evidence, so this sits in front of the live-repo heuristic, not instead of it. +_REFUSED_PATH_DIGESTS = frozenset( + { + "44b1bd2169fdc4af5ac880330a6917f49456f19ad63efeb79326655e5cd3b4d2", + "08a945e5d11024bbc32f34f1f5dbf0b57c3590c08177d0391c0e0172a777acf1", + } +) + + +def path_digest(path) -> str: + """sha256 of an absolute path string, trailing separator normalized away.""" + s = str(path).rstrip("/") + return hashlib.sha256(s.encode("utf-8")).hexdigest() + + +def is_refused(path, digests=_REFUSED_PATH_DIGESTS) -> bool: + """True if ``path``, its realpath, or any ancestor of either is refused.""" + for candidate in {str(path), os.path.realpath(str(path))}: + p = Path(candidate) + for ancestor in (p, *p.parents): + if path_digest(ancestor) in digests: + return True + return False + + +def resolve_corpus_root(environ, digests=_REFUSED_PATH_DIGESTS) -> tuple: + """Return ``(root, skip_reason)``. Exactly one of the two is None. + + Pure over its ``environ`` and ``digests`` arguments so both the skip + behaviour and the refusal branch are testable without depending on what the + ambient environment holds or on knowing a refused path. + """ + raw = environ.get(ENV_VAR, "").strip() + if not raw: + return None, SKIP_UNSET + root = Path(raw) + if is_refused(root, digests): + return None, SKIP_REFUSED + if not root.is_dir(): + return None, SKIP_MISSING + if (root / ".didrun").exists() and (root / ".git").exists(): + return None, SKIP_LIVE + return root, None + + +@dataclass(frozen=True) +class ReplaySource: + """One tree the legs run over. ``expected`` is set only for synthetic.""" + + name: str + root: Path + expected: Optional[synthetic.Expectations] = None + + +@pytest.fixture(scope="session") +def synthetic_corpus(tmp_path_factory) -> synthetic.SyntheticCorpus: + return synthetic.build_corpus(tmp_path_factory.mktemp("synthetic-corpus")) + + +@pytest.fixture(scope="session") +def known_bad_corpus(tmp_path_factory) -> Path: + return synthetic.build_known_bad(tmp_path_factory.mktemp("known-bad-corpus")) + + +@pytest.fixture(scope="session") +def corpus_root() -> Path: + root, reason = resolve_corpus_root(os.environ) + if root is None: + pytest.skip(reason) + return root + + +@pytest.fixture(params=["synthetic", "corpus"]) +def replay_source(request) -> ReplaySource: + if request.param == "synthetic": + fixture = request.getfixturevalue("synthetic_corpus") + return ReplaySource("synthetic", fixture.root, fixture.expected) + return ReplaySource("corpus", request.getfixturevalue("corpus_root")) + + +def has_git_repo(root: Path) -> bool: + proc = subprocess.run( + ["git", "-C", str(root), "rev-parse", "--git-dir"], capture_output=True + ) + return proc.returncode == 0 + + +# --- the run summary --------------------------------------------------------- +# Counts only. Every line put in here has been through _assert_no_content first. + +_SUMMARY: list = [] + + +def record_summary(line: str) -> None: + _SUMMARY.append(line) + + +def pytest_terminal_summary(terminalreporter, exitstatus, config) -> None: + tr = terminalreporter + tr.write_sep("-", "compat replay") + root, _reason = resolve_corpus_root(os.environ) + if root is None: + # In these exact words: a green synthetic run is not corpus validation. + tr.write_line("corpus: SKIPPED (unverified)") + for line in _SUMMARY: + tr.write_line(line) diff --git a/tests/compat/synthetic.py b/tests/compat/synthetic.py new file mode 100644 index 0000000..de09293 --- /dev/null +++ b/tests/compat/synthetic.py @@ -0,0 +1,383 @@ +"""A deterministic, archive-shaped synthetic corpus. + +The real corpus cannot ship: it is secret-bearing, it is 174 MB, and it belongs +to a running build. So CI gets this instead — a tree with the same *shapes* the +real archives have, generated from literals with no randomness and no wall +clock, carrying one instance of every outcome the replay legs are supposed to +detect. + +Nothing in here is real. Paths, digests, argv and labels are invented; the only +product code used is the code under test (``Event``, ``ChainEntry``, +``Manifest``), so the fixture's shape follows the schema instead of drifting +from it. +""" + +from __future__ import annotations + +import json +import os +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +from didrun.claims import Claim, GradeResult +from didrun.ledger import ( + GENESIS_HASH, + ChainEntry, + Event, + canonical_json, + sha256_hex, +) +from didrun.manifest import MANIFEST_VERSION, Manifest + +# Fixed identities and timestamps: the fixture must be byte-identical on every +# machine and every run, so nothing here may read the clock or the environment. +_GIT_ENV = { + "GIT_AUTHOR_NAME": "compat fixture", + "GIT_AUTHOR_EMAIL": "fixture@invalid", + "GIT_COMMITTER_NAME": "compat fixture", + "GIT_COMMITTER_EMAIL": "fixture@invalid", + "GIT_AUTHOR_DATE": "2020-01-01T00:00:00+00:00", + "GIT_COMMITTER_DATE": "2020-01-01T00:00:00+00:00", +} + +_SYNTHETIC_CWD = "/w/repo" +_SYNTHETIC_FINGERPRINT = "v1:0123456789abcdef" +_SYNTHETIC_TREE = "3" * 40 + + +@dataclass(frozen=True) +class Expectations: + """What the legs must report for this fixture. Declared, never derived.""" + + # Ordered by sorted ledger-directory path, which is the order the harness + # walks. See build_corpus for which unit is which. + log_statuses: tuple + log_entries: tuple + objects: dict + notes: int + claim_shape: dict + + +@dataclass(frozen=True) +class SyntheticCorpus: + root: Path + expected: Expectations + + +def _git(root: Path, *args: str, stdin: Optional[bytes] = None) -> bytes: + """Run git in ``root``. Never echoes git's output — it can carry paths.""" + proc = subprocess.run( + ["git", "-C", str(root), *args], + input=stdin, + capture_output=True, + env={**os.environ, **_GIT_ENV}, + ) + if proc.returncode != 0: + raise RuntimeError(f"synthetic fixture: git {args[0]} failed") + return proc.stdout + + +def _init_repo(root: Path) -> None: + root.mkdir(parents=True, exist_ok=True) + _git(root, "init", "-q") + _git(root, "config", "user.email", "fixture@invalid") + _git(root, "config", "user.name", "compat fixture") + _git(root, "config", "commit.gpgsign", "false") + # Git's background maintenance can repack while a test walks objects. + _git(root, "config", "gc.auto", "0") + _git(root, "config", "maintenance.auto", "false") + + +def _commit(root: Path, name: str, body: str) -> str: + (root / name).write_text(body, encoding="ascii") + _git(root, "add", name) + _git(root, "commit", "-qm", f"fixture {name}") + return _git(root, "rev-parse", "HEAD").decode("ascii").strip() + + +def _attach_note(root: Path, commit: str, body: bytes) -> None: + _git( + root, + "notes", + "--ref=refs/notes/didrun", + "add", + "-f", + "-F", + "-", + commit, + stdin=body, + ) + + +# --- the ledger side --------------------------------------------------------- + + +def _event(i: int) -> Event: + return Event( + argv=("pytest", "-q"), + cwd=_SYNTHETIC_CWD, + env_fingerprint=_SYNTHETIC_FINGERPRINT, + observed_via="wrapper", + coverage="complete", + exit_code=0, + started_at=1000000.0 + i, + ended_at=1000001.0 + i, + stdout_blob=sha256_hex(f"synthetic-stdout-{i}".encode("ascii")), + stderr_blob=None, + transcript_blob=None, + tree_before=_SYNTHETIC_TREE, + tree_after=_SYNTHETIC_TREE, + submodule_dirty=False, + ) + + +def _chain(count: int) -> list: + """``count`` well-formed, correctly chained records.""" + records = [] + prev = GENESIS_HASH + for i in range(count): + entry = ChainEntry(index=i, prev_hash=prev, event=_event(i)) + entry.entry_hash = entry.compute_hash() + records.append( + { + "index": i, + "prev_hash": prev, + "entry_hash": entry.entry_hash, + "event": entry.event.to_dict(), + } + ) + prev = entry.entry_hash + return records + + +def _write_log(path: Path, records: list, torn_tail: bool = False) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("wb") as fh: + for rec in records: + fh.write(canonical_json(rec) + b"\n") + if torn_tail: + # A process killed mid-append: a partial line, no newline. This is + # the commonest real-world damage and it must not read as tamper. + fh.write(canonical_json(_chain(1)[0])[:37]) + + +def _write_objects(objects: Path, flat: int, fanout: int) -> None: + """An objects/ dir carrying both namespaces, like 113 of the real archives.""" + objects.mkdir(parents=True, exist_ok=True) + for i in range(flat): + digest = sha256_hex(f"synthetic-blob-{i}".encode("ascii")) + (objects / digest).write_bytes(b"synthetic blob\n") + for i in range(fanout): + loose = sha256_hex(f"synthetic-loose-{i}".encode("ascii"))[:40] + d = objects / loose[:2] + d.mkdir(exist_ok=True) + (d / loose[2:]).write_bytes(b"synthetic loose object\n") + + +# --- the note side ----------------------------------------------------------- + + +def _claim_entry( + ctype: str, + label: str, + grade: str, + supporting: Optional[int], + declared_at: int, + pathspecs: tuple = (), + exit_code: Optional[int] = 0, +) -> dict: + """One graded claim in the shape seal() writes into a note.""" + result = GradeResult( + claim=Claim( + ctype=ctype, + label=label, + event_indices=(supporting,) if supporting is not None else (), + pathspecs=pathspecs, + declared_at_index=declared_at, + ), + grade=grade, + reason="synthetic fixture", + delta=[], + supporting_event_index=supporting, + exit_code=exit_code, + ) + d = result.to_dict() + d["claim"]["argv_preview"] = ["pytest", "-q"] + return d + + +def _coverage() -> dict: + return {"events": 3, "complete": 3, "observed_text_only": 0, "display_only": 0} + + +def build_corpus(root: Path) -> SyntheticCorpus: + """Build the good-path fixture and return it with its declared expectations. + + Ledger directories, in the sorted order the harness walks them: + + 0 unit-01-good-nested nested /.didrun/, 3 entries, intact + 1 unit-02-good-flat flat /, 1 entry, intact + 2 unit-03-torn-tail 2 entries + a partial final line + 3 unit-04-unknown-key an event key this binary does not know + 4 unit-05-unknown-preimage a preimage version this binary does not know + 5 unit-06-empty a zero-byte log + 6 unit-07-broken-hash 3 entries, entry 1's stored hash mutated + """ + root = Path(root) + _init_repo(root) + history = root / ".didrun-history" + + nested = history / "unit-01-good-nested" / ".didrun" + _write_log(nested / "session.log", _chain(3)) + _write_objects(nested / "objects", flat=2, fanout=2) + # Present in 94 / 62 of the real archives; the legs only stat their shape. + (nested / "claims.jsonl").write_bytes(b"") + (nested / "seals.jsonl").write_bytes(b"") + + flat = history / "unit-02-good-flat" + _write_log(flat / "session.log", _chain(1)) + _write_objects(flat / "objects", flat=1, fanout=1) + + _write_log( + history / "unit-03-torn-tail" / ".didrun" / "session.log", + _chain(2), + torn_tail=True, + ) + + unknown_key = _chain(1) + unknown_key[0]["event"]["signal"] = 9 + _write_log(history / "unit-04-unknown-key" / ".didrun" / "session.log", unknown_key) + + unknown_preimage = _chain(1) + unknown_preimage[0]["preimage_version"] = 2 + _write_log( + history / "unit-05-unknown-preimage" / ".didrun" / "session.log", + unknown_preimage, + ) + + empty = history / "unit-06-empty" / ".didrun" / "session.log" + empty.parent.mkdir(parents=True, exist_ok=True) + empty.write_bytes(b"") + + broken = _chain(3) + stored = broken[1]["entry_hash"] + broken[1]["entry_hash"] = ("0" if stored[0] != "0" else "1") + stored[1:] + _write_log(history / "unit-07-broken-hash" / ".didrun" / "session.log", broken) + + # Two notes. The second deliberately omits `secrets_override` — the one key + # from_json already reads with a v1-reproducing default — so the round-trip + # leg's additive-key allowlist is exercised by the good path, not only by a + # meta-test. + commit_a = _commit(root, "work-a.txt", "one\n") + manifest_a = Manifest( + version=MANIFEST_VERSION, + commit=commit_a, + tree=_SYNTHETIC_TREE, + claims=[ + _claim_entry("tests-pass", "suite", "tree-exact", 0, 0), + _claim_entry( + "lint-clean", "lint", "scope-exact", 1, 2, pathspecs=("src/",) + ), + _claim_entry("command-succeeded", "build", "stale", 0, 6), + ], + coverage=_coverage(), + secrets_override=False, + ) + _attach_note(root, commit_a, manifest_a.to_json()) + + commit_b = _commit(root, "work-b.txt", "two\n") + body_b = json.loads( + Manifest( + version=MANIFEST_VERSION, + commit=commit_b, + tree=_SYNTHETIC_TREE, + claims=[_claim_entry("tests-pass", "suite", "failed", 0, 0, exit_code=1)], + coverage=_coverage(), + secrets_override=False, + ).to_json() + ) + del body_b["secrets_override"] + _attach_note(root, commit_b, canonical_json(body_b)) + + expected = Expectations( + log_statuses=( + "intact", + "intact", + "torn-tail", + "unverifiable@0", + "unverifiable@0", + "empty", + "broken@1", + ), + log_entries=(3, 1, 2, 1, 1, 0, 3), + objects={"dirs": 2, "flat-64hex": 3, "git-fanout": 3}, + notes=2, + claim_shape={ + "notes": 2, + "claims": 4, + "multi-index": 0, + "max-pathspecs": 1, + "pathspecs": {0: 3, 1: 1}, + "gap-ge-1": 2, + "gap-ge-5": 1, + "grades": {"failed": 1, "scope-exact": 1, "stale": 1, "tree-exact": 1}, + }, + ) + return SyntheticCorpus(root=root, expected=expected) + + +def build_known_bad(root: Path) -> Path: + """A second tree whose *note* and *objects* legs must come back red. + + Kept separate from build_corpus so the good fixture stays green: a leg that + is red for a reason that is actually correct behaviour is a leg nobody will + trust. + """ + root = Path(root) + _init_repo(root) + history = root / ".didrun-history" + + # A third path shape in objects/: neither a flat 64-hex blob nor a git + # fan-out dir. The classifier must fail closed on it rather than ignore it. + residue = history / "unit-bad-objects" / ".didrun" / "objects" + _write_objects(residue, flat=1, fanout=1) + (residue / "pack").mkdir() + (residue / "notes.txt").write_bytes(b"third shape\n") + _write_log(history / "unit-bad-objects" / ".didrun" / "session.log", _chain(1)) + + commit = _commit(root, "work.txt", "bad\n") + # A note from a manifest version this binary must refuse, not coerce. + too_new = json.loads( + Manifest( + version=MANIFEST_VERSION, + commit=commit, + tree=_SYNTHETIC_TREE, + claims=[], + coverage=_coverage(), + ).to_json() + ) + too_new["version"] = MANIFEST_VERSION + 1 + _attach_note(root, commit, canonical_json(too_new)) + return root + + +def multi_index_note_body(commit: str = "a" * 40) -> bytes: + """A note carrying the claim shape the corpus says does not exist. + + 0 of 846 real claims bind more than one event; two of the audit's blocker + findings turned on that. This is the counter-example the leg must catch. + """ + entry = _claim_entry("tests-pass", "suite", "tree-exact", 0, 0) + entry["claim"]["event_indices"] = [0, 1] + two_specs = _claim_entry( + "lint-clean", "lint", "scope-exact", 1, 1, pathspecs=("src/", "tests/") + ) + return Manifest( + version=MANIFEST_VERSION, + commit=commit, + tree=_SYNTHETIC_TREE, + claims=[entry, two_specs], + coverage=_coverage(), + ).to_json() diff --git a/tests/compat/test_corpus_replay.py b/tests/compat/test_corpus_replay.py new file mode 100644 index 0000000..15e9413 --- /dev/null +++ b/tests/compat/test_corpus_replay.py @@ -0,0 +1,732 @@ +"""The compat corpus replay harness — four legs, run content-blind. + +THE CONTENT RULE (non-negotiable): this module may READ ledger bytes; it may +never print, log, echo, or assert on them. Every assertion below compares +digests, booleans, counts, or path shapes. ``assert computed == stored`` over +two hashes is fine — a hash is not content. ``assert line == ...`` is +forbidden, because pytest prints both sides of a failed comparison, and a +ledger line is a verbatim record of everything a session ran and printed: +argv, stdout, stderr, and whatever credential passed through them. The tree +this harness is aimed at is live, secret-bearing evidence for a build that is +still running; a harness that leaks one line of it into a CI log or a chat +transcript has done more damage than any bug it could have found. + +``_assert_no_content`` is the net under that rule. It is a bound on the SHAPE +of what can escape into an assertion message, not a proof that nothing does — +the only real guarantee is that no leg ever puts a ledger-derived value into a +payload in the first place. + +Why the legs are these four (red-team §3.5): a harness asserting the corpus's +815/30/1 grade histogram would have passed on every defect in the audit. A +histogram is a snapshot, not a regression test. The legs that earn their keep +are the chain recompute — which would have caught a schema break across 5,558 +archived events before it shipped — and the claim-SHAPE invariants, which is +where two of the audit's blocker findings actually lived. + +The corpus leg is env-gated and OFF by default. A green synthetic run is not +corpus validation, and the run summary says so in those words. +""" + +from __future__ import annotations + +import json +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Optional + +import pytest + +from didrun.ledger import ( + GENESIS_HASH, + ChainEntry, + Event, + LedgerError, + sha256_hex, + unknown_event_keys_of, +) +from didrun.manifest import MANIFEST_VERSION, NOTES_REF, Manifest + +from . import synthetic +from .conftest import ( + ENV_VAR, + SKIP_LIVE, + SKIP_MISSING, + SKIP_REFUSED, + SKIP_UNSET, + has_git_repo, + is_refused, + path_digest, + record_summary, + resolve_corpus_root, +) + +# --- the content net --------------------------------------------------------- + +# Lowercase, no whitespace, bounded length: the shape of a status token, a key +# name, a grade, a digest, or a path-shape descriptor. Nothing a ledger line or +# a secret looks like survives it. +_STRUCTURAL = re.compile(r"\A[a-z0-9][a-z0-9_.:@#-]{0,63}\Z") + +_NON_STRUCTURAL = "non-structural-token" + + +def _safe_token(value) -> str: + """A source-derived string, reduced to a token or replaced outright. + + Manifest key names and grades come from a note body, which means a foreign + or corrupt note can put anything here. Anything that is not structural is + replaced rather than passed through, so the leg still reports the finding + without carrying the value into the failure message. + """ + if isinstance(value, str) and _STRUCTURAL.match(value): + return value + return _NON_STRUCTURAL + + +def _assert_no_content(obj) -> None: + """Fail if anything non-structural reached an assertion payload. + + The failure message deliberately reports only the TYPE and LENGTH of the + offending value. Echoing it is the exact accident this function exists to + prevent. + """ + stack = [obj] + while stack: + item = stack.pop() + if item is None or isinstance(item, (bool, int, float)): + continue + if isinstance(item, str): + if not _STRUCTURAL.match(item): + raise AssertionError( + f"a non-structural str of length {len(item)} reached an " + "assertion payload; the harness must never put ledger-derived " + "values in an assertion message (its value is withheld here " + "on purpose)" + ) + continue + if isinstance(item, dict): + stack.extend(item.keys()) + stack.extend(item.values()) + continue + if isinstance(item, (list, tuple, set, frozenset)): + stack.extend(item) + continue + raise AssertionError( + f"a {type(item).__name__} reached an assertion payload; only " + "counts, booleans, digests and shape tokens may appear there" + ) + + +def _summarize(leg: str, source: str, payload: dict) -> None: + _assert_no_content(payload) + body = json.dumps(payload, sort_keys=True, separators=(",", ":")) + record_summary(f"{leg} [{source}]: {body}") + + +# --- discovery --------------------------------------------------------------- + + +def ledger_dirs(root) -> list: + """Every directory holding a session.log, in a deterministic order. + + Covers both archive shapes without hardcoding either: nested + ``/.didrun/`` and flat ``/``. + """ + return sorted(p.parent for p in Path(root).rglob("session.log") if p.is_file()) + + +# --- leg 1: chain recompute -------------------------------------------------- + + +@dataclass(frozen=True) +class LogReplay: + """One log's outcome. + + ``label`` is an ordinal, not a path: archive directory names are not + ledger content, but they are not ours to put in a CI log or a transcript + either. The mapping stays recoverable locally without leaking anything — + the ordinal is the position in ``ledger_dirs(root)``. + """ + + label: str + shape: str # "nested" | "flat" + entries: int + status: str # empty | intact | torn-tail | malformed@N | broken@N | unverifiable@N + + +def _read_records(path: Path) -> tuple: + """Parse a log into records. Returns ``(records, torn_tail, malformed_at)``. + + A partial FINAL line is torn-tail — a process killed mid-append, which is + the commonest real damage and is never the same fact as tamper. A partial + line anywhere else is malformed and is reported at its position. + """ + raw = path.read_bytes() + if not raw: + return [], False, None + lines = raw.split(b"\n") + if lines and lines[-1] == b"": + lines.pop() + last = len(lines) - 1 + records: list = [] + torn = False + malformed_at: Optional[int] = None + for i, line in enumerate(lines): + stripped = line.strip() + if not stripped: + continue + try: + records.append(json.loads(stripped)) + except ValueError: + if i == last: + torn = True + elif malformed_at is None: + malformed_at = i + return records, torn, malformed_at + + +def _verify_records(records: list) -> tuple: + """Recompute the chain over parsed records. Returns ``(status, position)``. + + Mirrors Session.verify_chain_detail, but reads instead of constructing a + Session: Session.__init__ mkdirs its root and writes a .gitignore, which + would MODIFY an archive this harness is only ever allowed to read. + + Positions are the harness's own counter, never the stored index, so a + forged index cannot choose the number that gets reported. + """ + prev = GENESIS_HASH + for position, rec in enumerate(records): + try: + index = rec["index"] + prev_hash = rec["prev_hash"] + stored = rec["entry_hash"] + body = rec["event"] + except (KeyError, TypeError): + return "unverifiable", position + if not isinstance(body, dict): + return "unverifiable", position + if index != position or prev_hash != prev: + return "broken", position + if unknown_event_keys_of(body): + return "unverifiable", position + try: + entry = ChainEntry( + index=index, + prev_hash=prev_hash, + event=Event.from_dict(body), + entry_hash=stored, + preimage_version=rec.get("preimage_version", 1), + ) + recomputed = entry.compute_hash() + except (LedgerError, TypeError, ValueError): + return "unverifiable", position + if recomputed != stored: + return "broken", position + prev = stored + return "intact", None + + +def replay_chain(root) -> list: + out = [] + for i, d in enumerate(ledger_dirs(root)): + records, torn, malformed_at = _read_records(d / "session.log") + status, position = _verify_records(records) + if status != "intact": + status = f"{status}@{position}" + elif malformed_at is not None: + status = f"malformed@{malformed_at}" + elif torn: + status = "torn-tail" + elif not records: + status = "empty" + out.append( + LogReplay( + label=f"log#{i}", + shape="nested" if d.name == ".didrun" else "flat", + entries=len(records), + status=status, + ) + ) + return out + + +def _status_histogram(replays: list) -> dict: + hist: dict = {} + for r in replays: + key = r.status.split("@")[0] + hist[key] = hist.get(key, 0) + 1 + return hist + + +def test_leg1_chain_recompute(replay_source): + """Every archived entry_hash must still recompute under this binary. + + This is the leg that would have caught a chain-preimage change before it + shipped, across every archived event, instead of after. + """ + replays = replay_chain(replay_source.root) + payload = { + "logs": len(replays), + "entries": sum(r.entries for r in replays), + "nested": sum(1 for r in replays if r.shape == "nested"), + "flat": sum(1 for r in replays if r.shape == "flat"), + "status": _status_histogram(replays), + } + _summarize("leg1 chain-recompute", replay_source.name, payload) + + statuses = tuple(r.status for r in replays) + _assert_no_content(statuses) + if replay_source.expected is not None: + assert statuses == replay_source.expected.log_statuses + assert tuple(r.entries for r in replays) == replay_source.expected.log_entries + else: + # On a real archive tree the regression is `broken`: a stored hash that + # no longer recomputes. `unverifiable` is a different fact (written by + # something this binary does not fully understand) and is reported, not + # asserted away. + assert [r.label for r in replays if r.status.startswith("broken")] == [] + assert replays != [] + + +# --- leg 2: note round-trip -------------------------------------------------- + +# Keys `to_json` emits that a stored note of this version may legitimately +# lack, because `from_json` reads them with a v1-reproducing default. Hard +# coded per manifest version so a new manifest field is a deliberate one-line +# diff here and never a silent pass. +MANIFEST_ADDITIVE_KEYS = {1: frozenset({"secrets_override"})} + + +def note_violations(raw: bytes, ordinal: int, additive_keys=None) -> list: + """Apply the pack's three-part note round-trip criterion to one note body. + + Naive byte equality (`from_json(body).to_json() == body`) is NOT the + criterion and must not be written: it is false on every real note today + (git notes appends one trailing newline to every body) and it becomes false + a second way as Manifest gains additive fields. Instead: + + (a) newline discipline — strip AT MOST one trailing newline; more than + one is a malformed note and fails closed rather than being rstrip'd; + (b) key-superset semantic equality — every key in the source body must + come back with an equal value, and any extra key must be on this + version's additive allowlist; + (c) serialization idempotence — re-serializing is byte-stable. This is + where canonical_json's stability is actually pinned. + + Returns ``(ordinal, code)`` pairs. Notes are identified by ordinal, and + every code is a structural token: nothing derived from the body's VALUES + reaches the caller. + """ + if additive_keys is None: + additive_keys = MANIFEST_ADDITIVE_KEYS + + body = raw[:-1] if raw.endswith(b"\n") else raw + if body.endswith(b"\n"): + return [(ordinal, "multiple-trailing-newlines")] + try: + source = json.loads(body) + except ValueError: + return [(ordinal, "parse-error")] + if not isinstance(source, dict) or "version" not in source: + return [(ordinal, "not-a-manifest")] + version = source["version"] + if not isinstance(version, int) or isinstance(version, bool): + return [(ordinal, "bad-version-type")] + if version > MANIFEST_VERSION: + return [(ordinal, "version-too-new")] + + try: + first = Manifest.from_json(body).to_json() + except Exception: + return [(ordinal, "parse-error")] + round_tripped = json.loads(first) + + violations = [] + for key in sorted(source): + token = _safe_token(key) + if key not in round_tripped: + violations.append((ordinal, f"missing-key:{token}")) + elif round_tripped[key] != source[key]: + violations.append((ordinal, f"value-mismatch:{token}")) + allowed = additive_keys.get(version, frozenset()) + for key in sorted(set(round_tripped) - set(source)): + if key not in allowed: + violations.append((ordinal, f"undeclared-extra-key:{_safe_token(key)}")) + + # Compared by digest, not by bytes: an `assert a == b` over two manifest + # bodies would print both of them. + second = Manifest.from_json(first).to_json() + if sha256_hex(second) != sha256_hex(first): + violations.append((ordinal, "not-idempotent")) + return violations + + +def note_bodies(root) -> list: + """Note bodies via git metadata commands. No ledger file is opened.""" + listing = subprocess.run( + ["git", "-C", str(root), "notes", f"--ref={NOTES_REF}", "list"], + capture_output=True, + ) + if listing.returncode != 0: + return [] + blobs = sorted( + line.split()[0].decode("ascii") + for line in listing.stdout.splitlines() + if line.strip() + ) + bodies = [] + for blob in blobs: + proc = subprocess.run( + ["git", "-C", str(root), "cat-file", "-p", blob], capture_output=True + ) + if proc.returncode == 0: + bodies.append(proc.stdout) + return bodies + + +def replay_notes(root) -> tuple: + bodies = note_bodies(root) + violations = [] + for ordinal, raw in enumerate(bodies): + violations.extend(note_violations(raw, ordinal)) + return len(bodies), violations + + +def test_leg2_note_round_trip(replay_source): + """Every published note must still parse, round-trip and re-serialize. + + Six later prompts cite this leg as their compat gate, so its definition is + load-bearing: it is the criterion above, not byte equality. + """ + if not has_git_repo(replay_source.root): + pytest.skip("no git repo at the replay root; the note legs need refs/notes/didrun") + count, violations = replay_notes(replay_source.root) + _summarize( + "leg2 note-round-trip", + replay_source.name, + {"notes": count, "violations": len(violations)}, + ) + _assert_no_content(violations) + assert violations == [] + assert count >= 1 + if replay_source.expected is not None: + assert count == replay_source.expected.notes + + +# --- leg 3: layout classifier ------------------------------------------------ + +_FLAT_BLOB = re.compile(r"\A[0-9a-f]{64}\Z") +_FANOUT_DIR = re.compile(r"\A[0-9a-f]{2}\Z") +_FANOUT_FILE = re.compile(r"\A[0-9a-f]{38}\Z") + + +def _shape_token(p: Path) -> str: + kind = "dir" if p.is_dir() else "file" + cls = "hex" if re.fullmatch(r"[0-9a-f]+", p.name) else "nonhex" + return f"{kind}:len{len(p.name)}:{cls}" + + +def classify_objects(root) -> tuple: + """Partition every objects/ entry into exactly one of two shapes. + + 113 of the archived object dirs carry BOTH namespaces — flat SHA-256 blobs + written by the ledger and git fan-out dirs from the sealed object + redirect. A third shape is residue and fails closed; it is reported as a + shape descriptor, never as a name. + """ + counts = {"dirs": 0, "flat-64hex": 0, "git-fanout": 0} + residue: dict = {} + for d in ledger_dirs(root): + objects = d / "objects" + if not objects.is_dir(): + continue + counts["dirs"] += 1 + for entry in sorted(objects.iterdir()): + if entry.is_file() and _FLAT_BLOB.match(entry.name): + counts["flat-64hex"] += 1 + elif entry.is_dir() and _FANOUT_DIR.match(entry.name): + for child in sorted(entry.iterdir()): + if child.is_file() and _FANOUT_FILE.match(child.name): + counts["git-fanout"] += 1 + else: + token = "fanout-child:" + _shape_token(child) + residue[token] = residue.get(token, 0) + 1 + else: + token = _shape_token(entry) + residue[token] = residue.get(token, 0) + 1 + return counts, residue + + +def test_leg3_layout_classifier(replay_source): + counts, residue = classify_objects(replay_source.root) + payload = dict(counts) + payload["residue"] = sum(residue.values()) + _summarize("leg3 layout-classifier", replay_source.name, payload) + _assert_no_content(residue) + assert residue == {} + if replay_source.expected is not None: + assert counts == replay_source.expected.objects + + +# --- leg 4: claim-shape invariants ------------------------------------------- + + +def claim_shape_from_bodies(bodies: list) -> dict: + """The shape facts nobody measured until they broke three conclusions.""" + notes = 0 + claims = 0 + multi_index = 0 + pathspecs: dict = {} + gap_ge_1 = 0 + gap_ge_5 = 0 + grades: dict = {} + for raw in bodies: + body = raw[:-1] if raw.endswith(b"\n") else raw + try: + parsed = json.loads(body) + except ValueError: + continue + if not isinstance(parsed, dict): + continue + notes += 1 + for entry in parsed.get("claims", []) or []: + if not isinstance(entry, dict): + continue + claims += 1 + claim = entry.get("claim") or {} + if len(claim.get("event_indices") or []) > 1: + multi_index += 1 + n_specs = len(claim.get("pathspecs") or []) + pathspecs[n_specs] = pathspecs.get(n_specs, 0) + 1 + supporting = entry.get("supporting_event_index") + declared_at = claim.get("declared_at_index", -1) + if ( + isinstance(supporting, int) + and isinstance(declared_at, int) + and declared_at >= 0 + ): + gap = declared_at - supporting + if gap >= 1: + gap_ge_1 += 1 + if gap >= 5: + gap_ge_5 += 1 + grade = _safe_token(entry.get("grade")) + grades[grade] = grades.get(grade, 0) + 1 + return { + "notes": notes, + "claims": claims, + "multi-index": multi_index, + "max-pathspecs": max(pathspecs) if pathspecs else 0, + "pathspecs": pathspecs, + "gap-ge-1": gap_ge_1, + "gap-ge-5": gap_ge_5, + "grades": grades, + } + + +def claim_shape_stats(root) -> dict: + return claim_shape_from_bodies(note_bodies(root)) + + +def test_leg4_claim_shape_invariants(replay_source): + """The leg the red-team says earns its keep. + + A grade histogram is a snapshot and is REPORTED, not asserted, against a + real corpus — asserting it would have passed on every defect in the audit. + The shape invariants are the regression: had they existed, two blocker + findings would have been caught here instead of by a reviewer. The + declared-vs-supporting gap is a signal, so it is reported too. + """ + if not has_git_repo(replay_source.root): + pytest.skip("no git repo at the replay root; the note legs need refs/notes/didrun") + stats = claim_shape_stats(replay_source.root) + _summarize("leg4 claim-shape", replay_source.name, stats) + _assert_no_content(stats) + assert stats["multi-index"] == 0 + assert stats["max-pathspecs"] <= 1 + if replay_source.expected is not None: + # Synthetic only: here the histogram is testing the instrument against + # a fixture whose every value is declared, which is not the same thing + # as pinning a corpus snapshot. + assert stats == replay_source.expected.claim_shape + + +# --- meta: the harness must skip cleanly, and must be able to fail ----------- + + +def test_corpus_legs_skip_when_env_unset(tmp_path): + """CI is green by construction: no env var, no corpus leg, no error.""" + root, reason = resolve_corpus_root({}) + assert root is None + assert reason == SKIP_UNSET + assert ENV_VAR in reason + + root, reason = resolve_corpus_root({ENV_VAR: str(tmp_path / "nope")}) + assert root is None + assert reason == SKIP_MISSING + + root, reason = resolve_corpus_root({ENV_VAR: " "}) + assert root is None + assert reason == SKIP_UNSET + + live = tmp_path / "live" + (live / ".didrun").mkdir(parents=True) + (live / ".git").mkdir() + root, reason = resolve_corpus_root({ENV_VAR: str(live)}) + assert root is None + assert reason == SKIP_LIVE + + ok = tmp_path / "copy" + (ok / ".didrun-history").mkdir(parents=True) + root, reason = resolve_corpus_root({ENV_VAR: str(ok)}) + assert root == ok + assert reason is None + + +def test_refused_trees_are_refused_by_digest_including_subpaths(tmp_path): + """The named-tree refusal, and the proof it also covers paths inside it. + + The digests themselves are frozen in conftest; this exercises the + mechanism against a synthetic path so no private path appears here. + """ + forbidden = tmp_path / "forbidden-tree" + (forbidden / "sub" / "deeper").mkdir(parents=True) + digests = frozenset({path_digest(forbidden)}) + assert is_refused(forbidden, digests) + assert is_refused(forbidden / "sub" / "deeper", digests) + assert is_refused(str(forbidden) + "/", digests) + assert not is_refused(tmp_path / "other", digests) + + # The wiring: a path on the frozen list takes the refusal branch, and it is + # checked BEFORE the directory-exists test, so a refused tree is refused + # whether or not it is currently mounted. + root, reason = resolve_corpus_root({ENV_VAR: str(forbidden)}, digests=digests) + assert root is None + assert reason == SKIP_REFUSED + root, reason = resolve_corpus_root( + {ENV_VAR: str(tmp_path / "absent")}, digests=frozenset({path_digest(tmp_path)}) + ) + assert reason == SKIP_REFUSED + + from .conftest import _REFUSED_PATH_DIGESTS + + assert len(_REFUSED_PATH_DIGESTS) >= 1 + assert all(re.fullmatch(r"[0-9a-f]{64}", d) for d in _REFUSED_PATH_DIGESTS) + + +def test_synthetic_known_bad_logs_are_detected(synthetic_corpus, tmp_path): + """The harness can actually fail: torn, unknown, broken are all detected.""" + replays = replay_chain(synthetic_corpus.root) + by_status = {r.status for r in replays} + _assert_no_content(by_status) + assert "torn-tail" in by_status + assert "unverifiable@0" in by_status + assert "broken@1" in by_status + assert "empty" in by_status + assert sum(1 for r in replays if r.status == "intact") == 2 + + # Independent of the generator's declarations: tamper a good chain here and + # confirm the recompute catches it. + records = synthetic._chain(3) + synthetic._write_log(tmp_path / "good" / ".didrun" / "session.log", records) + before = replay_chain(tmp_path / "good")[0].status + assert before == "intact" + + stored = records[2]["entry_hash"] + records[2]["entry_hash"] = ("0" if stored[0] != "0" else "1") + stored[1:] + synthetic._write_log(tmp_path / "tampered" / ".didrun" / "session.log", records) + after = replay_chain(tmp_path / "tampered")[0].status + assert after == "broken@2" + + +def test_synthetic_known_bad_notes_are_detected(known_bad_corpus): + """Each part of the round-trip criterion fails on the thing it targets. + + Every result is bound to a local before it is asserted on. pytest's + assertion rewriting explains a failing comparison by printing the CALL and + its arguments, so ``assert note_violations(, 0) == []`` would + put a note body in the failure message. Same rule as everywhere else here, + and the reason the grep gate looks for exactly that shape. + """ + synthetic_note = synthetic.multi_index_note_body() + + # (a) newline discipline: one trailing newline is git's and is stripped; + # two is a malformed note and must fail closed rather than be rstrip'd. + clean = note_violations(synthetic_note, 0) + one_newline = note_violations(synthetic_note + b"\n", 0) + two_newlines = note_violations(synthetic_note + b"\n\n", 0) + assert clean == [] + assert one_newline == [] + assert two_newlines == [(0, "multiple-trailing-newlines")] + + # (b) the additive-key allowlist: with an empty allowlist, the one key a v1 + # note may legitimately lack becomes a reported violation. + stripped = json.loads(synthetic_note) + del stripped["secrets_override"] + trimmed = json.dumps(stripped, sort_keys=True, separators=(",", ":")).encode("ascii") + allowed = note_violations(trimmed, 3) + not_allowed = note_violations(trimmed, 3, additive_keys={1: frozenset()}) + assert allowed == [] + assert not_allowed == [(3, "undeclared-extra-key:secrets_override")] + + unparseable = note_violations(b"not json at all", 1) + not_manifest = note_violations(b'{"nope":1}', 2) + bad_version = note_violations(b'{"version":"1"}', 2) + assert unparseable == [(1, "parse-error")] + assert not_manifest == [(2, "not-a-manifest")] + assert bad_version == [(2, "bad-version-type")] + + # End to end through git: a note whose manifest version this binary must + # refuse rather than coerce. + count, violations = replay_notes(known_bad_corpus) + assert count == 1 + assert violations == [(0, "version-too-new")] + + +def test_synthetic_known_bad_objects_residue_is_detected(known_bad_corpus): + """A third path shape in objects/ fails closed and is named by shape only.""" + counts, residue = classify_objects(known_bad_corpus) + _assert_no_content(residue) + assert counts["flat-64hex"] == 1 + assert counts["git-fanout"] == 1 + assert residue == {"dir:len4:nonhex": 1, "file:len9:nonhex": 1} + + +def test_synthetic_known_bad_claims_are_detected(): + """The invariant leg catches the claim shapes the corpus says do not exist.""" + stats = claim_shape_from_bodies([synthetic.multi_index_note_body()]) + _assert_no_content(stats) + assert stats["multi-index"] == 1 + assert stats["max-pathspecs"] == 2 + assert stats["grades"] == {"tree-exact": 1, "scope-exact": 1} + + +def test_assert_no_content_refuses_content_and_never_echoes_it(): + """The net holds, and its own failure message does not leak the value.""" + _assert_no_content( + { + "logs": 7, + "status": {"intact": 2, "broken@1": 1}, + "ok": True, + "none": None, + "digest": "a" * 64, + "shape": "file:len9:nonhex", + } + ) + + secret = "AKIAIOSFODNN7EXAMPLE-with-a-space and a tail" + with pytest.raises(AssertionError) as exc: + _assert_no_content({"argv": [secret]}) + assert secret not in str(exc.value) + assert "AKIA" not in str(exc.value) + assert str(len(secret)) in str(exc.value) + + with pytest.raises(AssertionError): + _assert_no_content([Path("/w/repo")]) + + assert _safe_token("tree-exact") == "tree-exact" + assert _safe_token("Bearer ABC123") == _NON_STRUCTURAL + assert _safe_token(None) == _NON_STRUCTURAL diff --git a/tests/test_capture_claims_manifest.py b/tests/test_capture_claims_manifest.py index 3ab5f89..9e13add 100644 --- a/tests/test_capture_claims_manifest.py +++ b/tests/test_capture_claims_manifest.py @@ -2,12 +2,16 @@ from __future__ import annotations +import json +import re import subprocess import sys from pathlib import Path import pytest +from didrun import capture +from didrun import cli from didrun.ledger import Session from didrun.capture import run_wrapped from didrun.claims import Claim, ClaimError, grade @@ -194,3 +198,140 @@ def test_strict_nonzero_on_stale(repo: Path): M.seal(s, repo) report = M.verify(s, repo) assert not report.all_verified # → CLI --strict returns nonzero + + +# --- P0.3: the format contract ------------------------------------------------ + +def _note_body(version: int, tree: str, commit: str = "0" * 40) -> bytes: + """A structurally complete note body at an arbitrary manifest version.""" + return json.dumps( + { + "version": version, + "commit": commit, + "tree": tree, + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + "secrets_override": False, + } + ).encode("ascii") + + +def _attach_raw_note(repo: Path, commit: str, body: bytes) -> None: + """Publish arbitrary bytes as a note, bypassing Manifest entirely.""" + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-F", "-", commit], + cwd=str(repo), + input=body, + capture_output=True, + check=True, + ) + + +def _note_of(repo: Path, commit: str): + """The note attached to ``commit``, or None — read with raw git, not didrun.""" + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +def _commit_file(repo: Path, name: str) -> str: + """A commit with a tree distinct from every other commit in the fixture.""" + (repo / name).write_text(f"content of {name}\n") + _git(repo, "add", name) + _git(repo, "commit", "-qm", f"add {name}") + return gp.head_commit(repo) + + +def test_from_json_refuses_a_newer_manifest_version(repo: Path): + """Refuse, never coerce: a v99 note may have redefined a field this binary + reads, so grading it under v1 semantics would be a confident lie.""" + with pytest.raises(M.ManifestError) as exc: + M.Manifest.from_json(_note_body(99, "a" * 40)) + msg = str(exc.value) + assert "version 99" in msg # the version it met + assert f"max {M.MANIFEST_VERSION}" in msg # the version it understands + assert "upgrade didrun" in msg + + +def test_from_json_accepts_the_current_manifest_version(): + """The gate is `>`, not `!=` — every note already published is version 1.""" + m = M.Manifest.from_json(_note_body(M.MANIFEST_VERSION, "b" * 40)) + assert m.version == M.MANIFEST_VERSION + assert m.tree == "b" * 40 + + +def test_fallback_skips_an_unparseable_note_and_counts_it(repo: Path): + """A foreign or corrupt note under the same ref must not hide every good + one — but the skip is counted, never silently reported as "no manifest".""" + stranger = gp.head_commit(repo) + target = _commit_file(repo, "target.py") + _attach_raw_note(repo, stranger, b"not json") + assert _note_of(repo, target) is None # the commit-exact path finds nothing + + manifest, resolved_by, skipped = M._resolve_manifest( + repo, target, gp.commit_tree(repo, target) + ) + assert manifest is None and resolved_by == "none" + assert skipped == 1 + assert M.verify(_session(repo), repo, commitish=target).notes_skipped == 1 + + +def test_fallback_refuses_a_future_version_it_scans_past(repo: Path): + """Fail closed on the version even in the fallback loop, and do it + order-independently: NEITHER note's tree matches the commit under test, so + the loop cannot exit early on a match and must reach the v99 note whichever + order `git notes list` yields.""" + garbage_commit = gp.head_commit(repo) + future_commit = _commit_file(repo, "future.py") + target = _commit_file(repo, "target.py") + + target_tree = gp.commit_tree(repo, target) + future_tree = gp.commit_tree(repo, future_commit) + assert future_tree != target_tree # the fixture's whole point + + _attach_raw_note(repo, garbage_commit, b"not json") + _attach_raw_note(repo, future_commit, _note_body(99, future_tree, future_commit)) + assert _note_of(repo, target) is None # commit-exact found nothing first + + with pytest.raises(M.ManifestError) as exc: + M._resolve_manifest(repo, target, target_tree) + assert "version 99" in str(exc.value) + + +def test_cli_refuses_a_future_note_as_a_message_not_a_traceback(repo: Path, capsys): + """Fail-closed has to reach the operator as a refusal. A traceback is a + crash report, and a CI log full of stack frames reads as "didrun is broken" + rather than "this note was written by a newer didrun". Exit 2, not + --strict's 1: unreadable evidence is a different fact from bad evidence.""" + head = gp.head_commit(repo) + _attach_raw_note(repo, head, _note_body(99, gp.commit_tree(repo, head), head)) + rc = cli.main(["--repo", str(repo), "verify", "--strict"]) + err = capsys.readouterr().err + assert rc == 2 + assert "version 99" in err and "upgrade didrun" in err + assert "Traceback" not in err + + +def test_env_fingerprint_carries_its_version_in_band(): + assert re.fullmatch(r"v1:[0-9a-f]{16}", capture.env_fingerprint({"PATH": "/bin"})) + assert capture.fingerprint_version("v1:0123456789abcdef") == 1 + # A bare v0.1 digest: incomparable with a versioned one, not drifted. + assert capture.fingerprint_version("0123456789abcdef") is None + + +def test_env_fingerprint_is_deterministic_and_env_sensitive(): + """Determinism is a trust-path invariant; sensitivity is the point of the + field. No hard-coded digest — the key set changes in a later unit.""" + env = {"PATH": "/usr/bin", "SHELL": "/bin/zsh", "LANG": "C"} + assert capture.env_fingerprint(env) == capture.env_fingerprint(dict(env)) + assert capture.env_fingerprint({**env, "PATH": "/usr/local/bin"}) != capture.env_fingerprint(env) + + +def test_recorded_events_carry_the_versioned_fingerprint(repo: Path): + """The prefix reaches the ledger, not just the helper's return value.""" + s = _session(repo) + ev = run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + assert capture.fingerprint_version(ev.env_fingerprint) == 1 diff --git a/tests/test_gitplumbing.py b/tests/test_gitplumbing.py index d4d2348..72818f8 100644 --- a/tests/test_gitplumbing.py +++ b/tests/test_gitplumbing.py @@ -2,6 +2,7 @@ from __future__ import annotations +import inspect import re from pathlib import Path @@ -100,6 +101,57 @@ def test_ledger_excluded_from_digest_when_gitignored(repo: Path): assert before is not None and before == after +def _archive(repo: Path) -> None: + """A rotated ledger sitting under the archive root, in its usual shape.""" + d = repo / gp.ARCHIVE_DIRNAME / "unit-01" / ".didrun" + d.mkdir(parents=True) + (d / "session.log").write_text("rotated\n") + + +def test_archive_root_is_in_the_default_exclusion(): + """The exclusion is a default, not an opt-in — a caller passing nothing + still gets the archive root kept out.""" + default = inspect.signature(gp.tree_digest).parameters["exclude"].default + assert default == (gp.LEDGER_DIRNAME, gp.ARCHIVE_DIRNAME) + + +def test_archive_gitignored_digest_is_unchanged_by_the_new_exclusion(repo: Path): + """The compat criterion: where the archive root IS gitignored, v0.2 digests + a tree to the same id v0.1 did, because `add -A` already skipped it.""" + (repo / ".gitignore").write_text(".didrun/\n.didrun-history/\n") + _archive(repo) + v01 = gp.tree_digest(repo, exclude=(gp.LEDGER_DIRNAME,)) + v02 = gp.tree_digest(repo) + assert v01 is not None and v01 == v02 + + +def test_archive_not_gitignored_is_a_digest_semantics_change(repo: Path): + """Where the archive root is NOT gitignored, v0.2 and v0.1 disagree — and + v0.2 is the one that excludes it. Pinned, not hidden: this is the release + note. Without it, archiving a ledger moves the digest of the tree that + just archived it and stales the unit that did the archiving.""" + empty = gp.tree_digest(repo) # no archive root present yet + _archive(repo) + v01 = gp.tree_digest(repo, exclude=(gp.LEDGER_DIRNAME,)) + v02 = gp.tree_digest(repo) + assert v01 is not None and v02 is not None + assert v01 != v02 # the semantics change, stated + assert v02 == empty # and v0.2 is the side that ignores the archive + + +def test_ledger_dir_still_excluded_in_both_configs_with_the_archive_present(repo: Path): + """Adding a second exclusion must not weaken the first: `.didrun/` stays + out regardless of gitignore state, which is the guarantee grading rests on.""" + _archive(repo) + for ignore in ("", ".didrun/\n.didrun-history/\n"): + (repo / ".gitignore").write_text(ignore) + before = gp.tree_digest(repo) + (repo / ".didrun" / "objects").mkdir(parents=True, exist_ok=True) + (repo / ".didrun" / "objects" / "junk").write_text("noise\n") + after = gp.tree_digest(repo) + assert before is not None and before == after + + def test_tree_delta_reports_exact_path(repo: Path, tmp_path: Path): ledger_obj = tmp_path / "obj" a = gp.tree_digest(repo, ledger_obj) diff --git a/tests/test_ledger.py b/tests/test_ledger.py index e50ce04..a9f09bc 100644 --- a/tests/test_ledger.py +++ b/tests/test_ledger.py @@ -2,6 +2,7 @@ from __future__ import annotations +import json import subprocess import sys import textwrap @@ -10,12 +11,14 @@ import pytest from didrun.ledger import ( + GENESIS_HASH, Event, Session, BlobStore, LedgerError, canonical_json, sha256_hex, + unknown_event_keys_of, ) @@ -124,3 +127,145 @@ def test_self_stability_precondition(): assert ev.self_stable() ev2 = _wrapper_event(tree_before="aaa", tree_after="bbb") assert not ev2.self_stable() + + +# --- P0.2 — the frozen v1 preimage, unknown keys, unknown versions ----------- + + +def _records(session: Session) -> list[dict]: + return [json.loads(l) for l in session.log_path.read_text().splitlines() if l.strip()] + + +def _rewrite(session: Session, records: list[dict]) -> None: + session.log_path.write_bytes(b"".join(canonical_json(r) + b"\n" for r in records)) + + +def test_v1_record_on_disk_carries_no_preimage_version_key(tmp_path: Path): + """A v1 record must stay byte-shaped exactly as every previous release + wrote it, so an older binary keeps reading a ledger this one produced.""" + s = Session(tmp_path / ".didrun") + s.append(_wrapper_event()) + rec = _records(s)[0] + assert set(rec) == {"index", "prev_hash", "entry_hash", "event"} + + +def test_archived_shape_record_round_trips(tmp_path: Path): + """A record with no preimage_version key — i.e. every archived record — + reads back, recomputes, and grades intact. + + The expected hash is spelled out here with the pre-P0.2 preimage rather + than borrowed from compute_hash(), so this fails if the projection ever + stops being byte-identical to the whole event body at version 1. + """ + s = Session(tmp_path / ".didrun") + body = _wrapper_event(argv=("cmd", "0")).to_dict() + stored = sha256_hex( + GENESIS_HASH.encode("ascii") + + canonical_json({"index": 0, "prev": GENESIS_HASH, "event": body}) + ) + _rewrite( + s, + [{"index": 0, "prev_hash": GENESIS_HASH, "entry_hash": stored, "event": body}], + ) + + entries = list(s.entries()) + assert len(entries) == 1 + assert entries[0].preimage_version == 1 + assert entries[0].unknown_event_keys == () + assert entries[0].event.argv == ("cmd", "0") # argv tuple-ization survives + assert entries[0].compute_hash() == stored + assert s.verify_chain_detail().status == "intact" + assert s.verify_chain() == (True, None) + + +def test_unknown_event_key_is_readable_but_unverifiable(tmp_path: Path): + """A ledger written by a newer binary must stay readable, and must not be + reported as tamper — but must never be reported as intact either.""" + s = Session(tmp_path / ".didrun") + s.append(_wrapper_event()) + recs = _records(s) + recs[0]["event"]["umask"] = "0022" + _rewrite(s, recs) + + # Readable, not fatal: no TypeError out of cls(**d). + ev = Event.from_dict(recs[0]["event"]) + assert ev.argv == ("echo", "hi") + assert unknown_event_keys_of(recs[0]["event"]) == ("umask",) + + entries = list(s.entries()) + assert entries[0].unknown_event_keys == ("umask",) + # This is exactly why unverifiable has to exist: the projection cannot see + # the extra key, so the recomputed hash still matches and a hash check + # alone would call this chain intact. + assert entries[0].compute_hash() == entries[0].entry_hash + + verdict = s.verify_chain_detail() + assert verdict.status == "unverifiable" + assert verdict.unverifiable_index == 0 + assert verdict.first_broken_index is None + assert "umask" in verdict.reason + + ok, idx = s.verify_chain() + assert ok is False + assert idx == 0 + + +def test_unknown_preimage_version_fails_closed(tmp_path: Path): + s = Session(tmp_path / ".didrun") + for i in range(3): + s.append(_wrapper_event(argv=("cmd", str(i)))) + recs = _records(s) + recs[1]["preimage_version"] = 2 + _rewrite(s, recs) + + entries = list(s.entries()) + assert entries[1].preimage_version == 2 + with pytest.raises(LedgerError): + entries[1].compute_hash() + + verdict = s.verify_chain_detail() + assert verdict.status == "unverifiable" + assert verdict.unverifiable_index == 1 + assert verdict.first_broken_index is None + assert "2" in verdict.reason + + +def test_verify_chain_reports_rather_than_raises_on_unknown_version(tmp_path: Path): + """compute_hash now raises on an unknown version. A reader that crashes is + worse than one that reports, so the boolean wrapper must still return.""" + s = Session(tmp_path / ".didrun") + for i in range(3): + s.append(_wrapper_event(argv=("cmd", str(i)))) + recs = _records(s) + recs[1]["preimage_version"] = 2 + _rewrite(s, recs) + + ok, idx = s.verify_chain() # must not raise + assert ok is False + assert idx == 1 + + +def test_mutated_event_body_is_broken_not_unverifiable(tmp_path: Path): + """Tamper inside a known field is still tamper — unverifiable must not + become a laundering route for a mutated body.""" + s = Session(tmp_path / ".didrun") + for i in range(5): + s.append(_wrapper_event(argv=("cmd", str(i)))) + recs = _records(s) + recs[1]["event"]["argv"] = ["cmd", "tampered"] + _rewrite(s, recs) + + verdict = s.verify_chain_detail() + assert verdict.status == "broken" + assert verdict.first_broken_index == 1 + assert verdict.unverifiable_index is None + + ok, idx = s.verify_chain() + assert ok is False + assert idx == 1 + + +def test_empty_chain_verdict(tmp_path: Path): + s = Session(tmp_path / ".didrun") + assert s.verify_chain_detail().status == "empty" + assert s.verify_chain() == (True, None) diff --git a/tests/test_preimage_golden.py b/tests/test_preimage_golden.py new file mode 100644 index 0000000..4c90bdd --- /dev/null +++ b/tests/test_preimage_golden.py @@ -0,0 +1,125 @@ +"""P0.2 — the v1 chain preimage is frozen. Three assertions guard it. + +Every entry_hash ever written by didrun was computed over the v1 preimage. If +the preimage changes, every archived ledger reports tamper at index 0. The +literals below pin it. + +DO NOT REGENERATE THE LITERALS IN THIS FILE. If an assertion here fails, the +v1 chain preimage changed and every previously recorded entry_hash stopped +reproducing. That is the defect this file exists to catch — fix the code, or +define a *new* preimage version and leave version 1 alone. Recomputing a +literal to make the test pass destroys the only evidence that the freeze held. + +Three assertions, because one is not enough: + + (i) the golden entry_hash literal — catches any change to the hashed bytes; + (ii) _PREIMAGE_FIELDS[1] equals the frozen name list below — catches a field + added to the preimage list; + (iii) the frozen name list equals Event's dataclass fields — catches a field + added to Event. + +(ii) and (iii) together are what close the hole that set-equality alone leaves: +adding a field to *both* Event and _PREIMAGE_FIELDS[1] satisfies a check that +only compares those two to each other, while breaking every archived hash. + +Assertion (i) is deliberately written against the pre-P0.2 API surface +(``ChainEntry(index=..., prev_hash=..., event=...)`` and ``compute_hash()``), so +this file runs unchanged on the code as it stood *before* the preimage freeze +landed. That is what makes the golden literal falsifiable: it was generated +from pre-change code and must reproduce on post-change code. +""" + +from __future__ import annotations + +import dataclasses + +import pytest + +from didrun import ledger +from didrun.ledger import ChainEntry, Event, Session + +# --- the golden preimage ----------------------------------------------------- +# Generated from the code as it stood before the P0.2 edit. Synthetic values +# only: no real path, host, or digest appears here. + +GOLDEN_EVENT = { + "argv": ["pytest", "-q", "tests/"], + "cwd": "/w/repo", + "env_fingerprint": "0123456789abcdef", + "observed_via": "wrapper", + "coverage": "complete", + "exit_code": 0, + "started_at": 1000000.5, + "ended_at": 1000002.25, + "stdout_blob": "1" * 64, + "stderr_blob": "2" * 64, + "transcript_blob": None, + "tree_before": "3" * 40, + "tree_after": "3" * 40, + "submodule_dirty": False, +} +GOLDEN_INDEX = 7 +GOLDEN_PREV_HASH = "4" * 64 +GOLDEN_ENTRY_HASH = "99715b0cb8a7d3bef413a9d5c3692518b75ec5a5fddcf9bc7a7f6da333ee7d3c" + +# The v1 preimage field names, frozen as a literal. Not derived from anything. +V1_PREIMAGE_FIELD_NAMES = [ + "argv", + "coverage", + "cwd", + "ended_at", + "env_fingerprint", + "exit_code", + "observed_via", + "started_at", + "stderr_blob", + "stdout_blob", + "submodule_dirty", + "transcript_blob", + "tree_after", + "tree_before", +] + +# _PREIMAGE_FIELDS does not exist before P0.2. Assertion (i) is the evidence +# that crosses that boundary; (ii) can only run once the constant is there. +_HAS_PREIMAGE_FIELDS = hasattr(ledger, "_PREIMAGE_FIELDS") + + +def test_v1_chain_preimage_golden_hash(): + """(i) The hashed bytes of a v1 entry have not moved.""" + entry = ChainEntry( + index=GOLDEN_INDEX, + prev_hash=GOLDEN_PREV_HASH, + event=Event.from_dict(dict(GOLDEN_EVENT)), + ) + assert entry.compute_hash() == GOLDEN_ENTRY_HASH + + +@pytest.mark.skipif( + not _HAS_PREIMAGE_FIELDS, + reason="pre-P0.2 code has no _PREIMAGE_FIELDS; assertion (i) is the cross-boundary evidence", +) +def test_v1_preimage_field_list_matches_frozen_literal(): + """(ii) The declared v1 field list is exactly the frozen 14 names.""" + assert sorted(ledger._PREIMAGE_FIELDS[1]) == V1_PREIMAGE_FIELD_NAMES + + +def test_event_dataclass_matches_frozen_literal(): + """(iii) Event's field set is exactly the frozen 14 names. + + Runs on pre- and post-P0.2 code alike. Adding a field to Event fails here + even if the same field is also added to _PREIMAGE_FIELDS[1]. + """ + assert sorted(f.name for f in dataclasses.fields(Event)) == V1_PREIMAGE_FIELD_NAMES + + +def test_preimage_api_is_all_or_nothing(): + """The skip above must not be re-armable by deleting the constant. + + If this binary carries the P0.2 chain-verdict API, it must also carry both + preimage constants — so assertion (ii) cannot be silenced by removing + _PREIMAGE_FIELDS while keeping the rest of the freeze. + """ + if hasattr(Session, "verify_chain_detail"): + assert hasattr(ledger, "PREIMAGE_VERSION") + assert _HAS_PREIMAGE_FIELDS diff --git a/tests/test_redact_render.py b/tests/test_redact_render.py index d288d67..ee042db 100644 --- a/tests/test_redact_render.py +++ b/tests/test_redact_render.py @@ -52,6 +52,7 @@ class R: results: list = None coverage: dict = None secrets_override: bool = False + notes_skipped: int = 0 @property def worst_status(self): @@ -92,6 +93,23 @@ def test_cli_no_color_has_text_tokens(monkeypatch): assert "VERIFIED" not in out +def test_unparseable_notes_are_reported_on_both_verdict_paths(monkeypatch): + """"No manifest" and "there were notes and none could be read" are + different facts, and the second must reach the operator on the no-claims + path too — that is the path a repo with only foreign notes lands on.""" + monkeypatch.setenv("NO_COLOR", "1") + quiet = render.render_verdict(_report([])) + assert "skipped" not in quiet # never invented when the count is zero + + empty = _report([], worst="empty") + empty.notes_skipped = 2 + assert "2 notes skipped (unparseable)" in render.render_verdict(empty) + + graded = _report([_gr("tree-exact", "t")]) + graded.notes_skipped = 1 + assert "1 note skipped (unparseable)" in render.render_verdict(graded) + + def test_no_surface_says_verified(): """Honesty invariant: no human surface may label a grade 'VERIFIED'/'PROVEN'.""" results = [_gr("tree-exact", "t"), _gr("failed", "f")] diff --git a/tests/test_seal_publication.py b/tests/test_seal_publication.py new file mode 100644 index 0000000..ec4db17 --- /dev/null +++ b/tests/test_seal_publication.py @@ -0,0 +1,293 @@ +"""P0.1 — seal publishes fail-closed, on stdin, atomically with the watermark. + +The defect these pin: `_attach_note` discarded git's exit status, so a denied +write to refs/notes/didrun produced a silent half-seal — no note, watermark +advanced, exit 0. The pair (note published, watermark advanced) is now atomic +in both directions, and the rollback RESTORES a note it overwrote rather than +removing it. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + +# Captured at import so a selective patch can delegate without recursing. +_REAL_RUN = subprocess.run + +# Keys `to_json()` emits that a note of this manifest version never carried. +# Additive fields land here as a DELIBERATE one-line diff, never silently. +_ADDITIVE_KEYS_BY_MANIFEST_VERSION = {1: frozenset()} + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _seeded(repo: Path, labels=("t",)) -> Session: + """One recorded event plus one claim per label, ready to seal.""" + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + for label in labels: + M.declare_claim( + s, Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0) + ) + return s + + +def _seal_lines(repo: Path) -> int: + path = repo / ".didrun" / "seals.jsonl" + if not path.exists(): + return 0 + return len([ln for ln in path.read_text(encoding="ascii").splitlines() if ln.strip()]) + + +def _note_bytes(repo: Path, commit: str = "HEAD"): + """The raw note body, or None when the commit carries no note.""" + proc = _REAL_RUN( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +def _is_notes_add(argv) -> bool: + argv = list(argv) + return argv[:2] == ["git", "notes"] and "add" in argv + + +def _patch_notes_add(monkeypatch, returncode: int, seen: list): + """Fail (or merely observe) ONLY `git notes … add`. + + A blanket patch of `manifest.subprocess.run` is wrong: the same symbol + serves `_rev` and `_resolve_manifest`, so a blanket failure aborts the seal + before it ever reaches `_attach_note` and the test passes for the wrong + reason. A failing `git` shim on PATH has the same defect. + """ + + def fake(argv, *args, **kwargs): + if _is_notes_add(argv): + seen.append((list(argv), kwargs.get("input"))) + if returncode != 0: + return subprocess.CompletedProcess( + list(argv), returncode, b"", b"fatal: could not write note object" + ) + return _REAL_RUN(argv, *args, **kwargs) + + monkeypatch.setattr(M.subprocess, "run", fake) + + +def _round_trip(body: bytes) -> M.Manifest: + """The note round-trip criterion: newline discipline, key-superset semantic + equality, serialization idempotence. Deliberately NOT byte equality — + `git notes` appends a trailing newline to every body, so that assertion is + false on every real note. + """ + # (a) strip at most one trailing newline, and prove at most one was there. + assert body.endswith(b"\n"), "note body does not end in a newline" + stripped = body[:-1] + assert not stripped.endswith(b"\n"), "note body carries more than one trailing newline" + + manifest = M.Manifest.from_json(stripped) + source = json.loads(stripped) + emitted = json.loads(manifest.to_json()) + + # (b) every source key survives with an equal value; extras are allowlisted. + for key, value in source.items(): + assert key in emitted, f"round-trip dropped manifest key {key!r}" + assert emitted[key] == value, f"round-trip changed manifest key {key!r}" + extra = frozenset(emitted) - frozenset(source) + assert extra == _ADDITIVE_KEYS_BY_MANIFEST_VERSION[source["version"]], ( + f"unexpected extra manifest keys {sorted(extra)} — a field was added " + "without updating the per-version allowlist" + ) + + # (c) serialization is idempotent; this is where canonical_json is pinned. + once = manifest.to_json() + assert M.Manifest.from_json(once).to_json() == once + return manifest + + +def test_note_publication_failure_is_fatal(repo: Path, monkeypatch): + """git refuses the note -> seal raises, nothing is recorded, no note exists.""" + s = _seeded(repo) + seen: list = [] + _patch_notes_add(monkeypatch, 1, seen) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + + assert "could not publish note" in str(exc.value) + assert "fatal: could not write note object" in str(exc.value) + assert seen, "the selective patch never saw a `git notes add` — wrong seam" + assert _seal_lines(repo) == 0 + assert _note_bytes(repo) is None + + +def test_watermark_failure_rolls_back_and_leaves_no_note(repo: Path, monkeypatch): + """No prior note: a failed watermark write must leave the commit noteless.""" + s = _seeded(repo) + + def boom(*args, **kwargs): + raise OSError("read-only file system: seals.jsonl") + + monkeypatch.setattr(M, "_record_seal", boom) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + + message = str(exc.value) + assert "no watermark was recorded" in message + assert "rolled back" in message + assert "noteless" in message + assert "read-only file system" in message + assert _seal_lines(repo) == 0 + assert _note_bytes(repo) is None + + +def test_watermark_failure_restores_a_prior_note(repo: Path, monkeypatch): + """The rollback may never destroy evidence the commit already carried. + + This is the leg a `git notes remove` rollback fails: `_attach_note` forces, + so the seal overwrote a real prior note. + """ + s = _seeded(repo) + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-m", "prior evidence body"], + cwd=str(repo), + capture_output=True, + check=True, + ) + prior = _note_bytes(repo) + assert prior == b"prior evidence body\n" + + def boom(*args, **kwargs): + raise OSError("read-only file system: seals.jsonl") + + monkeypatch.setattr(M, "_record_seal", boom) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + + message = str(exc.value) + assert "no watermark was recorded" in message + assert "prior note was restored" in message + assert _seal_lines(repo) == 0 + assert _note_bytes(repo) == prior # byte-restored, not removed + + +def test_body_travels_on_stdin_not_argv(repo: Path, monkeypatch): + """The real criterion: `-F -` with the body on `input=`, and no `-m`.""" + s = _seeded(repo) + seen: list = [] + _patch_notes_add(monkeypatch, 0, seen) + + m = M.seal(s, repo) + + assert len(seen) == 1 + argv, stdin = seen[0] + assert "-F" in argv + assert "-" in argv + assert "-m" not in argv + body = m.to_json() + assert body not in [el.encode("ascii") if isinstance(el, str) else el for el in argv] + assert all(body.decode("ascii") != el for el in argv) + assert stdin == body + + +def test_note_larger_than_arg_max_publishes_and_resolves(repo: Path): + """A body over 1.5 MB — past Linux's 131,072-byte MAX_ARG_STRLEN per argument + and past macOS's 1,048,576-byte kern.argmax — publishes and re-resolves. + + 256 KB is the wrong number: it succeeds under `-m` on macOS and fails under + `-m` on Linux, so it proves nothing and diverges by platform. + """ + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label="x" * 1_600_000, + event_indices=(0,), + declared_at_index=0, + ), + ) + + m = M.seal(s, repo) + body = _note_bytes(repo) + assert body is not None + assert len(body) >= 1_572_864, f"body is only {len(body)} bytes — too small to bind" + + resolved = _round_trip(body) + assert resolved == m + assert M.verify(s, repo).all_verified + + +def test_happy_path_publishes_exactly_what_it_returns(repo: Path): + s = _seeded(repo) + + m = M.seal(s, repo) + + assert _seal_lines(repo) == 1 + body = _note_bytes(repo) + assert body is not None + assert _round_trip(body) == m + + +def test_happy_path_cli_still_exits_zero(repo: Path): + _seeded(repo) + + assert cli.main(["--repo", str(repo), "seal"]) == 0 + assert _seal_lines(repo) == 1 + assert _note_bytes(repo) is not None + + +def test_manifest_json_bytes_are_pinned(): + """The manifest's on-the-wire bytes are compat surface across 65 sealed + notes. This seal change moves no note byte; a `to_json` tidy-up would. + """ + m = M.Manifest( + version=1, + commit="0" * 40, + tree="1" * 40, + claims=[ + { + "claim": { + "argv_preview": ["pytest", "-q"], + "ctype": "tests-pass", + "declared_at_index": 0, + "event_indices": [0], + "label": "t", + "pathspecs": [], + }, + "delta": [], + "exit_code": 0, + "grade": "tree-exact", + "reason": "r", + "supporting_event_index": 0, + } + ], + coverage={"by_coverage": {"complete": 1}, "total_events": 1}, + ) + assert m.to_json() == ( + b'{"claims":[{"claim":{"argv_preview":["pytest","-q"],"ctype":"tests-pass",' + b'"declared_at_index":0,"event_indices":[0],"label":"t","pathspecs":[]},' + b'"delta":[],"exit_code":0,"grade":"tree-exact","reason":"r",' + b'"supporting_event_index":0}],' + b'"commit":"0000000000000000000000000000000000000000",' + b'"coverage":{"by_coverage":{"complete":1},"total_events":1},' + b'"secrets_override":false,' + b'"tree":"1111111111111111111111111111111111111111","version":1}' + ) From e6e47ab48f599af495e7618ed29516bbe912e0de Mon Sep 17 00:00:00 2001 From: Drew Date: Wed, 29 Jul 2026 01:46:45 -0700 Subject: [PATCH 2/8] verify: bind the verdict to the sealed evidence, and stop losing interrupted flights P1 and P2 of the v0.2 hardening pack. P1 -- the reason v0.2 exists. verify --strict was not bound to the evidence it sealed: it regraded claims against whatever ledger was on disk now, keyed by integer index, never called verify_chain, and sealed no event hash, tree or fingerprint into the note. Measured on v0.1: seal three claims, delete the ledger, run three unrelated commands (one /usr/bin/true) into a fresh ledger on the same tree, and it reported 3/3 recorded-exact, exit 0. That scenario now reports witness-unavailable on every claim and exits 1. Claims now carry an evidence block naming the entry that backed them, verify checks that binding, a broken chain dominates the verdict, and tree_delta returns None rather than [] when the delta cannot be computed -- an archived ledger used to produce the confident false sentence "evidence tree equals sealed tree". Manifest version 2; v1 notes still regrade exactly as before, down to the reason string. P2 -- operability. The ledger root is 0700 and read-only verify no longer creates one as a side effect. Output can stream (--tee/--heartbeat) instead of appearing only in a summary line after the command ends. Appends take an flock so concurrent writers cannot corrupt the chain. Interrupt safety needed two fixes, both found by reproducing the loss rather than reasoning about it. Signals are now held across the digest-and-append window, so a second Ctrl-C cannot unwind the recorder between the drain and the record (measured 12/30 flights lost before, 0/28 after). And the after-digest never costs the event on any path: a terminal Ctrl-C reaches the whole process group and kills the git the digest shells out to, even when the child exited cleanly, which lost a completed command's event 11/20 times. tree_after is None there and coverage says observed-text-only rather than claiming complete. COMPAT.md said a v0.1 reader parses a v2 note without crashing. Measured, it exits 2 -- correctly, by the refuse-never-coerce rule. Documented as the real forward incompatibility it is: upgrade the verifier before the sealer. 96 -> 193 passing, 4 env-gated skips; harness.recall still PASS at 100%. --- docs/COMPAT.md | 107 ++++ docs/MEASUREMENTS.md | 32 ++ src/didrun/capture.py | 534 +++++++++++++++++- src/didrun/claims.py | 61 ++- src/didrun/cli.py | 151 ++++- src/didrun/gitplumbing.py | 11 +- src/didrun/ledger.py | 361 ++++++++++-- src/didrun/manifest.py | 306 ++++++++++- src/didrun/render.py | 101 +++- tests/compat/test_corpus_replay.py | 7 +- tests/test_append_concurrency.py | 610 +++++++++++++++++++++ tests/test_chain_gate.py | 395 +++++++++++++ tests/test_evidence_binding.py | 406 ++++++++++++++ tests/test_grading_honesty.py | 209 +++++++ tests/test_interrupt_capture.py | 409 ++++++++++++++ tests/test_interrupt_records_the_flight.py | 121 ++++ tests/test_ledger.py | 37 ++ tests/test_ledger_permissions.py | 266 +++++++++ tests/test_redact_render.py | 20 +- tests/test_seal_publication.py | 4 +- tests/test_streaming_capture.py | 349 ++++++++++++ 21 files changed, 4369 insertions(+), 128 deletions(-) create mode 100644 tests/test_append_concurrency.py create mode 100644 tests/test_chain_gate.py create mode 100644 tests/test_evidence_binding.py create mode 100644 tests/test_grading_honesty.py create mode 100644 tests/test_interrupt_capture.py create mode 100644 tests/test_interrupt_records_the_flight.py create mode 100644 tests/test_ledger_permissions.py create mode 100644 tests/test_streaming_capture.py diff --git a/docs/COMPAT.md b/docs/COMPAT.md index 06cffdf..0939db7 100644 --- a/docs/COMPAT.md +++ b/docs/COMPAT.md @@ -40,6 +40,43 @@ guess: the chain check reports it as unverifiable and names the index. - **Every new manifest key is read with `.get()` and a default that reproduces v1 semantics.** A required key would break reading every note already published. +### Version 2 — a claim names the evidence it was sealed against + +A v2 note may carry an `evidence` block inside a claim entry, naming the recorded ledger +entry that backed that claim: its index, its chain `entry_hash`, and the environment +fingerprint and tree digest recorded on that event. `verify` checks that binding before +grading anything. When the live ledger cannot supply that entry — the index is out of +range, or a different entry sits there — the claim is reported as `witness-unavailable` +carrying the grade the note sealed, and `--strict` refuses it. + +The version bumped for the change of MEANING, not for the new key. Before v2, verify +silently regraded against whatever ledger is present, so archival, rebuild or index drift +changed a sealed verdict with no signal. That is the defect v2 closes. + +What this buys is **accident and drift detection on a published note**, not resistance to +a local forger. Whoever can substitute a ledger can regenerate its chain and re-run +`claim` and `seal` to mint fresh hash-bound claims. `ledger.py` states that non-guarantee +in the source and nothing here changes it. + +Old notes stay readable. New notes do not travel backwards: + +- **v0.2 reading a v1 note.** Every note published before v0.2 is version 1 and carries + no `evidence` block, so it takes the unbound path and regrades exactly as v0.1 regraded + it, down to the reason string. +- **v0.1 reading a v2 note — REFUSED, by design.** v0.1's `Manifest.from_json` raises on + any version above its own, so `didrun verify` exits 2 with + `manifest version 2 is newer than this didrun understands (max 1); upgrade didrun`. + This is the refuse-never-coerce rule at the top of this document applied to us: a v0.1 + reader would regrade a v2 note index-only and report a confident verdict without ever + checking the binding the note was sealed under. Refusing is the correct behaviour, but + it is a real forward incompatibility and it bites a mixed-version setup — seal on a + v0.2 workstation, verify in v0.1 CI, and every commit fails. **Upgrade the verifier + first, then the sealer.** + +The evidence block lives in the manifest, never in an `Event`. Putting it into the chain +preimage would invalidate every `entry_hash` ever written — the break the frozen preimage +above exists to prevent. + When didrun falls back to resolving a manifest by tree id it scans the notes ref. A note it cannot parse at all — a foreign note under the same ref, a corrupt body — is skipped and counted, and the count is reported, so one bad note cannot hide every good one. A @@ -77,6 +114,76 @@ proof about all repositories. Both configurations are pinned by tests. `.didrun/` itself remains excluded unconditionally, regardless of gitignore state. +## Ledger permissions (v0.2) and where the guarantee stops + +v0.2 forces the ledger ROOT to `0700` on every writable open — creation and re-open +both, because `mkdir(exist_ok=True)` never re-tightens a directory an older binary +already made at `0755`. Files this version creates (`session.log`, `claims.jsonl`, +`seals.jsonl`, `.gitignore`, blob temp files) are created `0600`. + +Three bounds, stated because a rounded-up version of any of them would be an overclaim: + +- **Inherited files keep their modes.** The create mode applies at creation only, and + `.gitignore` is written only when absent. A ledger a v0.1 binary produced keeps `0644` + on those files. The root's `0700` is what makes that acceptable — another uid cannot + reach them through the directory — and it is **not** the same statement as "the files + are 0600". +- **`.didrun/objects/**` is not covered.** git writes into it directly through the + `GIT_OBJECT_DIRECTORY` redirect, twice per `didrun run`, at its own modes. didrun does + not chmod what git owns. +- **Windows.** `os.chmod` is close to a no-op on NTFS, so `.didrun/` inherits the parent + directory's ACLs and this guarantee does not hold there. The ownership check that + guards the chmod uses `os.geteuid`, which does not exist on Windows; it is skipped. + +Read-only commands (`verify`, `show`) no longer open a writable session: they create no +ledger directory, write no `.gitignore`, and re-mode nothing. Before v0.2, `didrun verify` +in a repo with no ledger *manufactured* an empty world-readable one, after which every +claim graded `unknown`; pointing `show` at an archived ledger re-moded it. Nothing on disk +changes shape, and `.didrun/` is excluded from tree digests unconditionally, so no +archived tree id or sealed note is affected. + +## Append serialization is POSIX-only (v0.2, behaviour, not format) + +From v0.2 the read-tail-and-append pair inside `Session.append` — and the `claims.jsonl` / +`seals.jsonl` appends — run under an `fcntl.flock` on `.didrun/.lock`, so two didrun +processes writing to one ledger cannot both record the same chain index. **On a platform +without `fcntl` the lock is a declared no-op and didrun degrades to the v0.1 behaviour: +concurrent writers can fork the chain, which `verify_chain` then reports as a permanent +break.** No portable fallback is invented, because a fallback that does not actually +exclude would be worse than a stated gap. `flock` is also emulated or ignored on NFS and +some other network filesystems; the exclusion there is as good as the filesystem's and no +better. + +Nothing stored changes. The lock file lives inside the ledger directory, which is excluded +from every tree digest unconditionally and ignored by the `*` rule the ledger writes into +its own `.gitignore`, so it moves no tree id, reaches no sealed note, and cannot be +committed. The lock is held for one short local write and is **never** held across a +wrapped child process — a `didrun run` that takes an hour blocks nobody. + +Separately, a blob's temp file is now named per writer rather than `.tmp`. Two +recorders storing the same bytes at the same time — an empty stderr is a digest every +event shares — used to race on that one name, and the loser died with `FileNotFoundError` +instead of recording. Stored blob names, contents and modes are unchanged. + +## Tier-0 capture streams from v0.2 (behaviour, not format) + +v0.2 drains the wrapped command's pipes incrementally instead of buffering them to +completion. Nothing recorded changes: the stored blobs are byte-identical to what the +buffered path produced, per stream, and the event's field set, `observed_via`, `coverage` +and exit code are untouched. What changes is that `didrun run` can now show the command's +output while it runs (`--tee`) and print a content-free progress line (`--heartbeat N`). +Both default **off**, so a caller parsing didrun's stdout sees exactly what it saw before. + +The drain uses `selectors`, which on Windows is a `select()` over sockets and cannot +register an anonymous pipe. Rather than make Tier-0 capture POSIX-only, registration +failure falls back to one blocking reader thread per pipe — same behaviour, no new +dependency. **The fallback is declared, not measured:** CI runs ubuntu and macOS only, so +no Windows leg exercises it. + +`didrun show --event N --output` reads a recorded blob back (`--stream stderr`, +`--redacted`). It is a reader over the existing store, adds no format, and re-hashes on +read, so a corrupted blob is a refusal rather than bytes presented as the record. + ## Two things v0.2 does not close Stated here because a trust tool's silence about its own limits is the same defect as an diff --git a/docs/MEASUREMENTS.md b/docs/MEASUREMENTS.md index ec041eb..1118e46 100644 --- a/docs/MEASUREMENTS.md +++ b/docs/MEASUREMENTS.md @@ -34,6 +34,38 @@ Relevant detail for Claude Code specifically: its Bash tool spawns the **user's `/bin/zsh -c` on a default macOS box — which is why Tier 2 is per-shell dispatch rather than `BASH_ENV` alone. +## Capture: what the streaming pump costs (v0.2) + +Tier 0 drained the child through `subprocess.run(capture_output=True)` until +v0.2, which buffers both pipes to completion — so a wrapped command surfaced +nothing at all until it exited (measured on real sessions: single events of +201.95 s, 332.42 s, 447.77 s and 530.21 s with no progress, and a 14.6-hour +chain whose entire operator-visible output was one summary line per command). +v0.2 replaces it with a `selectors` pump that drains both pipes incrementally. + +The question that had to be answered with numbers rather than a shrug is what +the pump costs per spawn. Both arms measured **alternately in one process** (so +machine load is shared, not donated to whichever arm ran during a quiet minute), +macOS 26.5 / APFS / arm64, Python 3.14.5: + +| Shape | Old — `subprocess.run(capture_output=True)` | New — `selectors` pump | Delta (median) | +|---|---|---|---| +| `/bin/echo hi`, 200 paired runs | 2.340 ms median (mean 2.358, p90 2.601) | 2.364 ms median (mean 2.370, p90 2.628) | **+0.024 ms (+1.0%)** | +| 1 MiB to stdout, 40 paired runs | 24.099 ms median (p90 24.661) | 23.581 ms median (p90 24.309) | **−0.517 ms (−2.1%)** | + +So the pump is ~24 µs per short spawn — the same order as the wrapper's own ++55 µs, and invisible next to the two tree digests a `didrun run` already pays +(a full `run_wrapped` over `/bin/echo hi` is ~245 ms median on this machine, +almost all of it digest). On large output it is marginally *faster* than +`communicate()`. Both readings are single-machine and macOS-only; the Linux +rerun this file already gates on covers this table too. + +What the pump does **not** change is what is recorded: the drained blobs are +byte-identical to the buffered path's, per stream, over empty, 1 MiB, interleaved, +non-UTF-8 (NUL-bearing) and stderr-only output, and exit codes match including +signal deaths. That equality is the criterion, and it is a test +(`tests/test_streaming_capture.py`), not a claim made here. + ## Tree digests: why copied-index write-tree The digest must reflect staged + unstaged + **untracked** state, deterministically, diff --git a/src/didrun/capture.py b/src/didrun/capture.py index 2826344..d51ff37 100644 --- a/src/didrun/capture.py +++ b/src/didrun/capture.py @@ -21,11 +21,13 @@ import hashlib import os import selectors +import signal import subprocess import sys +import threading import time from pathlib import Path -from typing import Optional +from typing import Callable, Optional from .ledger import Event, Session from . import gitplumbing @@ -74,13 +76,456 @@ def fingerprint_version(value: str) -> Optional[int]: return int(digits) -def run_wrapped(argv: list[str], session: Session, repo: Optional[Path] = None) -> Event: +# --- Tier 0: the output pump -------------------------------------------------- +# +# v0.1 ran the child through subprocess.run(capture_output=True), which buffers +# both pipes to completion inside communicate(). The child's output therefore +# reached the terminal not progressively and not at the end but *never*: the +# whole operator-visible record of a 14.6-hour chain was one summary line per +# command. It is also why an interrupted command had no partial output to keep — +# subprocess.run's except clause kills the child and re-raises with the buffers +# still bound inside communicate(), where nothing can reach them. +# +# The pump below drains both pipes incrementally instead. What is *recorded* is +# unchanged, byte for byte; only when the bytes become visible changes. + +_PUMP_CHUNK = 65536 + + +class CaptureBuffers: + """The accumulating stdout/stderr of one wrapped child. + + Owned by the caller rather than hidden inside the pump so that whatever was + drained before an exception is still reachable afterwards. Byte counts are + kept alongside the chunks so a progress line can report volume without ever + touching content. + """ + + def __init__(self) -> None: + self._chunks: dict[str, list[bytes]] = {"stdout": [], "stderr": []} + self._counts: dict[str, int] = {"stdout": 0, "stderr": 0} + # The threaded drain has two writers; the selector drain has one. The + # lock costs nothing measurable and keeps one implementation of feed(). + self._lock = threading.Lock() + + def feed(self, stream: str, data: bytes) -> None: + with self._lock: + self._chunks[stream].append(data) + self._counts[stream] += len(data) + + def bytes_drained(self, stream: str) -> int: + with self._lock: + return self._counts[stream] + + @property + def stdout(self) -> bytes: + with self._lock: + return b"".join(self._chunks["stdout"]) + + @property + def stderr(self) -> bytes: + with self._lock: + return b"".join(self._chunks["stderr"]) + + +def _tee_chunk(stream: str, data: bytes) -> None: + """Write one drained chunk straight to this process's terminal. + + RAW BYTES, deliberately unredacted. Redaction is whole-buffer: a _TOKENISH + match can straddle a chunk boundary, so a streaming redactor would emit half + a secret and call it covered. ``--tee`` is local terminal output — the same + exposure as running the command directly — and is never an export. Every + export path (note, bundle, HTML, ``show --output --redacted``) still runs + through redact. + """ + out = sys.stdout if stream == "stdout" else sys.stderr + buf = getattr(out, "buffer", None) + if buf is not None: + buf.write(data) + buf.flush() + else: # a text-only stream (a test harness's capture object, say) + out.write(data.decode("utf-8", "replace")) + out.flush() + + +def _heartbeat_line(argv: list[str], elapsed: float, buffers: CaptureBuffers) -> str: + """One progress line: elapsed, command identity, volume. No content. + + The command identity is argv[0]'s basename and an argument count, not the + argv itself: argv is the carrier redact.scrub_argv exists for (``FOO_TOKEN=…`` + is a measured shape), and a heartbeat that pasted it into a build log would + be a leak this file introduced. Byte counts are volume, never a digest of + the output text — a rolling digest of content is content. + """ + name = os.path.basename(argv[0]) if argv else "?" + return ( + f"didrun: {int(elapsed)}s {name} ({max(len(argv) - 1, 0)} args) " + f"stdout={buffers.bytes_drained('stdout')}B " + f"stderr={buffers.bytes_drained('stderr')}B" + ) + + +def _emit_heartbeat(argv: list[str], elapsed: float, buffers: CaptureBuffers) -> None: + # stderr, so a caller parsing didrun's stdout is unaffected even with the + # flag on. + print(_heartbeat_line(argv, elapsed, buffers), file=sys.stderr, flush=True) + + +def _selector_for(proc: "subprocess.Popen") -> Optional[selectors.BaseSelector]: + """A selector with both pipes registered, or None if they cannot be. + + ``selectors.DefaultSelector`` on Windows is a select() over sockets and + cannot take an anonymous pipe. Registration fails before a single byte is + read, so returning None here is a clean handover to the threaded drain + rather than a partial one. + """ + sel = selectors.DefaultSelector() + try: + sel.register(proc.stdout, selectors.EVENT_READ, "stdout") + sel.register(proc.stderr, selectors.EVENT_READ, "stderr") + except (ValueError, OSError, NotImplementedError): + sel.close() + return None + return sel + + +def _drain_via_selector( + sel: selectors.BaseSelector, + buffers: CaptureBuffers, + tee: bool, + heartbeat: Optional[float], + argv: list[str], + origin: float, +) -> None: + # One chunk per ready stream per pass: neither pipe is ever read to + # completion while the other fills, which is the deadlock the buffered path + # avoided by using communicate(). + next_beat = origin + heartbeat if heartbeat else None + while sel.get_map(): + timeout = None + if next_beat is not None: + timeout = max(0.0, next_beat - time.monotonic()) + for key, _mask in sel.select(timeout): + data = os.read(key.fd, _PUMP_CHUNK) + if not data: # EOF on this pipe + sel.unregister(key.fileobj) + key.fileobj.close() + continue + buffers.feed(key.data, data) + if tee: + _tee_chunk(key.data, data) + if next_beat is not None and time.monotonic() >= next_beat: + _emit_heartbeat(argv, time.monotonic() - origin, buffers) + next_beat = time.monotonic() + heartbeat + + +def _drain_via_threads( + proc: "subprocess.Popen", + buffers: CaptureBuffers, + tee: bool, + heartbeat: Optional[float], + argv: list[str], + origin: float, +) -> None: + """Fallback drain: one blocking reader thread per pipe. + + Same no-starvation property as the selector pump (neither stream waits on + the other) at the cost of two threads per wrapped command. stdlib only — + this is the alternative to declaring Tier-0 capture POSIX-only, and adding + a dependency to pump a pipe is not on the table. + """ + tee_lock = threading.Lock() + + def reader(stream: str, pipe) -> None: + try: + while True: + data = pipe.read1(_PUMP_CHUNK) + if not data: + break + buffers.feed(stream, data) + if tee: + with tee_lock: + _tee_chunk(stream, data) + finally: + pipe.close() + + threads = [ + threading.Thread(target=reader, args=("stdout", proc.stdout), daemon=True), + threading.Thread(target=reader, args=("stderr", proc.stderr), daemon=True), + ] + for t in threads: + t.start() + next_beat = origin + heartbeat if heartbeat else None + while True: + alive = [t for t in threads if t.is_alive()] + if not alive: + return + timeout = None + if next_beat is not None: + timeout = max(0.0, next_beat - time.monotonic()) + alive[0].join(timeout) + if next_beat is not None and time.monotonic() >= next_beat: + _emit_heartbeat(argv, time.monotonic() - origin, buffers) + next_beat = time.monotonic() + heartbeat + + +# --- Tier 0: interruption ----------------------------------------------------- +# +# An operator SIGINT reaches the whole foreground process group, so the child +# takes it too. v0.1 let the resulting KeyboardInterrupt propagate out of +# subprocess.run: the child was killed, the buffers were discarded inside +# communicate(), and CPython exited 130 with a traceback BEFORE session.append +# ran. The flight recorder lost the flight — the exact failure it exists to +# prevent, and a silent one, because absence of evidence stopped being visible. +# +# What replaces it records the same event the normal path would, minus the two +# facts nobody witnessed: no exit code, and coverage "unobserved". Both are +# EXISTING vocabulary (ledger.COVERAGE), so no Event field moves and no archived +# entry_hash changes. The grading ladder needs no edit either: _supporting_event +# skips the event (exit_code != 0) and _witnessed_failure skips it (exit_code is +# None), so a claim bound to it grades `unknown` — the correct answer, for free. + +_INTERRUPT_SIGNALS = ("SIGINT", "SIGTERM") + +# How long the child gets after the signal is forwarded before it is killed. +# The wait is a timer rather than a sleep because the handler runs on the main +# thread, which is the thread draining the pipes — sleeping there would stall +# the drain that keeps the child from blocking on a full pipe. +_CHILD_GRACE_SECONDS = 2.0 + + +def signal_name(signum: int) -> str: + """``SIGINT`` for 2, and a readable fallback for anything unnamed.""" + try: + return signal.Signals(signum).name + except ValueError: + return f"signal {signum}" + + +class CaptureInterrupted(Exception): + """A wrapped command was interrupted AND its event has been recorded. + + Raised only after ``session.append`` returns, so the exception means "the + flight is on the record", never "the record was lost". It carries the + signal number so the CLI can exit on the conventional code without a + traceback, and the appended event's index so an operator is told where the + partial record is. + """ + + def __init__(self, signum: int, event_index: Optional[int] = None) -> None: + self.signum = signum + self.event_index = event_index + super().__init__( + f"interrupted by {signal_name(signum)}; " + f"recorded event {event_index} as unobserved" + ) + + @property + def exit_code(self) -> int: + # 128 + N, the shell convention: 130 for SIGINT, 143 for SIGTERM. + return 128 + self.signum + + +def _kill_quietly(proc: "subprocess.Popen") -> None: + try: + proc.kill() + except (OSError, ValueError): + pass + + +def _install_interrupt_handlers( + proc: "subprocess.Popen", caught: dict +) -> Callable[[], None]: + """Forward SIGINT/SIGTERM to the child, remember the first one that arrived. + + Returns a callable that restores the previous handlers. The caller MUST run + it in a ``finally``: leaving didrun's handlers installed would change the + interrupt behaviour of an embedder's entire process, long after the child + it was installed for is gone. + + MAIN THREAD ONLY. ``signal.signal`` raises ``ValueError: signal only works + in main thread`` anywhere else, and ``run_wrapped`` is a library entry point + a harness or an embedder may legitimately call from a worker. Off the main + thread this installs nothing and records nothing new — the uninterrupted + path is byte-for-byte what it was. + """ + if threading.current_thread() is not threading.main_thread(): + return lambda: None + + timers: list = [] + previous: dict = {} + + def handler(signum, _frame): + if caught["signum"] is None: + caught["signum"] = signum + # Forward first. A terminal Ctrl-C already delivered the signal to the + # whole foreground process group, in which case this is a harmless + # no-op; a signal sent to didrun alone (a supervisor, a test) reaches + # the child only here. + try: + proc.send_signal(signum) + except (OSError, ValueError): + pass + # Then escalate, so a child that ignores the signal cannot hang the + # recorder. The kill closes the pipes, which ends the drain. + timer = threading.Timer(_CHILD_GRACE_SECONDS, _kill_quietly, args=(proc,)) + timer.daemon = True + timer.start() + timers.append(timer) + + for name in _INTERRUPT_SIGNALS: + signum = getattr(signal, name, None) + if signum is None: # not every platform defines both + continue + try: + previous[signum] = signal.signal(signum, handler) + except (ValueError, OSError, RuntimeError): + continue + + def restore() -> None: + for timer in timers: + timer.cancel() + for signum, prev in previous.items(): + try: + signal.signal(signum, prev) + except (ValueError, OSError, RuntimeError): + pass + + return restore + + +def _defer_interrupts(caught: dict) -> Callable[[], None]: + """Hold SIGINT/SIGTERM until the flight is on the record. + + The child is already reaped by the time this is installed, so there is + nothing left to forward to and nothing to escalate against — the only job + is to stop a signal from unwinding the recorder between the drain and + ``session.append``. Without it a second Ctrl-C lands in the digest/append + window and raises ``KeyboardInterrupt`` out of ``run_wrapped``, losing the + event for a command that actually ran and, on the drain path, discarding + the interrupt evidence the drain went to the trouble of collecting. + + The signal is recorded, not dropped: the caller re-raises it as + ``CaptureInterrupted`` once the entry is durable, so the operator still + gets the interrupt they asked for and the conventional 130 exit. + + MAIN THREAD ONLY, for the same reason as ``_install_interrupt_handlers``. + """ + if threading.current_thread() is not threading.main_thread(): + return lambda: None + + previous: dict = {} + + def handler(signum, _frame): + if caught.get("deferred") is None: + caught["deferred"] = signum + + for name in _INTERRUPT_SIGNALS: + signum = getattr(signal, name, None) + if signum is None: + continue + try: + previous[signum] = signal.signal(signum, handler) + except (ValueError, OSError, RuntimeError): + continue + + def restore() -> None: + for signum, prev in previous.items(): + try: + signal.signal(signum, prev) + except (ValueError, OSError, RuntimeError): + pass + + return restore + + +def _spawn_and_drain( + argv: list[str], + cwd: str, + buffers: CaptureBuffers, + tee: bool = False, + heartbeat: Optional[float] = None, +) -> tuple: + """Run ``argv``, draining both pipes as they fill. + + Returns ``(exit_code, signum)``, where ``signum`` is the interrupting signal + or ``None``. On interruption the exit code is still read — the child is dead + by then — but the caller must not record it: nobody witnessed the command + finish, and a kill-induced ``-2`` is not the command's verdict. + + The bytes fed into ``buffers`` are the bytes ``capture_output=True`` would + have returned — that equality is the criterion this change is held to, and + it is tested per stream over empty, large, interleaved and non-UTF-8 output + rather than asserted here. + """ + caught: dict = {"signum": None} + with subprocess.Popen( + argv, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE + ) as proc: + origin = time.monotonic() + restore = _install_interrupt_handlers(proc, caught) + try: + sel = _selector_for(proc) + if sel is None: + _drain_via_threads(proc, buffers, tee, heartbeat, argv, origin) + else: + with sel: + _drain_via_selector(sel, buffers, tee, heartbeat, argv, origin) + # Reaped inside the handlers' lifetime on purpose: a second signal + # arriving here is forwarded to an already-dead child instead of + # raising KeyboardInterrupt through the recording that follows. + exit_code = proc.wait() + except BaseException: + # Same disposition as the buffered path: kill the child and let + # Popen.__exit__ reap it. Unlike the buffered path, the caller's + # buffers hold everything drained up to this point. + proc.kill() + raise + finally: + restore() + return exit_code, caught["signum"] + + +def _tree_digest_or_none(repo: Path, ledger_objects: Path) -> Optional[str]: + """``tree_digest``, with a failure recorded as ``None`` instead of raised. + + The interrupt path only. The signal went to the whole foreground process + group, so git may have taken it too and the digest can fail in ways the + normal path never sees. ``None`` is the honest answer — the claim grades + ``unknown`` — and taking the digest must never cost the event being + recorded, which is the whole point of this path. + """ + try: + return gitplumbing.tree_digest(repo, ledger_objects) + except Exception: + return None + + +def run_wrapped( + argv: list[str], + session: Session, + repo: Optional[Path] = None, + tee: bool = False, + heartbeat: Optional[float] = None, + buffers: Optional[CaptureBuffers] = None, +) -> Event: """Tier 0: run a command under direct capture and record a complete Event. Captures argv, cwd, an env fingerprint, exit code, stdout/stderr as blobs, and git tree digests before and after. This is the only path that witnesses a real exit code, so its events are ``coverage="complete"``, ``observed_via="wrapper"``. + + ``tee`` mirrors the child's raw output to this terminal as it arrives, and + ``heartbeat`` prints a content-free progress line to stderr at most every N + seconds. Both are local display, both default off, and neither changes one + byte of what is recorded. ``buffers`` lets a caller own the accumulator and + read partial output back after an interrupt. + + On SIGINT/SIGTERM the event is still recorded — argv, cwd, fingerprint, both + tree digests and whatever output was drained before the signal — and then + ``CaptureInterrupted`` is raised. The event says what was not witnessed + rather than guessing it: no exit code, ``coverage="unobserved"``. """ repo = Path(repo or os.getcwd()) ledger_objects = session.blobs.root # digest objects live beside blobs @@ -88,31 +533,70 @@ def run_wrapped(argv: list[str], session: Session, repo: Optional[Path] = None) tree_before = gitplumbing.tree_digest(repo, ledger_objects) submod = gitplumbing.submodule_dirty(repo) + if buffers is None: + buffers = CaptureBuffers() started = time.time() - proc = subprocess.run(argv, cwd=str(repo), capture_output=True) - ended = time.time() - - tree_after = gitplumbing.tree_digest(repo, ledger_objects) - - stdout_digest = session.blobs.put(proc.stdout) - stderr_digest = session.blobs.put(proc.stderr) - - event = Event( - argv=tuple(argv), - cwd=str(repo), - env_fingerprint=env_fingerprint(), - observed_via="wrapper", - coverage="complete", - exit_code=proc.returncode, - started_at=started, - ended_at=ended, - stdout_blob=stdout_digest, - stderr_blob=stderr_digest, - tree_before=tree_before, - tree_after=tree_after, - submodule_dirty=submod, + exit_code, signum = _spawn_and_drain( + list(argv), str(repo), buffers, tee=tee, heartbeat=heartbeat ) - session.append(event) + ended = time.time() + interrupted = signum is not None + + # The child is reaped, but the flight is not on the record until append + # returns. Digesting a large worktree and writing two blobs is a real + # window, and it is exactly where a second Ctrl-C lands. Hold interrupts + # across it; honour them immediately afterwards. + deferred: dict = {"deferred": None} + stop_deferring = _defer_interrupts(deferred) + try: + # The after-digest never costs the event, on ANY path. A terminal + # Ctrl-C reaches the whole foreground process group, so the git the + # digest shells out to can be killed even when the child had already + # exited cleanly and the drain saw no signal at all. Losing a witnessed + # command over a failed digest is the worse trade every time: `None` + # grades `unknown`, which is the honest answer, while no event at all + # says the command never ran. + tree_after = _tree_digest_or_none(repo, ledger_objects) + digest_lost = tree_after is None and not interrupted + + stdout_digest = session.blobs.put(buffers.stdout) + stderr_digest = session.blobs.put(buffers.stderr) + + if interrupted: + coverage = "unobserved" + elif digest_lost: + # argv, streams and exit code were all witnessed; the resulting + # tree was not. Say exactly that rather than claiming "complete". + coverage = "observed-text-only" + else: + coverage = "complete" + + event = Event( + argv=tuple(argv), + cwd=str(repo), + env_fingerprint=env_fingerprint(), + observed_via="wrapper", + coverage=coverage, + exit_code=None if interrupted else exit_code, + started_at=started, + ended_at=ended, + stdout_blob=stdout_digest, + stderr_blob=stderr_digest, + tree_before=tree_before, + tree_after=tree_after, + submodule_dirty=submod, + ) + entry = session.append(event) + finally: + stop_deferring() + + if interrupted: + raise CaptureInterrupted(signum, entry.index) + # The command itself completed and is fully recorded; the operator + # interrupted the recorder afterwards. Report the interrupt, not a + # fabricated clean return. + if deferred["deferred"] is not None: + raise CaptureInterrupted(deferred["deferred"], entry.index) return event diff --git a/src/didrun/claims.py b/src/didrun/claims.py index b92d3f6..e79eda7 100644 --- a/src/didrun/claims.py +++ b/src/didrun/claims.py @@ -46,6 +46,13 @@ # The strongest negative signal: didrun WITNESSED the claimed command fail. This # is distinct from `unknown` (no evidence at all) — it is a claim caught lying. GRADE_FAILED = "failed" +# The note records a grade for this claim, but the live ledger cannot supply the +# evidence it was sealed against — this is not a verification. It is never +# produced by ``grade()``: the ladder below grades a claim against evidence it +# can see, and this state says the evidence itself is gone, or is not the +# evidence that was sealed. manifest.verify assigns it AROUND the ladder, and it +# is never admitted to the set --strict accepts. +GRADE_WITNESS_UNAVAILABLE = "witness-unavailable" class ClaimError(Exception): @@ -95,7 +102,13 @@ def from_dict(cls, d: dict) -> "Claim": @dataclass class GradeResult: - """The outcome of grading one claim against a target tree.""" + """The outcome of grading one claim against a target tree. + + ``evidence_bound`` and ``sealed_grade`` are verify-side facts, not grading + facts: they say whether this verdict was checked against the specific + recorded entry the seal named, and what that seal graded. ``grade()`` never + sets them, so a sealed note carries them at their defaults. + """ claim: Claim grade: str @@ -103,6 +116,8 @@ class GradeResult: delta: list = field(default_factory=list) # PathChange list when stale/scope supporting_event_index: Optional[int] = None exit_code: Optional[int] = None + evidence_bound: bool = False + sealed_grade: Optional[str] = None def to_dict(self) -> dict: return { @@ -112,6 +127,8 @@ def to_dict(self) -> dict: "delta": [{"status": c.status, "path": c.path} for c in self.delta], "supporting_event_index": self.supporting_event_index, "exit_code": self.exit_code, + "evidence_bound": self.evidence_bound, + "sealed_grade": self.sealed_grade, } @@ -157,20 +174,25 @@ def grade( The ordering is the honesty ladder: only concede a weaker grade when the stronger one cannot be honestly asserted. """ + # A witnessed failure among the bound events DOMINATES a witnessed success. + # Consulting _supporting_event first graded a claim bound to + # [failed, success] on the success and made the caught lie invisible. + # Reachable only through the library API today (the CLI's --event is + # type=int, so every CLI claim binds exactly one event). + failure = _witnessed_failure(claim, events) + if failure is not None: + fidx, fev = failure + return GradeResult( + claim, + GRADE_FAILED, + reason=f"the recorded command exited {fev.exit_code} — claim is not backed", + supporting_event_index=fidx, + exit_code=fev.exit_code, + ) + found = _supporting_event(claim, events) if found is None: - # No witnessed success. Is that because we watched it FAIL (a caught - # lie) or because there is no evidence at all (unknown)? - failure = _witnessed_failure(claim, events) - if failure is not None: - fidx, fev = failure - return GradeResult( - claim, - GRADE_FAILED, - reason=f"the recorded command exited {fev.exit_code} — claim is not backed", - supporting_event_index=fidx, - exit_code=fev.exit_code, - ) + # No witnessed success and no witnessed failure: no evidence at all. return GradeResult( claim, GRADE_UNKNOWN, @@ -223,6 +245,19 @@ def grade( # Compute the delta between the evidence tree and the sealed tree. delta = gitplumbing.tree_delta(repo, ev.tree_after, sealed_tree, ledger_objects) + if delta is None: + # Fail closed. `None` is "git could not compute it" — typically the + # archived-ledger case where the evidence tree's objects have moved. + # Every branch below would otherwise render that failure as a fact + # ("0 path(s) differ", "evidence tree equals sealed tree"), and an + # empty delta would also violate the stale-carries-the-delta invariant. + return GradeResult( + claim, + GRADE_UNKNOWN, + reason="delta not computable: evidence tree objects unavailable", + supporting_event_index=idx, + exit_code=ev.exit_code, + ) # scope-exact: every change is within the claimant-declared pathspecs. if claim.pathspecs: diff --git a/src/didrun/cli.py b/src/didrun/cli.py index fc2ac4e..d402138 100644 --- a/src/didrun/cli.py +++ b/src/didrun/cli.py @@ -16,10 +16,16 @@ from typing import Optional from . import __version__ -from .ledger import Session -from .capture import run_wrapped, DIDRUN_ACTIVE_ENV +from .ledger import LedgerError, Session +from .capture import ( + CaptureInterrupted, + DIDRUN_ACTIVE_ENV, + run_wrapped, + signal_name, +) from .claims import Claim from . import manifest as _manifest +from . import redact from . import render from . import gitplumbing @@ -28,8 +34,15 @@ def _ledger_dir(repo: Path) -> Path: return Path(repo) / ".didrun" -def _session(repo: Path) -> Session: - return Session(_ledger_dir(repo)) +def _session(repo: Path, readonly: bool = False) -> Session: + """Open the repo's ledger. + + ``readonly=True`` is for the commands that only read one: it creates no + directory, writes no .gitignore, and re-modes nothing. Without it `verify` + on a fresh clone MANUFACTURED an empty ledger — which then graded every + claim unknown — and pointing `show` at someone else's archive mutated it. + """ + return Session(_ledger_dir(repo), readonly=readonly) def cmd_run(args) -> int: @@ -46,7 +59,9 @@ def cmd_run(args) -> int: session = _session(repo) # Mark the session active so Tier-1/2 hooks (if installed) are live only now. os.environ.setdefault(DIDRUN_ACTIVE_ENV, "1") - event = run_wrapped(list(args.command), session, repo) + event = run_wrapped( + list(args.command), session, repo, tee=args.tee, heartbeat=args.heartbeat + ) coverage = "shim" if args.shim else "wrapper" print( f"recorded {coverage} event argv={' '.join(event.argv)!r} " @@ -117,7 +132,7 @@ def cmd_seal(args) -> int: def cmd_verify(args) -> int: repo = Path(args.repo or os.getcwd()) - session = _session(repo) + session = _session(repo, readonly=True) try: report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") except _manifest.ManifestError as exc: @@ -131,6 +146,14 @@ def cmd_verify(args) -> int: print(f"wrote HTML report to {args.html}") if not args.quiet: print(render.render_verdict(report)) + else: + # --quiet suppresses the VERDICT, not the ledger's integrity. A + # broken chain says the file that verdict was computed from does not + # recompute; a reader who sent the detail to HTML still needs that + # sentence where they are looking. + banner = render.render_chain_banner(report) + if banner: + print(banner) else: print(render.render_verdict(report)) if args.strict: @@ -138,9 +161,59 @@ def cmd_verify(args) -> int: return 0 +def _show_output(session: Session, args) -> int: + """Print one recorded output blob to stdout. + + The only path in the CLI that reads a blob back. Without it, reading the + output of a recorded failure meant resolving digests by hand under + `.didrun/objects`. `BlobStore.get` re-hashes on read, so a corrupted or + edited blob is a refusal here rather than bytes presented as the record. + """ + if args.event is None: + print( + "didrun show: --output needs --event N (which recorded event to read)", + file=sys.stderr, + ) + return 2 + events = session.events() + if not 0 <= args.event < len(events): + print( + f"didrun show: no event at index {args.event} " + f"(the session has {len(events)} events)", + file=sys.stderr, + ) + return 2 + ev = events[args.event] + digest = ev.stdout_blob if args.stream == "stdout" else ev.stderr_blob + if digest is None: + print( + f"didrun show: event {args.event} recorded no {args.stream} blob", + file=sys.stderr, + ) + return 2 + try: + data = session.blobs.get(digest) + except LedgerError as exc: + # Missing or corrupt evidence is a graded refusal, never a traceback. + print(f"didrun show: {exc}", file=sys.stderr) + return 2 + if args.redacted: + text, _findings = redact.redact(data) + data = text.encode("utf-8", "replace") + out = getattr(sys.stdout, "buffer", None) + if out is not None: + out.write(data) + out.flush() + else: + sys.stdout.write(data.decode("utf-8", "replace")) + return 0 + + def cmd_show(args) -> int: repo = Path(args.repo or os.getcwd()) - session = _session(repo) + session = _session(repo, readonly=True) + if args.output: + return _show_output(session, args) if args.session: # Session history view (subsumes the old `log` command). entries = list(session.entries()) @@ -150,6 +223,16 @@ def cmd_show(args) -> int: ok, broke = session.verify_chain() chain = "intact" if ok else f"BROKEN at index {broke}" print(f"session: {len(entries)} events chain {chain}") + if session.tail_truncated: + # A lost event is not tamper, and the chain line above is about the + # surviving prefix only. Saying so is the difference between a + # reader who knows an event is missing and one who does not. + print( + f" note: the log's final record is torn at byte " + f"{session.tail_truncated_offset} — one event was lost to an " + f"interrupted write, and `didrun run` will refuse to append " + f"until that is resolved" + ) for e in entries: ev = e.event exit_txt = f"exit {ev.exit_code}" if ev.exit_code is not None else "no-exit" @@ -183,6 +266,23 @@ def build_parser() -> argparse.ArgumentParser: pr = sub.add_parser("run", help="record a wrapped command execution (Tier 0)") pr.add_argument("--shim", action="store_true", help=argparse.SUPPRESS) + pr.add_argument( + "--tee", + action="store_true", + help=( + "also write the command's raw output to this terminal as it arrives " + "(local output only: NOT redacted, never an export)" + ), + ) + pr.add_argument( + "--heartbeat", + type=float, + metavar="SECONDS", + help=( + "print a progress line to stderr at most every SECONDS " + "(elapsed, command, bytes drained; no output content)" + ), + ) pr.add_argument("command", nargs=argparse.REMAINDER, help="-- [args…]") pr.set_defaults(func=cmd_run) @@ -210,6 +310,23 @@ def build_parser() -> argparse.ArgumentParser: pw.add_argument("--commit", help="commit to show (default: HEAD)") pw.add_argument("--session", action="store_true", help="show the recorded session history (subsumes `log`)") pw.add_argument("--html", help="write an HTML report to this path") + pw.add_argument("--event", type=int, metavar="N", help="event index to read with --output") + pw.add_argument( + "--output", + action="store_true", + help="print the recorded output of event N to stdout", + ) + pw.add_argument( + "--stream", + choices=["stdout", "stderr"], + default="stdout", + help="which recorded stream --output reads (default: stdout)", + ) + pw.add_argument( + "--redacted", + action="store_true", + help="run --output through the redactor before printing", + ) pw.set_defaults(func=cmd_show) return p @@ -218,7 +335,25 @@ def build_parser() -> argparse.ArgumentParser: def main(argv: Optional[list] = None) -> int: parser = build_parser() args = parser.parse_args(argv) - return args.func(args) + try: + return args.func(args) + except CaptureInterrupted as exc: + # The event IS on the record — that is what the exception means. One + # machine-greppable line and the conventional exit code (130 for SIGINT, + # 143 for SIGTERM), so a caller that used to detect interruption by + # scraping a traceback gets a better signal than the one it lost. + print( + f"didrun: interrupted by {signal_name(exc.signum)} — recorded event " + f"{exc.event_index} as unobserved (no exit code was witnessed)", + file=sys.stderr, + ) + return exc.exit_code + except KeyboardInterrupt: + # Backstop for an interrupt outside a wrapped child's lifetime — during + # a seal, say, where there is no in-flight command to record. A + # traceback is still the wrong way to report Ctrl-C. + print("didrun: interrupted (SIGINT)", file=sys.stderr) + return 130 if __name__ == "__main__": diff --git a/src/didrun/gitplumbing.py b/src/didrun/gitplumbing.py index 7bcbfab..1015832 100644 --- a/src/didrun/gitplumbing.py +++ b/src/didrun/gitplumbing.py @@ -228,12 +228,19 @@ class PathChange: def tree_delta( repo: Path, tree_a: str, tree_b: str, ledger_objects: Optional[Path] = None -) -> list[PathChange]: +) -> Optional[list[PathChange]]: """Exact path-level delta between two tree objects. Requires that both trees' objects are reachable — the ledger's redirected objects plus the repo's own. This is what turns "stale" from a shrug into a measured statement ("stale: 1 line changed in src/foo.py"). + + Two return values that must never be conflated: ``[]`` means *computed, and + the trees are equal*; ``None`` means *not computable* — git failed, so + nothing at all is known about the difference. Returning ``[]`` on failure + (v0.1 did) lets the caller render "0 path(s) differ" and "evidence tree + equals sealed tree" as facts derived from a subprocess that failed, which + is exactly the archived-ledger case where the evidence objects are gone. """ env = {} if ledger_objects is not None: @@ -246,7 +253,7 @@ def tree_delta( check=False, ) if proc.returncode != 0: - return [] + return None changes: list[PathChange] = [] for line in proc.stdout.splitlines(): parts = line.split("\t") diff --git a/src/didrun/ledger.py b/src/didrun/ledger.py index 080516a..6579e4b 100644 --- a/src/didrun/ledger.py +++ b/src/didrun/ledger.py @@ -18,12 +18,39 @@ from __future__ import annotations +import contextlib import hashlib import json import os +import sys +import tempfile from dataclasses import dataclass, field, asdict, fields as dataclass_fields from pathlib import Path -from typing import Any, Iterator, Optional +from typing import IO, Any, Iterator, Optional + +try: + import fcntl +except ImportError: # pragma: no cover - exercised only off POSIX + fcntl = None # type: ignore[assignment] + +# True when this platform can serialize appends between processes. False means +# the degradation documented in docs/COMPAT.md is live: concurrent writers can +# fork the chain, exactly as v0.1 did everywhere. No portable fallback is +# invented here — a fallback that does not actually exclude would be worse than +# a declared gap, because callers would stop treating concurrency as a hazard. +APPEND_LOCK_AVAILABLE = fcntl is not None + +# The ledger is secret-bearing by the tool's own account (docs/TRUST_MODEL.md): +# it holds argv verbatim, full stdout/stderr, and a content-addressed snapshot of +# every non-ignored file in the tree. 0700 on the root is the durable control — +# see Session.harden for what it does and does not cover. +LEDGER_DIR_MODE = 0o700 +LEDGER_FILE_MODE = 0o600 + +# The append mutex. It lives INSIDE the ledger directory, which is excluded from +# every tree digest unconditionally and self-ignored by the "*" rule __init__ +# writes — so it moves no digest and can never be committed. +LOCK_FILENAME = ".lock" # The genesis link. The first entry chains from this fixed, well-known value so # an empty vs tampered-truncated log are distinguishable. @@ -91,6 +118,59 @@ class LedgerError(Exception): """Raised on a structural violation of a ledger invariant.""" +def _open_private_append(path: Path) -> IO[bytes]: + """Open ``path`` for binary append, creating it 0600 rather than 0666&~umask. + + The mode argument to ``os.open`` applies at CREATION only: a file an older + binary already created at 0644 keeps 0644. That bound is deliberate and is + documented on ``Session.harden`` — the root's 0700 is what makes it + acceptable, and it is not the same statement as "the files are 0600". + """ + fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_APPEND, LEDGER_FILE_MODE) + return os.fdopen(fd, "ab") + + +@contextlib.contextmanager +def _append_lock(root: Path) -> Iterator[None]: + """Serialize a read-tail-and-append pair against other didrun processes. + + The hazard is corruption, not throughput. ``Session.append`` reads the tail, + derives ``(index, prev_hash)`` from it, then writes; two processes that + interleave those steps both write the SAME index chained from the SAME + predecessor, and the fork is permanent — repairing it means rewriting the + log, the one thing an evidence store must not do. + + ``flock`` is the right primitive precisely because it is owned by the kernel + and released when the holder dies, however it dies. A PID-file or + ``O_EXCL`` sentinel would introduce a stale-lock class that a SIGKILLed + process leaves behind forever; this introduces none. + + Two bounds, stated rather than glossed: + + - Without ``fcntl`` this is a no-op and concurrent writers can fork the + chain exactly as they could in v0.1. See ``APPEND_LOCK_AVAILABLE``. + - ``flock`` over NFS and some network filesystems is emulated or ignored. + The exclusion is as good as the filesystem's, and no better. + + The critical section is microseconds of local I/O. It is never held across a + wrapped child process: the child has already exited by the time an event + exists to append. + """ + if fcntl is None: + yield + return + path = Path(root) / LOCK_FILENAME + fd = os.open(str(path), os.O_RDWR | os.O_CREAT, LEDGER_FILE_MODE) + try: + fcntl.flock(fd, fcntl.LOCK_EX) + try: + yield + finally: + fcntl.flock(fd, fcntl.LOCK_UN) + finally: + os.close(fd) + + @dataclass(frozen=True) class Event: """One observed execution. @@ -225,18 +305,43 @@ class BlobStore: the seal step; here it is a plain directory of ```` files. """ - def __init__(self, root: Path) -> None: + def __init__(self, root: Path, readonly: bool = False) -> None: self.root = Path(root) - self.root.mkdir(parents=True, exist_ok=True) + self.readonly = readonly + # A read-only store creates nothing: it is opened by commands that only + # read a ledger, and a directory conjured by `verify` is the same defect + # as a world-readable one. + if not readonly: + self.root.mkdir(parents=True, exist_ok=True) def put(self, data: bytes) -> str: + if self.readonly: + raise LedgerError("read-only blob store: refusing to write a blob") digest = sha256_hex(data) path = self.root / digest if not path.exists(): # Write-once. Atomic via a temp file + rename so a crash mid-write # never leaves a partial blob under a valid-looking digest. - tmp = self.root / (digest + ".tmp") - tmp.write_bytes(data) + # + # The temp name is unique PER WRITER, not `.tmp`. Two didrun + # processes recording concurrently store the same blob constantly — + # an empty stderr is one digest shared by every event — and with a + # shared temp name the second writer's rename deletes the first + # writer's temp out from under it, so the first dies on chmod or + # replace with FileNotFoundError. Measured, not theorised: 2 of 6 + # eight-way concurrent `didrun run` attempts crashed that way. Both + # writers producing the same bytes is exactly what content + # addressing guarantees, so the last rename winning is correct. + tmp_fd, tmp_name = tempfile.mkstemp( + prefix=digest + ".", suffix=".tmp", dir=str(self.root) + ) + tmp = Path(tmp_name) + with os.fdopen(tmp_fd, "wb") as fh: + fh.write(data) + # os.replace preserves the SOURCE file's mode, so the blob inherits + # whatever the temp had. mkstemp creates 0600, but state it rather + # than depend on it: the blob is the one durable artifact here. + os.chmod(tmp, LEDGER_FILE_MODE) os.replace(tmp, path) return digest @@ -262,38 +367,179 @@ class Session: the first break. """ - def __init__(self, root: Path) -> None: + def __init__(self, root: Path, readonly: bool = False) -> None: self.root = Path(root) - self.root.mkdir(parents=True, exist_ok=True) - # The ledger is secret-bearing by construction (it records everything a - # session printed). Self-ignore the whole directory so a user's - # `git add -A` can never stage it — even if they forgot a root-level - # rule. A `*` here ignores every path in the ledger, including itself. - gitignore = self.root / ".gitignore" - if not gitignore.exists(): - gitignore.write_text("*\n", encoding="ascii") + self.readonly = readonly + # Set by every full read of the log (see _iter_records). A torn final + # record is a lost event, never tamper, so it is reported rather than + # raised — but it is never silent either. + self.tail_truncated = False + self.tail_truncated_offset: Optional[int] = None + # Recorded, not asserted: on a platform without fcntl the append is not + # serialized and two writers can fork the chain. A caller that wants to + # say so can read this; nothing in the trust path branches on it, because + # a lock is not evidence. + self.append_serialized = APPEND_LOCK_AVAILABLE + self.lock_path = Path(root) / LOCK_FILENAME + if not readonly: + self.root.mkdir(parents=True, exist_ok=True) + self.harden() + # The ledger is secret-bearing by construction (it records + # everything a session printed). Self-ignore the whole directory so + # a user's `git add -A` can never stage it — even if they forgot a + # root-level rule. A `*` here ignores every path in the ledger, + # including itself. + gitignore = self.root / ".gitignore" + if not gitignore.exists(): + with _open_private_append(gitignore) as fh: + fh.write(b"*\n") self.log_path = self.root / "session.log" - self.blobs = BlobStore(self.root / "objects") + self.blobs = BlobStore(self.root / "objects", readonly=readonly) + + def harden(self) -> None: + """Force the ledger ROOT to 0700, repairing a ledger already on disk. + + The chmod is the load-bearing half: ``mkdir(exist_ok=True)`` never + re-tightens an existing directory, so a ledger a v0.1 binary created at + 0755 would stay 0755 forever without this. It therefore runs on every + writable open, not only on creation. + + **The bound, stated because a rounded-up version of it would be a lie.** + This repairs the root and only the root. A ``session.log``, + ``claims.jsonl`` or ``.gitignore`` an older binary created at 0644 keeps + 0644 — ``__init__`` writes ``.gitignore`` only when it is absent, so it + is never re-created. 0700 on the root is what makes that acceptable: + another uid cannot reach those files through the directory. It is NOT + the same guarantee as "the files are 0600". + + ``objects/`` is deliberately not walked. git populates it directly via + ``GIT_OBJECT_DIRECTORY``, twice per ``didrun run``, at git's own modes, + so chmod-ing it would be undone by the next command. + + Never chmods a directory this process does not own: warns once on stderr + and continues. ``os.geteuid`` does not exist on Windows, where the check + is skipped (and where ``os.chmod`` is close to a no-op anyway — see + docs/COMPAT.md). + """ + geteuid = getattr(os, "geteuid", None) + if geteuid is not None: + try: + owner = self.root.stat().st_uid + except OSError as exc: + print(f"didrun: cannot stat {self.root}: {exc}", file=sys.stderr) + return + if owner != geteuid(): + print( + f"didrun: not tightening {self.root} to 0700 — it is owned " + f"by uid {owner}, not this process", + file=sys.stderr, + ) + return + try: + os.chmod(self.root, LEDGER_DIR_MODE) + except OSError as exc: + print( + f"didrun: could not set {self.root} to 0700: {exc}", + file=sys.stderr, + ) + + # --- reading the raw log -------------------------------------------------- + + def _iter_records(self) -> Iterator[dict]: + """Yield each stored record, tolerating a torn FINAL line. + + A process killed mid-append leaves a partial last line. ``json.loads`` + on it used to raise, and every reader called it, so one lost event + killed the entire ledger — reads AND writes. The tolerance here is + deliberately narrow: only the last line may fail to parse, the fact is + recorded on ``tail_truncated`` rather than swallowed, and an + unparseable line with data after it is corruption in the middle of the + log, which stays a hard error. + + Read in binary so a tear that split a multi-byte sequence is a parse + failure like any other rather than a decode error from the reader. + """ + self.tail_truncated = False + self.tail_truncated_offset = None + if not self.log_path.exists(): + return + torn_at: Optional[int] = None + with self.log_path.open("rb") as fh: + offset = 0 + for raw in fh: + start = offset + offset += len(raw) + if torn_at is not None: + raise LedgerError( + f"session.log has an unparseable record at byte " + f"{torn_at} with {offset - torn_at} bytes of data after " + f"it: that is corruption inside the log, not an " + f"interrupted final write" + ) + line = raw.strip() + if not line: + continue + try: + rec = json.loads(line) + except ValueError: + torn_at = start + continue + yield rec + if torn_at is not None: + self.tail_truncated = True + self.tail_truncated_offset = torn_at # --- writing ------------------------------------------------------------- def _last(self) -> tuple[int, str]: - """Return (last_index, last_hash) by reading the tail of the log.""" - if not self.log_path.exists(): - return -1, GENESIS_HASH - last_line = None - with self.log_path.open("r", encoding="ascii") as fh: - for line in fh: - line = line.strip() - if line: - last_line = line - if last_line is None: + """Return (last_index, last_hash) from the last PARSEABLE record. + + Shares ``_iter_records``' torn-tail tolerance deliberately. Giving it + only to ``entries()`` would leave every subsequent ``append`` raising + after an interrupted write: the reader would recover and the recorder + would stay dead, which is the failure this is here to prevent. Reading + past the tear is not the same as writing past it — ``append`` still + refuses that. + """ + last: Optional[dict] = None + for rec in self._iter_records(): + last = rec + if last is None: return -1, GENESIS_HASH - rec = json.loads(last_line) - return rec["index"], rec["entry_hash"] + return last["index"], last["entry_hash"] def append(self, event: Event) -> ChainEntry: + # Checked BEFORE the lock so a read-only session never creates or opens + # the lock file: `verify` and `show` must leave no trace in a ledger and + # must never contend with a writer. + if self.readonly: + raise LedgerError("read-only session: refusing to append to the ledger") + with _append_lock(self.root): + return self._append_locked(event) + + def _append_locked(self, event: Event) -> ChainEntry: + """The critical section: read the tail, derive the link, write the line. + + Split out only so the lock's extent is legible. Everything here must be + cheap and local — a wrapped child has long since exited by the time an + event exists to append, and the lock must never be widened to cover one. + """ last_index, last_hash = self._last() + if self.tail_truncated: + # Refuse; never truncate. Two reasons, and the second is the sharp + # one. The ledger is evidence, so a writer that edits it to make + # room is the one thing it must not do. And appending past the tear + # would leave those partial bytes in the MIDDLE of the log, where + # they are unparseable forever — a permanently broken chain, which + # turns one lost event into a dead ledger. The offset is named so + # an operator can decide. + raise LedgerError( + f"session.log has a torn final record at byte " + f"{self.tail_truncated_offset}; refusing to append. One event " + f"was lost to an interrupted write. Recover deliberately: " + f"truncate the file to {self.tail_truncated_offset} bytes, or " + f"archive this ledger and start a new one." + ) entry = ChainEntry(index=last_index + 1, prev_hash=last_hash, event=event) entry.entry_hash = entry.compute_hash() rec = { @@ -307,33 +553,32 @@ def append(self, event: Event) -> ChainEntry: if entry.preimage_version != 1: rec["preimage_version"] = entry.preimage_version # Append one canonical line. Newline-delimited so the tail read is cheap. - with self.log_path.open("ab") as fh: + with _open_private_append(self.log_path) as fh: fh.write(canonical_json(rec) + b"\n") + # The line is evidence the moment append() returns: a crash between + # here and the kernel's own flush would lose an event this process + # already reported as recorded. fsync changes durability, never + # bytes. + fh.flush() + os.fsync(fh.fileno()) return entry # --- reading / verifying ------------------------------------------------- def entries(self) -> Iterator[ChainEntry]: - if not self.log_path.exists(): - return - with self.log_path.open("r", encoding="ascii") as fh: - for line in fh: - line = line.strip() - if not line: - continue - rec = json.loads(line) - body = rec["event"] - entry = ChainEntry( - index=rec["index"], - prev_hash=rec["prev_hash"], - event=Event.from_dict(body), - entry_hash=rec["entry_hash"], - # Absent means version 1: that is what every archived - # record looks like and it must keep meaning v1. - preimage_version=rec.get("preimage_version", 1), - unknown_event_keys=unknown_event_keys_of(body), - ) - yield entry + for rec in self._iter_records(): + body = rec["event"] + entry = ChainEntry( + index=rec["index"], + prev_hash=rec["prev_hash"], + event=Event.from_dict(body), + entry_hash=rec["entry_hash"], + # Absent means version 1: that is what every archived + # record looks like and it must keep meaning v1. + preimage_version=rec.get("preimage_version", 1), + unknown_event_keys=unknown_event_keys_of(body), + ) + yield entry def events(self) -> list[Event]: return [e.event for e in self.entries()] @@ -347,6 +592,12 @@ def verify_chain_detail(self) -> ChainVerdict: ``unverifiable``: the projection would quietly hash a subset and reproduce the stored digest, so reporting ``intact`` there would be a lie and reporting ``broken`` would be a different one. + + A torn final record is none of those three. Every link that IS present + recomputes, so the status is ``intact`` over the surviving prefix and + the truncation travels in the reason. Calling it ``broken`` would read + as tamper for what is only a lost event, and that is the one + mistranslation this tool cannot afford. """ prev = GENESIS_HASH expected_index = 0 @@ -394,9 +645,19 @@ def verify_chain_detail(self) -> ChainVerdict: ) prev = entry.entry_hash expected_index += 1 + # Only reachable once the iteration finished, which is exactly when the + # torn-tail flag is known: every early return above stopped at a fault + # that dominates this one. + tail = "" + if self.tail_truncated: + tail = ( + f"the log's final record is torn at byte " + f"{self.tail_truncated_offset} — one event was lost to an " + f"interrupted write; this verdict covers the surviving prefix" + ) if not seen: - return ChainVerdict("empty") - return ChainVerdict("intact") + return ChainVerdict("empty", reason=tail) + return ChainVerdict("intact", reason=tail) def verify_chain(self) -> tuple[bool, Optional[int]]: """Recompute the chain. Returns (ok, first_broken_index). diff --git a/src/didrun/manifest.py b/src/didrun/manifest.py index 2fce0a6..2970a97 100644 --- a/src/didrun/manifest.py +++ b/src/didrun/manifest.py @@ -21,11 +21,16 @@ from pathlib import Path from typing import Optional -from .ledger import Session, canonical_json -from .claims import Claim, GradeResult, grade +from .ledger import Session, canonical_json, _append_lock, _open_private_append +from .claims import GRADE_WITNESS_UNAVAILABLE, Claim, GradeResult, grade from . import gitplumbing, redact -MANIFEST_VERSION = 1 +# 2: a claim may carry an `evidence` block naming the recorded entry that backed +# it, and verify CHECKS that binding instead of regrading by index alone. The +# bump is for the change of meaning, not for the new key: an older reader parses +# a v2 note fine (every key is additive) and regrades it index-only, exactly as +# it does today. See docs/COMPAT.md. +MANIFEST_VERSION = 2 NOTES_REF = "refs/notes/didrun" @@ -135,7 +140,11 @@ def seal( if tree is None: raise ManifestError(f"cannot resolve tree for commit {commit}") - events = session.events() + # Entries, not just events: the seal records WHICH recorded entry backed + # each claim, and an entry's chain hash is the only handle on that which + # survives the ledger being archived, rebuilt, or replaced. + entries = list(session.entries()) + events = [e.event for e in entries] ledger_objects = session.blobs.root # Seal scoping: a manifest carries the claims declared SINCE THE LAST SEAL — @@ -164,7 +173,7 @@ def seal( version=MANIFEST_VERSION, commit=commit, tree=tree, - claims=[_redact_result(r, session) for r in results], + claims=[_redact_result(r, session, entries) for r in results], coverage=_coverage_statement(session), secrets_override=bool(findings) and allow_secrets, ) @@ -196,6 +205,17 @@ def seal( return manifest +# A chain fault is not a grade any claim can carry; it is a statement about the +# ledger every grade was read out of. It dominates the verdict for that reason. +CHAIN_BROKEN = "chain-broken" + +# The two chain states that are faults. `empty` and `absent` are not: a missing +# or empty ledger is a witness-availability fact, already reported per claim as +# `witness-unavailable`, and calling it tamper would make every verify against +# an archived ledger look like an attack. +_CHAIN_FAULTS = ("broken", "unverifiable") + + @dataclass class VerifyReport: commit: str @@ -205,11 +225,34 @@ class VerifyReport: coverage: dict secrets_override: bool = False notes_skipped: int = 0 # notes the tree fallback could not parse at all + # The live ledger's chain state. Defaults to "absent" rather than "intact": + # a report built without consulting a ledger has not checked the chain, and + # saying "intact" there would be the same silent lie an unregistered grade + # is in render.py. + chain_status: str = "absent" # intact | broken | unverifiable | empty | absent + chain_broken_index: Optional[int] = None # set only when status is "broken" + chain_reason: str = "" + + @property + def chain_faulted(self) -> bool: + return self.chain_status in _CHAIN_FAULTS @property def worst_status(self) -> str: """Worst grade present — drives the verdict and the CI exit code.""" - order = ["failed", "unknown", "stale", "scope-exact", "tree-exact"] + # A broken chain dominates every grade, including "empty": the grades + # below it were read out of a ledger that does not recompute, so none of + # them is a statement anyone should act on. + if self.chain_faulted: + return CHAIN_BROKEN + order = [ + "failed", + "unknown", + GRADE_WITNESS_UNAVAILABLE, + "stale", + "scope-exact", + "tree-exact", + ] present = {r.grade for r in self.results} if not present: return "empty" @@ -220,20 +263,55 @@ def worst_status(self) -> str: @property def all_verified(self) -> bool: + # Membership is deliberately NOT widened. `witness-unavailable` reports + # a grade this run did not check, so admitting it here would make + # --strict pass on evidence nobody has seen. + if self.chain_faulted: + return False return bool(self.results) and all( r.grade in ("tree-exact", "scope-exact") for r in self.results ) + @property + def evidence_bound_count(self) -> int: + """Claims whose verdict was checked against the entry the seal named. + + The complement is not a failure: a v1 note carries no binding to check, + so its claims are regraded by index exactly as they always were. The + counter exists so a reader can tell the two apart at a glance. + """ + return sum(1 for r in self.results if r.evidence_bound) + + @property + def total(self) -> int: + return len(self.results) + def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyReport: """Resolve and re-grade a manifest for ``commitish``. Resolves by commit id, then by tree id (survives amend/rebase). Degrades a claim to unknown ("referenced object gc'd") rather than erroring. + + A claim the note bound to a specific recorded entry is only regraded when + the live ledger still holds that entry; otherwise the sealed grade is + reported as ``witness-unavailable`` and --strict refuses it. This detects + ACCIDENT AND DRIFT on a published note — an archived, rebuilt or replaced + ledger. It is not a forger barrier: whoever can substitute a ledger can + regenerate its chain and re-run claim+seal to mint fresh hash-bound claims + (ledger.py's declared non-guarantee, and docs/TRUST_MODEL.md). + + The chain is checked BEFORE any grading and a fault dominates the verdict: + a ledger whose links do not recompute cannot support a statement about + anything it holds. Until now ``verify_chain`` had exactly one caller and it + was ``show --session``, so a forked or mid-file-rewritten ledger certified + green under --strict. ``seal`` still does not consult the chain; that is a + separate policy decision, deliberately out of scope here. """ repo = Path(repo) commit = _rev(repo, commitish) tree = gitplumbing.commit_tree(repo, commitish) if commit else None + chain_status, chain_index, chain_reason = _chain_state(session) manifest, resolved_by, notes_skipped = _resolve_manifest(repo, commit, tree) if manifest is None: @@ -244,23 +322,24 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor results=[], coverage={}, notes_skipped=notes_skipped, + chain_status=chain_status, + chain_broken_index=chain_index, + chain_reason=chain_reason, ) # Re-grade the MANIFEST'S OWN claims against its sealed tree (deterministic # replay). The manifest is the only thing verify reads besides the repo — # the source-of-truth rule. Reading the live session claims file # here would let later, unsealed claims leak into an old commit's verdict. - events = session.events() + entries = list(session.entries()) + events = [e.event for e in entries] ledger_objects = session.blobs.root results: list[GradeResult] = [] for c in manifest.claims: claim = Claim.from_dict(c["claim"]) - try: - results.append(grade(claim, manifest.tree, events, repo, ledger_objects)) - except Exception as exc: # object gc'd or unreadable → unknown, not crash - results.append( - GradeResult(claim, "unknown", reason=f"regrade failed: {exc}") - ) + results.append( + _verify_claim(c, claim, manifest, entries, events, repo, ledger_objects) + ) return VerifyReport( commit=manifest.commit, tree=manifest.tree, @@ -269,9 +348,146 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor coverage=manifest.coverage, secrets_override=manifest.secrets_override, notes_skipped=notes_skipped, + chain_status=chain_status, + chain_broken_index=chain_index, + chain_reason=chain_reason, + ) + + +def _chain_state(session: Session) -> tuple[str, Optional[int], str]: + """The live ledger's chain state, as (status, broken_index, reason). + + ``verify_chain_detail`` reports a log with no records as ``empty`` whether + the file is missing or present-and-zero-length. Those are different facts + here: a missing log is an ARCHIVED (or never-created) ledger, which P1.2 + already reports honestly per claim as ``witness-unavailable``, and reporting + it as a chain state at all would put a ledger-integrity sentence in front of + a reader whose ledger is simply somewhere else. + + ``broken_index`` is populated only for ``broken``. An unverifiable entry has + no broken index — nothing was shown not to recompute; this binary could not + recompute it — so its index travels in the reason instead, which is where + ``verify_chain_detail`` already put it. + + A log this binary cannot even READ — a torn line, a record whose event body + carries keys the schema rejects — is ``unverifiable`` rather than a + traceback. Two reasons: the package's stated posture is that evidence it + cannot read is a graded refusal, not a crash (``cmd_verify``'s exit 2, + ``_regrade``'s fallback); and this runs BEFORE the manifest is resolved, so + without it a torn ledger would start crashing ``verify`` on commits with no + sealed note at all — a path that never read the ledger before. + """ + try: + verdict = session.verify_chain_detail() + except Exception as exc: + # The message is structural (a parse position, a rejected key name), + # never a ledger line: nothing here echoes recorded content. + return "unverifiable", None, f"the ledger could not be read: {exc}" + status = verdict.status + if status == "empty" and not session.log_path.exists(): + status = "absent" + return ( + status, + verdict.first_broken_index if status == "broken" else None, + verdict.reason, ) +def _verify_claim( + stored: dict, + claim: Claim, + manifest: Manifest, + entries: list, + events: list, + repo: Path, + ledger_objects: Optional[Path], +) -> GradeResult: + """Grade one sealed claim, checking the note's binding to its evidence first. + + Two paths, and the split is what protects notes published before v0.2: + + - **No evidence block** — a v1 note. Regrade exactly as v0.1 regraded it, + unbound. There is nothing to check: the note never recorded which entry + backed the claim, and inventing a binding for it after the fact would be a + verdict about evidence this run cannot identify. + - **An evidence block** — check it before grading anything. If the live + ledger cannot supply that entry, report the SEALED grade under + ``witness-unavailable`` rather than recomputing a fresh verdict from + whatever the ledger holds now. That silent recomputation is the defect + this whole unit exists to close. + """ + evidence = stored.get("evidence") + if not isinstance(evidence, dict): + return _regrade(claim, manifest, events, repo, ledger_objects) + + sealed_grade = stored.get("grade") + idx = evidence.get("event_index") + problem = _witness_problem(entries, idx, evidence.get("entry_hash")) + if problem is not None: + return GradeResult( + claim, + GRADE_WITNESS_UNAVAILABLE, + reason=( + f"sealed as {sealed_grade} at commit {manifest.commit[:12]}; " + f"witness ledger unavailable ({problem})" + ), + supporting_event_index=idx if _is_index(idx) else None, + sealed_grade=sealed_grade, + ) + + result = _regrade(claim, manifest, events, repo, ledger_objects) + result.evidence_bound = True + result.sealed_grade = sealed_grade + return result + + +def _is_index(value) -> bool: + """A real integer index. ``bool`` is an ``int`` subclass and is not one.""" + return isinstance(value, int) and not isinstance(value, bool) + + +def _witness_problem(entries: list, idx, recorded_hash) -> Optional[str]: + """Why the live ledger cannot supply the sealed evidence, or None. + + Two mechanisms, and they are not the same fact. The ledger may be too SHORT + to hold that index — the archived-ledger case, which an index-keyed regrade + already noticed, if only by grading everything unknown. Or it may be long + enough and hold a DIFFERENT entry there, which an index-keyed regrade + reports as a confident verdict about evidence it never saw. The second is + the dangerous one, so the reason names which happened. + + The comparison is against the entry's STORED hash. Whether that hash is the + one the entry's own body recomputes to is the chain check's question, not + this one; conflating them would report a ledger this binary merely cannot + recompute as evidence that went missing. + """ + if not _is_index(idx): + return "the sealed evidence block names no usable event index" + if not (0 <= idx < len(entries)): + return ( + f"recorded event index {idx} is out of range " + f"(the live ledger holds {len(entries)} event(s))" + ) + if not isinstance(recorded_hash, str) or not recorded_hash: + return f"the sealed evidence block carries no entry hash for index {idx}" + if entries[idx].entry_hash != recorded_hash: + return f"entry hash mismatch at index {idx}" + return None + + +def _regrade( + claim: Claim, + manifest: Manifest, + events: list, + repo: Path, + ledger_objects: Optional[Path], +) -> GradeResult: + try: + return grade(claim, manifest.tree, events, repo, ledger_objects) + except Exception as exc: # object gc'd or unreadable → unknown, not crash + return GradeResult(claim, "unknown", reason=f"regrade failed: {exc}") + + # --- claim persistence (declared claims live in the ledger dir) -------------- def _claims_path(session: Session) -> Path: @@ -296,18 +512,33 @@ def _last_seal_watermark(session: Session) -> int: def _record_seal(session: Session, commit: str, tree: str, claims_watermark: int) -> None: - with _seals_path(session).open("a", encoding="ascii") as fh: - fh.write( - canonical_json( - {"commit": commit, "tree": tree, "claims_watermark": claims_watermark} - ).decode("ascii") - + "\n" - ) + # Private-append: these carry operator-authored labels and commit ids into + # the secret-bearing ledger directory, and a plain open() would create them + # 0666 & ~umask. canonical_json is already ascii bytes, so the byte written + # is unchanged from the text-mode open this replaced. + # + # Under the ledger's append lock for the same reason as declare_claim below: + # a seal line carries a commit id and a watermark other seals read back, so a + # torn one loses the watermark and re-seals every claim. + with _append_lock(session.root): + with _open_private_append(_seals_path(session)) as fh: + fh.write( + canonical_json( + {"commit": commit, "tree": tree, "claims_watermark": claims_watermark} + ) + + b"\n" + ) def declare_claim(session: Session, claim: Claim) -> None: - with _claims_path(session).open("a", encoding="ascii") as fh: - fh.write(canonical_json(claim.to_dict()).decode("ascii") + "\n") + # O_APPEND is atomic only below PIPE_BUF, and a claim line is not bounded by + # that: every declared pathspec lengthens it. Two concurrent `didrun claim` + # invocations could therefore interleave into one unparseable record, so this + # takes the same ledger-wide lock the event append takes. Cheap by + # construction — one short local write, never a child process. + with _append_lock(session.root): + with _open_private_append(_claims_path(session)) as fh: + fh.write(canonical_json(claim.to_dict()) + b"\n") def _load_claims(session: Session) -> list[Claim]: @@ -495,15 +726,44 @@ def _scan_for_secrets(session: Session, results: list[GradeResult]) -> list[reda return findings -def _redact_result(r: GradeResult, session: Session) -> dict: +def _redact_result(r: GradeResult, session: Session, entries: list) -> dict: d = r.to_dict() # Redact argv in the exported claim view. d["claim"]["argv_preview"] = redact.scrub_argv( list(_event_argv(session, r.supporting_event_index)) ) + block = _evidence_block(entries, r.supporting_event_index) + if block is not None: + d["evidence"] = block return d +def _evidence_block(entries: list, idx: Optional[int]) -> Optional[dict]: + """Which recorded entry backed this claim, named by its chain hash. + + This is the handle the note never had. A sealed grade used to be reachable + only through an integer index into whatever ledger happens to be on disk at + verify time, so an archived, rebuilt or substituted ledger changed the + verdict with no signal. The entry hash makes "index 4 of the ledger you have + now" checkable against "index 4 of the ledger this was sealed from". + + Recorded in the MANIFEST, never in the ledger: the v1 chain preimage is + frozen (docs/COMPAT.md), and putting a field into Event would invalidate + every entry hash ever written. Omitted entirely when no event backs the + claim — an absent block means "nothing to bind", which is not the same fact + as "bound to nothing". + """ + if idx is None or not (0 <= idx < len(entries)): + return None + entry = entries[idx] + return { + "event_index": idx, + "entry_hash": entry.entry_hash, + "env_fingerprint": entry.event.env_fingerprint, + "tree_after": entry.event.tree_after, + } + + def _event_argv(session: Session, idx: Optional[int]): if idx is None: return () diff --git a/src/didrun/render.py b/src/didrun/render.py index 50501ce..3e90a4a 100644 --- a/src/didrun/render.py +++ b/src/didrun/render.py @@ -24,16 +24,45 @@ # The token NEVER overclaims: the strongest positive says TREE-EXACT ("the # evidence tree equals the sealed tree"), never "VERIFIED"/"PROVEN" — didrun # records, it does not prove. The marker carries severity when color is absent. +# +# EVERY grade must be registered in BOTH tables. Both are read through +# `.get(…, UNKNOWN)`, so an unregistered grade does not raise — it renders as +# UNKNOWN, which is a silent lie about what the verdict actually said. +# +# `chain-broken` is the one row here that is not a claim grade: no GradeResult +# ever carries it, so no table row ever displays it. It is a whole-report +# verdict — `VerifyReport.worst_status` returns it when the live ledger's chain +# does not recompute — and `render_verdict` looks the VERDICT up in this same +# table, so without a row the worst thing didrun can say about a ledger would +# print as UNKNOWN. _GRADE_DISPLAY = { + "chain-broken": ("CHAIN-BROKEN", "31", "x", "the ledger chain does not recompute"), "failed": ("FAILED", "31", "x", "recorded command exited non-zero"), "unknown": ("UNKNOWN", "33", "?", "no honest binding"), + "witness-unavailable": ( + "WITNESS-UNAVAIL", + "33", + "?", + "sealed grade shown; live evidence not found", + ), "stale": ("STALE", "33", "!", "tree moved since evidence"), "scope-exact": ("SCOPE-EXACT", "36", "~", "change within declared scope"), "tree-exact": ("TREE-EXACT", "32", "=", "recorded against the sealed tree"), } # Sort order: worst first. A reviewer sees problems before recorded-exact rows. -_SORT_RANK = {"failed": 0, "unknown": 1, "stale": 2, "scope-exact": 3, "tree-exact": 4} +# `chain-broken` sorts above `failed` for completeness only — it is a report-level +# verdict, never a per-claim grade, so nothing is ever sorted by this rank. Do not +# go looking for the rows; there are none. +_SORT_RANK = { + "chain-broken": -1, + "failed": 0, + "unknown": 1, + "witness-unavailable": 2, + "stale": 3, + "scope-exact": 4, + "tree-exact": 5, +} def _use_color() -> bool: @@ -75,14 +104,48 @@ def _skipped_notes(n: int) -> str: return f"{n} note{'' if n == 1 else 's'} skipped (unparseable)" +def chain_banner_text(report) -> str: + """The chain sentence a reviewer must not miss, or "" when there is none. + + Printed for `broken` and `unverifiable` only. `absent` and `empty` are NOT + chain faults — a ledger that is elsewhere, or has recorded nothing yet, is a + witness-availability fact that the per-claim rows already state as + `witness-unavailable`. Rendering a missing ledger as tamper would make every + verify against an archived ledger read like an attack. + + `broken` and `unverifiable` get different words on purpose: broken means an + entry did not recompute to its stored hash, unverifiable means this binary + could not recompute it at all. Only the first is evidence of mutation. + """ + status = report.chain_status + if status == "broken": + where = "" if report.chain_broken_index is None else f" at index {report.chain_broken_index}" + return f"ledger chain BROKEN{where} — every grade below is unreliable" + if status == "unverifiable": + reason = _sanitize(report.chain_reason) or "this binary cannot recompute it" + return f"ledger chain unverifiable — {reason}; every grade below is unreliable" + return "" + + +def render_chain_banner(report) -> str: + """The terminal form of the chain sentence: marker, colour, "" when clean.""" + text = chain_banner_text(report) + return _c(f"x {text}", "31") if text else "" + + def render_verdict(report, width: int = 80) -> str: """Render a VerifyReport as the CLI instrument panel.""" lines: list[str] = [] results = sorted(report.results, key=lambda r: _SORT_RANK.get(r.grade, 9)) + banner = render_chain_banner(report) worst = report.worst_status if report.results else "empty" if worst == "empty": lines.append(_c("○ NO CLAIMS", "33") + " nothing sealed for this commit") + # A broken chain is worth saying even with nothing sealed here: the + # ledger is still the thing every other command reads. + if banner: + lines.append(banner) lines.append(f" commit {report.commit[:12] or '(none)'} · resolved-by {report.resolved_by}") if report.notes_skipped: lines.append(_c(f" ! {_skipped_notes(report.notes_skipped)}", "33")) @@ -96,11 +159,16 @@ def render_verdict(report, width: int = 80) -> str: # carries severity even with color stripped. headline = "ALL RECORDED-EXACT" if report.all_verified else f"{token} — review needed" lines.append(_c(f"{marker} {headline}", color)) + if banner: + lines.append(banner) lines.append( f" {verified}/{total} claims recorded-exact · " f"commit {report.commit[:12]} · tree {report.tree[:12]} · " f"resolved-by {report.resolved_by}" ) + # How many of those verdicts were checked against the recorded entry the + # seal named, rather than regraded against whatever ledger is on disk now. + lines.append(f" {report.evidence_bound_count}/{total} claims evidence-bound") if report.secrets_override: lines.append(_c(" ! sealed with --allow-secrets (redacted export)", "33")) if report.notes_skipped: @@ -171,8 +239,21 @@ def esc(s: str) -> str: f"" ) + # The chain sentence gets its own block above the table, not a table row: + # it is a statement about the ledger every row was read out of, so a row + # would file it alongside the very grades it invalidates. + chain_text = chain_banner_text(report) + chain_html = ( + f'
{esc(chain_text)}
' if chain_text else "" + ) + if all_ok: verdict_class, verdict_text = "ok", "ALL RECORDED-EXACT" + elif worst == "chain-broken": + verdict_class = "failed" + verdict_text = ( + f"{esc(_GRADE_DISPLAY['chain-broken'][0])} — the ledger chain does not recompute" + ) elif worst == "failed": verdict_class = "failed" verdict_text = f"{esc(_GRADE_DISPLAY['failed'][0])} — a claimed command was recorded failing" @@ -225,6 +306,10 @@ def esc(s: str) -> str: .verdict.ok {{ color:var(--ok); }} .verdict.attention {{ color:var(--warn); }} .verdict.failed {{ color:var(--bad); }} + /* Filled, not tinted: a chain fault invalidates every row under it, so it is + the one block that must survive a fast scan and a monochrome printout. */ + .chainbanner {{ margin:8px 0; padding:8px 10px; border-radius:3px; + background:var(--bad-fill); color:var(--on-fill); font-weight:700; }} .meta {{ color:var(--dim); font-size:12px; word-break:break-all; }} table {{ width:100%; border-collapse:collapse; margin-top:8px; table-layout:fixed; }} th {{ text-align:left; color:var(--dim); font-weight:500; font-size:11px; @@ -239,16 +324,18 @@ def esc(s: str) -> str: font-weight:700; letter-spacing:.03em; border:1px solid currentColor; white-space:nowrap; }} .badge.tree_exact {{ color:var(--ok); }} .badge.scope_exact {{ color:var(--scope); }} - .badge.stale, .badge.unknown {{ color:var(--warn); }} + .badge.stale, .badge.unknown, .badge.witness_unavailable {{ color:var(--warn); }} /* Failure and stale are the MOST-designed states: solid-filled badges + a left accent bar so a problem row is unmistakable on a fast scan. */ .badge.failed {{ color:var(--on-fill); background:var(--bad-fill); border-color:var(--bad-fill); }} .badge.stale {{ color:var(--on-fill); background:var(--warn-fill); border-color:var(--warn-fill); }} - tr.stale td, tr.unknown td {{ background:color-mix(in srgb, var(--warn) 10%, transparent); }} + tr.stale td, tr.unknown td, tr.witness_unavailable td {{ + background:color-mix(in srgb, var(--warn) 10%, transparent); }} tr.failed td {{ background:color-mix(in srgb, var(--bad) 12%, transparent); }} tr.failed td.status {{ box-shadow:inset 3px 0 0 var(--bad); }} tr.stale td.status {{ box-shadow:inset 3px 0 0 var(--warn); }} tr.unknown td.status {{ box-shadow:inset 3px 0 0 var(--warn); }} + tr.witness_unavailable td.status {{ box-shadow:inset 3px 0 0 var(--warn); }} .detail {{ color:var(--dim); }} .detail .reason {{ display:block; }} .exit {{ display:inline-block; margin-top:2px; color:var(--mono); font-size:12px; }} @@ -263,7 +350,11 @@ def esc(s: str) -> str: .badge {{ padding:2px 5px; font-size:10px; }} }} @media print {{ body {{ background:#fff; color:#000; }} .wrap {{ max-width:none; }} - tr.failed td, tr.stale td, tr.unknown td {{ background:transparent; }} }} + tr.failed td, tr.stale td, tr.unknown td, tr.witness_unavailable td {{ + background:transparent; }} + /* Browsers drop backgrounds when printing, which would leave the chain + sentence white-on-white — the one line that must never disappear. */ + .chainbanner {{ background:transparent; color:#000; border:2px solid #000; }} }} @@ -271,7 +362,9 @@ def esc(s: str) -> str:
didrun · flight record
{verdict_text}
+ {chain_html}
{esc(verified)}/{esc(total)} claims recorded-exact · + {esc(report.evidence_bound_count)}/{esc(total)} claims evidence-bound · commit {esc(report.commit[:12])} · tree {esc(report.tree[:12])} · resolved-by {esc(report.resolved_by)} · coverage {coverage_html}
diff --git a/tests/compat/test_corpus_replay.py b/tests/compat/test_corpus_replay.py index 15e9413..6421f23 100644 --- a/tests/compat/test_corpus_replay.py +++ b/tests/compat/test_corpus_replay.py @@ -297,7 +297,12 @@ def test_leg1_chain_recompute(replay_source): # lack, because `from_json` reads them with a v1-reproducing default. Hard # coded per manifest version so a new manifest field is a deliberate one-line # diff here and never a silent pass. -MANIFEST_ADDITIVE_KEYS = {1: frozenset({"secrets_override"})} +# v2 introduced no top-level manifest key — the evidence binding lives inside +# each claim entry — so its allowlist is v1's. +MANIFEST_ADDITIVE_KEYS = { + 1: frozenset({"secrets_override"}), + 2: frozenset({"secrets_override"}), +} def note_violations(raw: bytes, ordinal: int, additive_keys=None) -> list: diff --git a/tests/test_append_concurrency.py b/tests/test_append_concurrency.py new file mode 100644 index 0000000..417ceb3 --- /dev/null +++ b/tests/test_append_concurrency.py @@ -0,0 +1,610 @@ +"""P2.4 — the read-tail-and-append pair is atomic between didrun processes. + +The hazard being closed is corruption, not wall-clock. ``Session.append`` reads +the tail, derives ``(index, prev_hash)`` from it, then writes; two processes +that interleave those three steps both write the same index chained from the +same predecessor, and ``verify_chain`` reports a break that cannot be repaired +without rewriting the log — the one edit an evidence store must never make. + +Two things make the positive leg here mean something. + + - **The negative leg runs the identical harness.** ``test_1`` and ``test_2`` + drive the same child script with the same contention profile; the only + difference between them is whether ``ledger._append_lock`` is the real + context manager or ``contextlib.nullcontext``. A green positive leg that + could not fail is worth nothing, so the negative leg proves the harness + races. + - **The lock's extent is asserted, not assumed.** Test 4 pins that it is not + held across the wrapped child (the failure mode that would trade a rare + recoverable break for a routine hang), and test 3 pins that a read-only + verify never waits on it. + +Honest scope: the deployment class this protects — concurrent didrun +invocations against one ledger — has zero observed instances in the corpus this +work was measured against. The two preserved incidents behind it come from the +evidence documents, not from a race reproduced here. +""" + +from __future__ import annotations + +import contextlib +import os +import shutil +import stat +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from didrun import gitplumbing as gp +from didrun import ledger as ledger_mod +from didrun import manifest as manifest_mod +from didrun.claims import Claim +from didrun.ledger import Event, Session + +# flock is the whole mechanism. Off POSIX the manager is a declared no-op and +# there is nothing here to assert but the degradation (test 11). +fcntl = ledger_mod.fcntl + +ECHO = "/bin/echo" +SH = "/bin/sh" + +pytestmark = pytest.mark.skipif( + not ledger_mod.APPEND_LOCK_AVAILABLE + or not (os.path.exists(ECHO) and os.path.exists(SH)), + reason="POSIX flock semantics and a POSIX /bin layout", +) + +# How many writers race for one ledger. Eight is the figure the design was +# specified against and is enough to make a fork overwhelming rather than +# marginal when the lock is removed. +RACERS = 8 + +# Seconds each racer spends between reading the tail and writing the line, +# injected identically into both legs. See _race() for why this is here and why +# it does not weaken the positive leg. +WIDEN_SECONDS = 0.05 + + +# The racer. It drives the REAL CLI in a real process — the corruption this +# guards against is between processes, so an in-process thread test would not +# reach it. Everything it does is switched by environment variable so the +# locked and unlocked legs run byte-identical code. +_RACER = ''' +import contextlib +import os +import sys +import time + +import didrun.ledger as ledger +from didrun.ledger import Session + +if os.environ.get("DIDRUN_TEST_NO_LOCK") == "1": + # The negative leg. contextlib.nullcontext accepts and ignores the root, so + # this is exactly Session.append with no mutual exclusion: v0.1's behaviour. + ledger._append_lock = contextlib.nullcontext + +WIDEN = float(os.environ.get("DIDRUN_TEST_WIDEN") or 0.0) +RENDEZVOUS = os.environ.get("DIDRUN_TEST_RENDEZVOUS") or "" +PEERS = int(os.environ.get("DIDRUN_TEST_PEERS") or 0) + + +def _rendezvous(path, peers, timeout=120.0): + """Block until `peers` processes have arrived. Single-byte O_APPEND writes.""" + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600) + try: + os.write(fd, b"x") + finally: + os.close(fd) + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + if os.path.getsize(path) >= peers: + return + except OSError: + pass + time.sleep(0.005) + raise SystemExit("rendezvous timed out") + + +_real_last = Session._last +_real_append = Session.append + + +def _last(self): + out = _real_last(self) + if WIDEN: + # Between reading the tail and writing the line. Under the lock this is + # inside the critical section and merely slows the run down; without it + # this is the window every racer reads the same tail through. + time.sleep(WIDEN) + return out + + +def _append(self, event): + if RENDEZVOUS: + # BEFORE the lock, never inside it: a barrier inside the critical + # section would deadlock the locked leg by construction. + _rendezvous(RENDEZVOUS, PEERS) + return _real_append(self, event) + + +Session._last = _last +Session.append = _append + +from didrun.cli import main + +sys.exit(main(sys.argv[1:])) +''' + +_CLI_BOOT = "import sys; from didrun.cli import main; sys.exit(main(sys.argv[1:]))" + +# Acquires the ledger's append lock, publishes readiness, then blocks forever. +# Used to prove the lock is really held (test 3, test 6) and that the kernel +# releases it when the holder is killed (test 6). +_HOLDER = ''' +import os +import sys +import time + +from didrun.ledger import _append_lock + +root, ready = sys.argv[1], sys.argv[2] +with _append_lock(root): + tmp = ready + ".tmp" + open(tmp, "w").write(str(os.getpid())) + os.replace(tmp, ready) + time.sleep(600) +''' + + +def _wait_for(predicate, timeout: float, interval: float = 0.01) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(interval) + return predicate() + + +def _lock_is_held(root: Path) -> bool: + """True when some open file description holds the ledger's exclusive flock. + + A real probe even from the process that holds the lock: flock conflicts are + per open-file-description, so a second os.open in this process is refused + the same way another process would be. Test 7 pins that this probe + discriminates, so a False here is a measurement and not a tautology. + """ + path = Path(root) / ledger_mod.LOCK_FILENAME + fd = os.open(str(path), os.O_RDWR | os.O_CREAT, 0o600) + try: + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + return True + fcntl.flock(fd, fcntl.LOCK_UN) + return False + finally: + os.close(fd) + + +def _wrapper_event(argv=("echo", "hi"), exit_code=0, **kw) -> Event: + return Event( + argv=tuple(argv), + cwd="/tmp", + env_fingerprint="abc123", + observed_via="wrapper", + coverage="complete", + exit_code=exit_code, + **kw, + ) + + +def _read_session(repo: Path) -> Session: + """A read-only view of the ledger — the reader must never create anything.""" + return Session(repo / ".didrun", readonly=True) + + +def _race(repo: Path, tmp_path: Path, tag: str, *, no_lock: bool): + """Run RACERS concurrent `didrun run` invocations against one ledger. + + Returns ``(returncodes, verdict, indices)`` read back through a read-only + session. The contention profile is IDENTICAL in both legs — a start + rendezvous immediately before ``append`` plus WIDEN_SECONDS between the tail + read and the write — so the only variable across test 1 and test 2 is the + lock itself. + """ + script = tmp_path / "racer.py" + script.write_text(_RACER, encoding="utf-8") + env = dict(os.environ) + env["DIDRUN_TEST_WIDEN"] = str(WIDEN_SECONDS) + env["DIDRUN_TEST_RENDEZVOUS"] = str(tmp_path / f"rendezvous-{tag}") + env["DIDRUN_TEST_PEERS"] = str(RACERS) + if no_lock: + env["DIDRUN_TEST_NO_LOCK"] = "1" + procs = [ + subprocess.Popen( + [ + sys.executable, + str(script), + "--repo", + str(repo), + "run", + "--", + ECHO, + str(i), + ], + cwd=str(repo), + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + for i in range(RACERS) + ] + codes = [] + for proc in procs: + try: + proc.communicate(timeout=180) + except subprocess.TimeoutExpired: + proc.kill() + proc.communicate(timeout=30) + pytest.fail("a racer never finished — the lock is wider than the append") + codes.append(proc.returncode) + session = _read_session(repo) + verdict = session.verify_chain_detail() + indices = [e.index for e in session.entries()] + return codes, verdict, indices + + +# --- 1. the race, with the lock ---------------------------------------------- + + +def test_eight_concurrent_runs_leave_one_intact_chain(repo: Path, tmp_path: Path): + codes, verdict, indices = _race(repo, tmp_path, "locked", no_lock=False) + + assert codes == [0] * RACERS + assert verdict.status == "intact", verdict.reason + assert len(indices) == RACERS + assert set(indices) == set(range(RACERS)) + # Sequence, not just membership: a fork that happened to land on a complete + # set would still be out of order on disk. + assert indices == list(range(RACERS)) + + +# --- 2. the negative leg: the same harness, no lock -------------------------- + + +def test_the_same_race_forks_the_chain_without_the_lock(repo: Path, tmp_path: Path): + """The control. Without the lock this harness must actually corrupt. + + **What the harness had to do to make this fail** — recorded because a + negative leg that needed help is only honest if the help is named, and + because both helpers are applied to the LOCKED leg too: + + 1. A start rendezvous immediately before ``Session.append``, outside the + lock. Eight ``didrun run`` invocations otherwise finish their tree + digests at scattered times and mostly do not overlap at all; the + barrier makes them arrive together, which is the situation the lock + exists for and not one it gets to dodge. + 2. ``WIDEN_SECONDS`` of sleep between the tail read and the write. The + unlocked window is a few hundred microseconds of local I/O, so without + widening the outcome is a coin toss per attempt rather than a + demonstration. + + Neither weakens test 1: the identical profile runs there, and the lock has + to hold the chain together through it. + + Measured on the machine this was written on: 3 of 3 unlocked attempts forked + (duplicate indices, chain reported broken at index 0). The assertion stays + at "at least one" so a slower or busier machine cannot turn a real + demonstration into a red suite. + """ + attempts = 3 + forks = 0 + for attempt in range(attempts): + shutil.rmtree(repo / ".didrun", ignore_errors=True) + _codes, verdict, indices = _race( + repo, tmp_path, f"unlocked-{attempt}", no_lock=True + ) + duplicated = len(indices) != len(set(indices)) + if duplicated or verdict.status != "intact": + forks += 1 + + assert forks > 0, ( + "the unlocked race never forked the chain, so test 1 proves nothing — " + "raise RACERS or WIDEN_SECONDS until it does" + ) + + +# --- 3. the read path takes no lock ------------------------------------------ + + +def test_a_read_only_verify_does_not_wait_on_the_append_lock( + repo: Path, tmp_path: Path, git +): + """A held write lock must not stall a reader. Verification is the hot path + in CI and a reader that contends turns a corruption fix into an outage.""" + subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "run", "--", ECHO, "hi"], + cwd=str(repo), + capture_output=True, + check=True, + ) + subprocess.run( + [ + sys.executable, + "-c", + _CLI_BOOT, + "--repo", + str(repo), + "claim", + "tests-pass", + "--label", + "reader", + ], + cwd=str(repo), + capture_output=True, + check=True, + ) + git(repo, "commit", "-qm", "seal point", "--allow-empty") + subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "seal"], + cwd=str(repo), + capture_output=True, + check=True, + ) + + root = repo / ".didrun" + with ledger_mod._append_lock(root): + assert _lock_is_held(root), "the fixture did not actually hold the lock" + # No timeout= fallback: a reader that needs one has already failed. + proc = subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "verify"], + cwd=str(repo), + capture_output=True, + timeout=60, + ) + assert proc.returncode in (0, 1), proc.stderr.decode() + show = subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "show", "--session"], + cwd=str(repo), + capture_output=True, + timeout=60, + ) + assert show.returncode == 0, show.stderr.decode() + + +# --- 4. the lock is never held across the wrapped child ---------------------- + + +def test_the_lock_is_not_held_across_the_wrapped_child(repo: Path, tmp_path: Path): + """The failure this prompt was specified to avoid. + + Acquiring for the duration of `cli.main` would be trivially correct and + would make a second `didrun run` block for as long as the first command + takes — hours, in the workload this tool was built for. + """ + marker = tmp_path / "child-running" + slow = subprocess.Popen( + [ + sys.executable, + "-c", + _CLI_BOOT, + "--repo", + str(repo), + "run", + "--", + SH, + "-c", + f"touch {marker}; sleep 5", + ], + cwd=str(repo), + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + try: + assert _wait_for(marker.exists, 60), "the wrapped child never started" + started = time.monotonic() + quick = subprocess.run( + [ + sys.executable, + "-c", + _CLI_BOOT, + "--repo", + str(repo), + "run", + "--", + ECHO, + "x", + ], + cwd=str(repo), + capture_output=True, + timeout=60, + ) + elapsed = time.monotonic() - started + assert quick.returncode == 0, quick.stderr.decode() + # The wrapped child sleeps 5s and has barely started. Anything near that + # means the lock covered it. + assert elapsed < 3.0, f"the second run took {elapsed:.2f}s" + finally: + slow.kill() + slow.communicate(timeout=30) + + +# --- 5. lock file hygiene ---------------------------------------------------- + + +def test_lock_file_is_private_ignored_and_outside_every_tree_digest( + repo: Path, git +): + root = repo / ".didrun" + session = Session(root) + before = gp.tree_digest(repo, ledger_objects=session.blobs.root) + assert before is not None + + with ledger_mod._append_lock(root): + pass + + lock = root / ledger_mod.LOCK_FILENAME + assert lock.exists() + assert stat.S_IMODE(lock.stat().st_mode) == 0o600 + + # Ignored by the ledger's own "*" rule, so no `git add -A` can stage it. + ignored = subprocess.run( + ["git", "check-ignore", "-v", str(lock.relative_to(repo))], + cwd=str(repo), + capture_output=True, + text=True, + ) + assert ignored.returncode == 0, ignored.stdout + ignored.stderr + + # And excluded from the digest unconditionally, not merely by being ignored. + after = gp.tree_digest(repo, ledger_objects=session.blobs.root) + assert after == before + listing = subprocess.run( + ["git", "ls-tree", "-r", "--name-only", after], + cwd=str(repo), + capture_output=True, + text=True, + env={ + **os.environ, + "GIT_OBJECT_DIRECTORY": str(session.blobs.root), + "GIT_ALTERNATE_OBJECT_DIRECTORIES": str(gp.objects_dir(repo)), + }, + ) + assert listing.returncode == 0, listing.stderr + assert ".didrun" not in listing.stdout + + +# --- 6. a killed holder leaves no stale lock --------------------------------- + + +def test_a_sigkilled_holder_leaves_no_stale_lock(repo: Path, tmp_path: Path): + """flock is the reason this test is short. A PID file or an O_EXCL sentinel + would need liveness checks and a recovery path; the kernel drops an flock + when the holder dies, however it dies.""" + root = repo / ".didrun" + session = Session(root) + script = tmp_path / "holder.py" + script.write_text(_HOLDER, encoding="utf-8") + ready = tmp_path / "holder.ready" + + holder = subprocess.Popen( + [sys.executable, str(script), str(root), str(ready)], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + try: + assert _wait_for(ready.exists, 60), "the holder never acquired the lock" + assert _lock_is_held(root), "the holder claimed the lock but does not hold it" + finally: + holder.kill() + holder.communicate(timeout=30) + + assert _wait_for(lambda: not _lock_is_held(root), 10), "the lock outlived its holder" + started = time.monotonic() + entry = session.append(_wrapper_event()) + assert time.monotonic() - started < 5.0 + assert entry.index == 0 + + +# --- 7. the probe discriminates (so tests 3 and 6 are measurements) ---------- + + +def test_the_lock_probe_reports_false_when_nothing_holds_the_lock(repo: Path): + root = repo / ".didrun" + Session(root) + assert _lock_is_held(root) is False + with ledger_mod._append_lock(root): + assert _lock_is_held(root) is True + assert _lock_is_held(root) is False + + +# --- 8. a read-only session never touches the lock file ---------------------- + + +def test_a_read_only_session_never_creates_the_lock_file(tmp_path: Path): + root = tmp_path / ".didrun" + session = Session(root, readonly=True) + with pytest.raises(ledger_mod.LedgerError): + session.append(_wrapper_event()) + assert not root.exists() + assert not (root / ledger_mod.LOCK_FILENAME).exists() + + +# --- 9 + 10. the claim and seal appends take the same lock ------------------- + + +def _asserting_lock(recorded: list, real): + @contextlib.contextmanager + def probing(root): + with real(root): + recorded.append(_lock_is_held(Path(root))) + yield + + return probing + + +def test_declare_claim_holds_the_lock_while_it_writes(repo: Path, monkeypatch): + """A claims.jsonl line is not bounded by PIPE_BUF once a claim declares + pathspecs, so O_APPEND atomicity is not a guarantee here.""" + session = Session(repo / ".didrun") + session.append(_wrapper_event()) + held: list = [] + monkeypatch.setattr( + manifest_mod, "_append_lock", _asserting_lock(held, ledger_mod._append_lock) + ) + + manifest_mod.declare_claim( + session, + Claim( + ctype="tests-pass", + label="locked", + event_indices=(0,), + pathspecs=("src/", "tests/"), + declared_at_index=0, + ), + ) + + assert held == [True] + assert len(manifest_mod._load_claims(session)) == 1 + assert _lock_is_held(session.root) is False + + +def test_record_seal_holds_the_lock_while_it_writes(repo: Path, monkeypatch): + session = Session(repo / ".didrun") + held: list = [] + monkeypatch.setattr( + manifest_mod, "_append_lock", _asserting_lock(held, ledger_mod._append_lock) + ) + + manifest_mod._record_seal(session, "c" * 40, "t" * 40, 1) + + assert held == [True] + assert manifest_mod._last_seal_watermark(session) == 1 + assert _lock_is_held(session.root) is False + + +# --- 11. the platform degradation is recorded, not asserted away ------------- + + +def test_without_fcntl_the_manager_is_a_declared_no_op(tmp_path: Path, monkeypatch): + """docs/COMPAT.md states that append serialization is POSIX-only. That + sentence has to be backed by behaviour: no invented fallback, no lock file, + and appends that still work.""" + root = tmp_path / ".didrun" + monkeypatch.setattr(ledger_mod, "fcntl", None) + monkeypatch.setattr(ledger_mod, "APPEND_LOCK_AVAILABLE", False) + + session = Session(root) + assert session.append_serialized is False + with ledger_mod._append_lock(root): + pass + assert not (root / ledger_mod.LOCK_FILENAME).exists() + + entry = session.append(_wrapper_event()) + assert entry.index == 0 + assert not (root / ledger_mod.LOCK_FILENAME).exists() + assert session.verify_chain_detail().status == "intact" diff --git a/tests/test_chain_gate.py b/tests/test_chain_gate.py new file mode 100644 index 0000000..f889844 --- /dev/null +++ b/tests/test_chain_gate.py @@ -0,0 +1,395 @@ +"""P1.3 — `verify` consults the ledger hash chain and fails closed. + +The defect these pin: `verify_chain` had exactly one caller in the whole +package and it was `show --session`. `didrun verify --strict` — the command an +entire discipline gates on — never recomputed the chain, so a forked ledger +(duplicate indices) or a mid-file rewrite certified green and exited 0. + +What a chain check buys, stated as narrowly as it deserves: accident and drift +detection on the ledger the verdict was read out of. It is not a forger +barrier — whoever can rewrite a record can recompute every hash after it +(ledger.py's declared non-guarantee). A `broken` chain says an entry did not +recompute; it does not say who moved it. + +Two states that are deliberately NOT faults: `absent` (the ledger is elsewhere, +which P1.2 already reports per claim as witness-unavailable) and `empty`. +Treating either as tamper would make every verify against an archived ledger +read like an attack, which is the failure mode this file's tests 4 and 5 exist +to stop. +""" + +from __future__ import annotations + +import json +import shutil +import sys +from pathlib import Path + +from didrun import cli +from didrun import manifest as M +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import ChainVerdict, Session, canonical_json + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _sealed(repo: Path, events: int = 3) -> Session: + """`events` recorded successes, one claim over event 0, sealed clean. + + The claim binds event 0 on purpose: every corruption below lands on a LATER + record, so the claim's own evidence stays intact and gradeable. A test whose + claim went witness-unavailable would fail for the wrong reason and stop + testing the chain gate. + """ + s = _session(repo) + for i in range(events): + run_wrapped([sys.executable, "-c", f"print({i!r})"], s, repo) + M.declare_claim( + s, Claim(ctype="tests-pass", label="suite", event_indices=(0,), declared_at_index=0) + ) + M.seal(s, repo) + return s + + +def _records(s: Session) -> list[dict]: + return [ + json.loads(line) + for line in s.log_path.read_text(encoding="ascii").splitlines() + if line.strip() + ] + + +def _rewrite(s: Session, mutate) -> None: + """Round-trip the log through canonical JSON, mutating it via ``mutate``. + + The assertion is the load-bearing part: it proves the round-trip is + byte-stable on the records nobody touched, so a corruption test can never + pass merely because re-serializing the log moved a byte somewhere else. + """ + original = s.log_path.read_bytes() + recs = _records(s) + assert b"".join(canonical_json(r) + b"\n" for r in recs) == original, ( + "the canonical round-trip is not byte-stable; every corruption test in " + "this file would then be testing the round-trip, not the corruption" + ) + mutate(recs) + s.log_path.write_bytes(b"".join(canonical_json(r) + b"\n" for r in recs)) + + +def _flip(h: str) -> str: + return ("0" if h[0] != "0" else "1") + h[1:] + + +def _corrupt_middle_hash(s: Session) -> None: + """Flip a byte of the STORED hash of a middle record. + + Chosen over corrupting the event body so the events themselves are + untouched: every claim still grades exactly as it did, and the only thing + that changed in the whole report is the chain. That is what makes the + falsification test below a clean control. + """ + _rewrite(s, lambda recs: recs[1].__setitem__("entry_hash", _flip(recs[1]["entry_hash"]))) + + +# --- test 1: a broken chain fails --strict ----------------------------------- + + +def test_broken_chain_dominates_the_verdict_and_fails_strict(repo: Path, monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + _corrupt_middle_hash(s) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "broken" + assert report.chain_broken_index == 1 + assert report.worst_status == "chain-broken" + assert not report.all_verified + + out = render.render_verdict(report) + assert "ledger chain BROKEN at index 1" in out + assert "every grade below is unreliable" in out + assert "ALL RECORDED-EXACT" not in out + + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +def test_the_chain_check_is_the_only_thing_failing_that_report(repo: Path, monkeypatch): + """The falsification half of test 1, on the SAME corrupted fixture. + + Neutralize only the chain check and the identical corrupted ledger goes + green — which is the proof that the chain gate, and nothing else in the + report, is what test 1 detected. If this test ever fails, test 1 has started + passing for some other reason and is no longer evidence of anything. + """ + s = _sealed(repo) + _corrupt_middle_hash(s) + + monkeypatch.setattr( + Session, "verify_chain_detail", lambda self: ChainVerdict("intact") + ) + report = M.verify(_session(repo), repo) + assert report.chain_status == "intact" + assert report.all_verified, ( + "with the chain check neutralized this corrupted ledger must grade " + "green; if it does not, test 1's red came from somewhere else" + ) + assert report.worst_status == "tree-exact" + assert "ledger chain" not in render.render_verdict(report) + + +# --- test 2: a fork (duplicate index) ---------------------------------------- + + +def test_duplicate_index_reads_as_a_broken_chain(repo: Path, monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + # Re-append the last record verbatim: same index, same prev_hash, same + # stored hash. Every hash in the file still recomputes -- only the SEQUENCE + # is wrong, which is the shape a forked ledger actually has. + _rewrite(s, lambda recs: recs.append(dict(recs[-1]))) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "broken" + assert report.chain_broken_index == 2 + assert not report.all_verified + assert "ledger chain BROKEN at index 2" in render.render_verdict(report) + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +# --- test 3: unverifiable is not "intact", and is not "broken" either -------- + + +def test_unknown_preimage_version_is_unverifiable_not_broken(repo: Path, monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + _rewrite(s, lambda recs: recs[1].__setitem__("preimage_version", 2)) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "unverifiable" + assert report.chain_broken_index is None # nothing was shown not to recompute + assert report.worst_status == "chain-broken" + assert not report.all_verified + + # The verdict TOKEN is CHAIN-BROKEN for both fault states — one registered + # verdict, per the pack. The distinction lives in the banner, which is the + # line that says what actually happened, and it must not say BROKEN: a + # record this binary cannot recompute is not evidence of mutation, and + # saying so would accuse a future writer of tampering. + banner = render.chain_banner_text(report) + assert "unverifiable" in banner + assert "BROKEN" not in banner + assert "unknown chain preimage version: 2" in banner # the reason, named + + out = render.render_verdict(report) + assert banner in out + assert "ALL RECORDED-EXACT" not in out + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +# --- test 4: an absent ledger is not tamper ---------------------------------- + + +def test_absent_ledger_is_witness_unavailable_not_a_broken_chain( + repo: Path, tmp_path: Path, monkeypatch +): + monkeypatch.setenv("NO_COLOR", "1") + _sealed(repo) + shutil.move(str(repo / ".didrun"), str(tmp_path / "archived-ledger")) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "absent" + assert report.worst_status == "witness-unavailable" # the claims, not the chain + assert [r.grade for r in report.results] == ["witness-unavailable"] + assert not report.all_verified + + out = render.render_verdict(report) + assert "ledger chain" not in out, "a ledger that is elsewhere has not been tampered with" + assert "BROKEN" not in out + assert "witness ledger unavailable" in out + + # --strict still refuses, for the honest reason: nobody has seen the evidence. + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +def test_empty_log_is_not_a_chain_fault(repo: Path, monkeypatch): + """`empty` is a present-but-zero-length log, and it is not tamper either. + + Distinct from `absent` (no file at all) so the two stay tellable apart, and + excluded from the fault set for the same reason: nothing failed to + recompute. The claims still refuse --strict on their own. + """ + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + s.log_path.write_bytes(b"") + + report = M.verify(_session(repo), repo) + assert report.chain_status == "empty" + assert report.worst_status == "witness-unavailable" + assert not report.all_verified + assert "ledger chain" not in render.render_verdict(report) + + +def test_unreadable_log_refuses_instead_of_crashing_a_commit_with_no_note(repo: Path): + """An unreadable ledger must not start crashing verify on an unsealed commit. + + The chain check runs before the manifest is resolved, so it reads a ledger + on a path that previously never touched one. Nothing is sealed here: before + P1.3 this returned a clean "no claims" report, and it still must — as a + refusal that names the problem, not a traceback. + + The damage used here is corruption in the MIDDLE of the log. P2.3 gave the + reader a narrow tolerance for a torn FINAL record — a lost event, not + tamper, pinned by the test below — and that tolerance stops at the last + line on purpose: a hole with records after it is still unreadable, and + still a graded refusal rather than a crash. + """ + s = _session(repo) + for i in range(3): + run_wrapped([sys.executable, "-c", f"print({i!r})"], s, repo) + lines = s.log_path.read_bytes().splitlines(keepends=True) + assert len(lines) == 3 + s.log_path.write_bytes(lines[0] + lines[1][:40] + b"\n" + lines[2]) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "unverifiable" + assert report.results == [] + assert "could not be read" in report.chain_reason + assert "ledger chain unverifiable" in render.render_verdict(report) + + +def test_a_torn_final_record_is_a_lost_event_not_a_chain_fault(repo: Path): + """P2.3: the tail an interrupted append leaves must not read as tamper. + + Before P2.3 this was `unverifiable` — the log could not be read at all, so + the ledger's last word on an interrupted session was "this binary cannot + recompute me". It is now the honest thing: every link that survives does + recompute, the chain is `intact` over that prefix, and the truncation + travels in the reason. The fault this file exists to make dominant must NOT + fire here, because nothing failed to recompute. + """ + s = _session(repo) + for i in range(2): + run_wrapped([sys.executable, "-c", f"print({i!r})"], s, repo) + lines = s.log_path.read_bytes().splitlines(keepends=True) + assert len(lines) == 2 + # A partial final record: the shape a crash mid-append leaves behind. Cut at + # a byte offset, not a line boundary, or the log is merely SHORT and parses. + s.log_path.write_bytes(lines[0] + lines[1][:40]) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "intact" + assert report.results == [] + assert "torn at byte" in report.chain_reason + # Surfaced, but never as a chain fault: no banner, no tamper sentence. + assert "ledger chain" not in render.render_verdict(report) + + +# --- test 5: the happy path is untouched ------------------------------------- + + +def test_intact_chain_adds_nothing_to_the_verdict(repo: Path, monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + _sealed(repo) + + report = M.verify(_session(repo), repo) + assert report.chain_status == "intact" + assert report.chain_broken_index is None + assert report.worst_status == "tree-exact" + assert report.all_verified + + out = render.render_verdict(report) + assert "ALL RECORDED-EXACT" in out + assert "chain" not in out.lower(), "an intact chain is not news; it prints nothing" + # The class is always in the stylesheet; the block must not be in the body. + assert '
' not in render.render_html(report) + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_show_session_is_unchanged(repo: Path, capsys): + """`show --session` was the sole caller of verify_chain and keeps its shape.""" + s = _sealed(repo) + assert cli.main(["--repo", str(repo), "show", "--session"]) == 0 + assert "chain intact" in capsys.readouterr().out + + _corrupt_middle_hash(s) + assert cli.main(["--repo", str(repo), "show", "--session"]) == 1 + assert "BROKEN at index 1" in capsys.readouterr().out + + +# --- test 6: the render surfaces cannot silently drop it --------------------- + + +def test_chain_broken_is_registered_in_both_render_tables(): + """An unregistered verdict renders as UNKNOWN through `.get(…, UNKNOWN)`. + + That is the silent lie render.py's own comment warns about, and it would + turn the strongest statement didrun can make about a ledger into its + weakest. + """ + assert "chain-broken" in render._GRADE_DISPLAY + assert "chain-broken" in render._SORT_RANK + assert render._SORT_RANK["chain-broken"] < render._SORT_RANK["failed"] + assert render._GRADE_DISPLAY["chain-broken"][0] == "CHAIN-BROKEN" + + +def test_neither_surface_reads_as_passed_when_the_chain_is_broken( + repo: Path, monkeypatch +): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + _corrupt_middle_hash(s) + report = M.verify(_session(repo), repo) + + out = render.render_verdict(report) + assert "CHAIN-BROKEN" in out + assert "ALL RECORDED-EXACT" not in out + assert "UNKNOWN" not in out, "the verdict must not fall back to UNKNOWN" + for overclaim in ("VERIFIED", "PASSED", "PROVEN"): + assert overclaim not in out.upper() + + html_out = render.render_html(report) + assert "CHAIN-BROKEN" in html_out + assert '
' in html_out + assert "ledger chain BROKEN at index 1" in html_out + assert "ALL RECORDED-EXACT" not in html_out + + +def test_quiet_html_still_prints_the_chain_banner(repo: Path, tmp_path: Path, capsys, monkeypatch): + """--quiet suppresses the verdict, never the ledger's integrity.""" + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed(repo) + _corrupt_middle_hash(s) + out_html = tmp_path / "report.html" + + rc = cli.main( + ["--repo", str(repo), "verify", "--strict", "--html", str(out_html), "--quiet"] + ) + printed = capsys.readouterr().out + assert rc == 1 + assert "ledger chain BROKEN at index 1" in printed + assert "STATUS" not in printed, "--quiet still suppresses the per-claim table" + + +def test_no_claims_report_still_reports_a_broken_chain(repo: Path, monkeypatch): + """A commit with nothing sealed still gets the sentence. + + The ledger is what every other command reads; "nothing sealed here" is not a + reason to stay quiet about it not recomputing. + """ + monkeypatch.setenv("NO_COLOR", "1") + s = _session(repo) + for i in range(3): + run_wrapped([sys.executable, "-c", f"print({i!r})"], s, repo) + _corrupt_middle_hash(s) + + report = M.verify(_session(repo), repo) + assert report.results == [] + assert report.chain_status == "broken" + assert report.worst_status == "chain-broken" + out = render.render_verdict(report) + assert "NO CLAIMS" in out + assert "ledger chain BROKEN at index 1" in out diff --git a/tests/test_evidence_binding.py b/tests/test_evidence_binding.py new file mode 100644 index 0000000..9919285 --- /dev/null +++ b/tests/test_evidence_binding.py @@ -0,0 +1,406 @@ +"""P1.2 — a sealed claim is bound to the evidence it was sealed against. + +The defect these pin: `verify` re-graded every claim from scratch against +whatever ledger was on disk, keyed by integer index, and threw away the grades +it sealed. So an archived ledger made every claim `unknown` with the same reason +string a genuinely unbacked claim gets, and a SUBSTITUTED ledger of unrelated +exit-0 commands on the same tree graded a full set of claims recorded-exact and +exited 0. + +What this closes, stated honestly: verify used to silently regrade against +whatever ledger is present, so archival, rebuild or index drift changed a sealed +verdict with no signal. Binding the claim to its entry hash buys ACCIDENT AND +DRIFT DETECTION on a published note. It does not stop a forger — whoever can +substitute a ledger can regenerate its chain and re-run claim+seal to mint fresh +hash-bound claims (ledger.py's declared non-guarantee). +""" + +from __future__ import annotations + +import json +import shutil +import subprocess +import sys +from pathlib import Path + +from didrun import cli +from didrun import gitplumbing as gp +from didrun import manifest as M +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import Claim, GradeResult +from didrun.ledger import Session, canonical_json + + +def _git(repo: Path, *a: str) -> None: + subprocess.run(["git", *a], cwd=str(repo), capture_output=True, text=True, check=True) + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _ok(repo: Path, session: Session, text: str): + return run_wrapped([sys.executable, "-c", f"print({text!r})"], session, repo) + + +def _sealed_green(repo: Path, label: str = "suite") -> Session: + """One recorded success, one claim over it, sealed on a clean tree.""" + s = _session(repo) + _ok(repo, s, "the unit under test") + M.declare_claim( + s, Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0) + ) + M.seal(s, repo) + return s + + +def _substitute_log(repo: Path, session: Session, tmp_path: Path, texts) -> None: + """Replace session.log with a ledger of DIFFERENT commands on the same tree. + + Built by running real commands into a fresh Session, never by splicing log + lines: a spliced log has broken prev_hash linkage, so it reports a broken + chain, which dominates and fails --strict regardless of whether the evidence + binding works at all. The test would then pass for the wrong reason and stop + testing this unit. + + The substitute ledger lives OUTSIDE the repo. A ledger directory inside it + under any name but `.didrun` would enter the tree digest, moving tree_after + away from the sealed tree — and the whole point of this fixture is a + substitute whose events still look tree-exact. + """ + alt = Session(tmp_path / "alt-ledger") + for text in texts: + _ok(repo, alt, text) + shutil.copyfile(alt.log_path, session.log_path) + + +# --- test 1: the falsification test (criterion per red-team §3.3) ------------ + + +def test_substituted_ledger_of_the_same_length_is_caught(repo: Path, tmp_path: Path): + """A same-length ledger of different commands on the same tree. + + This fails on pre-P1.2 code, which grades it `tree-exact` and exits 0 — the + substitute's events are self-stable against the sealed tree, which is all + the index-keyed regrade ever looked at. It would ALSO fail with only the + witness-unavailable half: `supporting_event_index >= len(events)` is false + here, because the substitute ledger is exactly as long as the one it + replaced. Only the hash binding detects it. + """ + s = _sealed_green(repo) + sealed_tree = gp.commit_tree(repo, "HEAD") + _substitute_log(repo, s, tmp_path, ["something else entirely"]) + + live = list(s.entries()) + assert len(live) == 1, "the substitute must be the SAME LENGTH, or test 2 covers it" + assert live[0].event.tree_after == sealed_tree, ( + "the substitute must still look tree-exact, or the tree check catches it " + "and the hash binding is never exercised" + ) + assert s.verify_chain_detail().status == "intact", ( + "a spliced log would report a broken chain and dominate the verdict; " + "evidence binding must be the only thing detecting this" + ) + + report = M.verify(s, repo) + assert [r.grade for r in report.results] == ["witness-unavailable"] + assert "hash mismatch" in report.results[0].reason + assert report.results[0].sealed_grade == "tree-exact" + assert not report.all_verified # → --strict exits non-zero + assert report.evidence_bound_count == 0 + assert "hash mismatch" in render.render_verdict(report) + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +# --- test 2: a ledger that is merely short ------------------------------------ + + +def test_truncated_ledger_names_the_out_of_range_index(repo: Path, tmp_path: Path): + s = _session(repo) + _ok(repo, s, "first") + _ok(repo, s, "second") + M.declare_claim( + s, Claim(ctype="tests-pass", label="late", event_indices=(1,), declared_at_index=1) + ) + M.seal(s, repo) + + lines = s.log_path.read_bytes().splitlines(keepends=True) + assert len(lines) == 2 + s.log_path.write_bytes(lines[0]) # drop the backing entry + + report = M.verify(s, repo) + assert [r.grade for r in report.results] == ["witness-unavailable"] + reason = report.results[0].reason + assert "index 1" in reason and "out of range" in reason + assert "1 event(s)" in reason + assert not report.all_verified + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +# --- test 3: the archived ledger ---------------------------------------------- + + +# The exact sentence a genuinely unbacked claim gets. An archived ledger used to +# produce this byte-for-byte, which made "the evidence is elsewhere" and "there +# never was any evidence" indistinguishable in the report. +_UNBACKED_REASON = "no witnessed successful command backs this claim" + + +def test_archived_ledger_reports_the_sealed_grade_not_a_bare_unknown( + repo: Path, tmp_path: Path +): + s = _sealed_green(repo) + shutil.move(str(repo / ".didrun"), str(tmp_path / "archived-ledger")) + + report = M.verify(_session(repo), repo) + assert [r.grade for r in report.results] == ["witness-unavailable"] + reason = report.results[0].reason + assert "sealed as tree-exact" in reason + assert _UNBACKED_REASON not in reason + assert report.results[0].sealed_grade == "tree-exact" + assert not report.all_verified + assert report.evidence_bound_count == 0 + + +# --- test 4: a genuinely unbacked claim is still `unknown` -------------------- + + +def test_unbacked_claim_stays_unknown_with_the_original_reason(repo: Path): + """The two states must remain distinguishable — that is the whole point. + + A claim bound to an index that never had an event has nothing to bind to, so + the seal writes no evidence block and verify says what it has always said. + """ + s = _session(repo) + _ok(repo, s, "unrelated") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="phantom", event_indices=(7,), declared_at_index=0), + ) + M.seal(s, repo) + + body = json.loads(_note_bytes(repo)) + assert "evidence" not in body["claims"][0], "nothing backed it; nothing to bind" + + report = M.verify(s, repo) + assert [r.grade for r in report.results] == ["unknown"] + assert report.results[0].reason == _UNBACKED_REASON + assert report.evidence_bound_count == 0 + + +# --- test 5: a legacy v1 note verifies exactly as it did before --------------- + +# Generated by running this fixture against the code as it stood BEFORE P1.2 +# (see the unit's report), then pasted in. Hard-coded on purpose: computing them +# from the code under test would make "identical to pre-change behaviour" a +# claim this test cannot actually make. +_LEGACY_EXPECTED = [ + ("fresh", "tree-exact", "self-stable command ran against the sealed tree", []), + ("drifted", "stale", "tree moved since evidence: 1 path(s) differ", [("M", "calc.py")]), + ("unbacked", "unknown", "no witnessed successful command backs this claim", []), + ("caught", "failed", "the recorded command exited 1 — claim is not backed", []), +] + + +def _v1_claim_entry(ctype, label, indices, declared_at, grade, supporting, exit_code): + """A claim entry in the shape a v1 note carried — no `evidence` key.""" + return { + "claim": { + "ctype": ctype, + "label": label, + "event_indices": list(indices), + "pathspecs": [], + "declared_at_index": declared_at, + "argv_preview": [], + }, + "grade": grade, + "reason": "sealed", + "delta": [], + "supporting_event_index": supporting, + "exit_code": exit_code, + } + + +def _attach_raw_note(repo: Path, commit: str, body: bytes) -> None: + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-F", "-", commit], + cwd=str(repo), + input=body, + capture_output=True, + check=True, + ) + + +def _note_bytes(repo: Path, commit: str = "HEAD") -> bytes: + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + check=True, + ) + return proc.stdout + + +def test_legacy_v1_note_verifies_byte_for_byte_as_before(repo: Path): + """The corpus-protection test: 65 published notes are all version 1. + + Every one of them carries no evidence block, so every one of them must take + the unbound path and produce the grades, reasons and deltas it produced + before this unit existed — down to the string. + """ + s = _session(repo) + _ok(repo, s, "1") # event 0 @T0 + run_wrapped([sys.executable, "-c", "import sys; sys.exit(1)"], s, repo) # event 1 @T0 + (repo / "calc.py").write_text("def add(a, b):\n return a + b # edit\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit") + _ok(repo, s, "2") # event 2 @T1 + + commit = gp.head_commit(repo) + body = canonical_json( + { + "version": 1, + "commit": commit, + "tree": gp.commit_tree(repo, commit), + "claims": [ + _v1_claim_entry("tests-pass", "fresh", (2,), 2, "tree-exact", 2, 0), + _v1_claim_entry("command-succeeded", "drifted", (0,), 0, "stale", 0, 0), + _v1_claim_entry("lint-clean", "unbacked", (7,), 2, "unknown", None, None), + _v1_claim_entry("tests-pass", "caught", (1,), 1, "failed", 1, 1), + ], + "coverage": {"total_events": 3, "by_coverage": {"complete": 3}}, + "secrets_override": False, + } + ) + _attach_raw_note(repo, commit, body) + + report = M.verify(s, repo) + observed = [ + (r.claim.label, r.grade, r.reason, [(c.status, c.path) for c in r.delta]) + for r in report.results + ] + assert observed == _LEGACY_EXPECTED + assert report.worst_status == "failed" + assert not report.all_verified + assert report.evidence_bound_count == 0 + assert report.total == 4 + assert all(r.sealed_grade is None for r in report.results) + + +# --- test 6: the happy path --------------------------------------------------- + + +def test_happy_path_is_evidence_bound_and_strict_clean(repo: Path): + s = _session(repo) + _ok(repo, s, "one") + _ok(repo, s, "two") + M.declare_claim( + s, Claim(ctype="tests-pass", label="a", event_indices=(0,), declared_at_index=1) + ) + M.declare_claim( + s, Claim(ctype="lint-clean", label="b", event_indices=(1,), declared_at_index=1) + ) + M.seal(s, repo) + + report = M.verify(s, repo) + assert [r.grade for r in report.results] == ["tree-exact", "tree-exact"] + assert all(r.evidence_bound for r in report.results) + assert report.evidence_bound_count == 2 and report.total == 2 + assert report.all_verified + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_seal_records_the_entry_hash_of_the_backing_event(repo: Path): + """The binding is the recorded entry's own chain hash, taken from the ledger.""" + s = _sealed_green(repo) + entry = list(s.entries())[0] + evidence = json.loads(_note_bytes(repo))["claims"][0]["evidence"] + assert evidence["event_index"] == 0 + assert evidence["entry_hash"] == entry.entry_hash + assert evidence["tree_after"] == entry.event.tree_after + assert evidence["env_fingerprint"] == entry.event.env_fingerprint + + +def test_amend_tree_fallback_still_resolves_and_stays_bound(repo: Path): + """The durable anchor keeps working: same tree, new commit id, still bound.""" + s = _sealed_green(repo) + original = gp.head_commit(repo) + _git(repo, "commit", "--amend", "-qm", "amended message") + assert gp.head_commit(repo) != original + + report = M.verify(s, repo) + assert report.resolved_by == "tree-fallback" + assert report.all_verified + assert report.evidence_bound_count == 1 + + +# --- test 7: the state is registered on the human surfaces ------------------- + + +def test_witness_unavailable_is_registered_in_both_render_tables(): + """Both tables are read through `.get(…, UNKNOWN)`, so an unregistered grade + renders as UNKNOWN — a silent lie rather than a loud error.""" + assert "witness-unavailable" in render._GRADE_DISPLAY + assert "witness-unavailable" in render._SORT_RANK + assert render._GRADE_DISPLAY["witness-unavailable"][0] == "WITNESS-UNAVAIL" + # Worse than stale, better than unknown — the order the verdict sorts by. + assert ( + render._SORT_RANK["unknown"] + < render._SORT_RANK["witness-unavailable"] + < render._SORT_RANK["stale"] + ) + + +def test_verdict_never_calls_a_witness_unavailable_report_recorded_exact( + repo: Path, tmp_path: Path, monkeypatch +): + monkeypatch.setenv("NO_COLOR", "1") + s = _sealed_green(repo) + _substitute_log(repo, s, tmp_path, ["something else entirely"]) + + out = render.render_verdict(M.verify(s, repo)) + assert "WITNESS-UNAVAIL" in out + assert "ALL RECORDED-EXACT" not in out + assert "0/1 claims evidence-bound" in out + for overclaim in ("VERIFIED", "PASSED", " OK"): + assert overclaim not in out + + +def test_html_report_shows_the_state_and_never_reads_as_passed( + repo: Path, tmp_path: Path +): + s = _sealed_green(repo) + _substitute_log(repo, s, tmp_path, ["something else entirely"]) + + html_out = render.render_html(M.verify(s, repo)) + assert "WITNESS-UNAVAIL" in html_out + assert "badge witness_unavailable" in html_out # styled, not bare ink + assert "ALL RECORDED-EXACT" not in html_out + assert "0/1 claims evidence-bound" in html_out + assert "VERIFIED" not in html_out and "PROVEN" not in html_out.upper() + + +def test_strict_never_accepts_witness_unavailable(): + """`all_verified`'s membership set is the one thing --strict rests on. + + Widening it to admit this state would make --strict pass on a claim whose + evidence nobody has seen, which is the exact lie the state exists to stop. + """ + report = M.VerifyReport( + commit="c" * 40, + tree="t" * 40, + resolved_by="commit", + results=[ + GradeResult( + Claim(ctype="tests-pass", label="x", event_indices=(0,)), + "witness-unavailable", + reason="sealed as tree-exact at commit cccccccccccc; " + "witness ledger unavailable (entry hash mismatch at index 0)", + sealed_grade="tree-exact", + ) + ], + coverage={}, + ) + assert not report.all_verified + assert report.worst_status == "witness-unavailable" diff --git a/tests/test_grading_honesty.py b/tests/test_grading_honesty.py new file mode 100644 index 0000000..36d6be5 --- /dev/null +++ b/tests/test_grading_honesty.py @@ -0,0 +1,209 @@ +"""P1.1 — the grading ladder must not turn a failed subprocess into a fact. + +Two defects are pinned here: + + 1. ``tree_delta`` failed open to ``[]``, so when the evidence tree's objects + were unreachable (the archived-ledger case) ``grade`` rendered one of two + false sentences — "tree moved since evidence: 0 path(s) differ" (which also + violates the module's own stale-MUST-carry-the-delta invariant) and + "evidence tree equals sealed tree but command was not self-stable" (an + assertion about tree equality derived from a subprocess that failed). + 2. ``grade`` consulted ``_supporting_event`` before ``_witnessed_failure``, so + a claim bound to [failed, success] graded on the success and the witnessed + failure was invisible. +""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import gitplumbing as gp +from didrun.capture import run_wrapped +from didrun.claims import Claim, grade +from didrun.ledger import Session + +# The two sentences the old fail-open path emitted. Neither may ever appear in +# a reason derived from a delta that was not computed. +FORBIDDEN = ("0 path(s) differ", "evidence tree equals sealed tree") + + +def _git(repo: Path, *a: str) -> None: + subprocess.run(["git", *a], cwd=str(repo), capture_output=True, text=True, check=True) + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _uncommitted_evidence(repo: Path) -> tuple[Session, str]: + """An event whose tree_after lives ONLY in the ledger's object dir. + + The dirt is never committed, so the tree object write-tree produced was + written under GIT_OBJECT_DIRECTORY into ``.didrun/objects`` and exists + nowhere else. Pointing a later grade at a different object dir reproduces + the archived-ledger case exactly: the event is present, the objects are not. + """ + s = _session(repo) + (repo / "scratch.py").write_text("x = 1\n") + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + sealed_tree = gp.commit_tree(repo, "HEAD") + ev = s.events()[0] + assert ev.tree_after is not None and ev.tree_after != sealed_tree + return s, sealed_tree + + +@pytest.mark.parametrize("pathspecs", [(), ("docs",)], ids=["no-pathspecs", "pathspecs"]) +def test_uncomputable_delta_grades_unknown_not_a_confident_sentence( + repo: Path, tmp_path: Path, pathspecs: tuple[str, ...] +): + """Objects gone -> `unknown`, and neither false sentence is emitted.""" + s, sealed_tree = _uncommitted_evidence(repo) + claim = Claim(ctype="tests-pass", label="t", event_indices=(0,), + pathspecs=pathspecs, declared_at_index=0) + + gone = tmp_path / "empty-objects" + gone.mkdir() + assert gp.tree_delta(repo, s.events()[0].tree_after, sealed_tree, gone) is None + + r = grade(claim, sealed_tree, s.events(), repo, gone) + assert r.grade == "unknown" + assert "not computable" in r.reason + for phrase in FORBIDDEN: + assert phrase not in r.reason + assert r.delta == [] # an uncomputed delta is never displayed + assert r.supporting_event_index == 0 + + # Control: the SAME claim over the SAME evidence with the objects present + # still grades stale. That is what proves the unknown above is caused by + # unavailability and not by the fixture being ungradeable in the first place. + reachable = grade(claim, sealed_tree, s.events(), repo, s.blobs.root) + assert reachable.grade == "stale" + assert "not computable" not in reachable.reason + assert reachable.delta, "the reachable control must carry a concrete delta" + + +def test_empty_delta_still_means_the_trees_are_equal(repo: Path): + """`[]` is "computed, and equal" — collapsing it into `None` breaks this. + + A command that mutates the tree is not self-stable, so it cannot be + tree-exact even when its resulting tree IS the sealed tree; the delta is + computed, comes back empty, and the stale-clean branch fires unchanged. + """ + s = _session(repo) + run_wrapped( + [sys.executable, "-c", "open('made.py', 'w').write('y = 2\\n')"], s, repo + ) + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "commit what the command produced") + sealed_tree = gp.commit_tree(repo, "HEAD") + + ev = s.events()[0] + assert not ev.self_stable() # the command moved the tree + assert ev.tree_after == sealed_tree # ...and landed exactly on the sealed one + + delta = gp.tree_delta(repo, ev.tree_after, sealed_tree, s.blobs.root) + assert delta is not None and delta == [] + + claim = Claim(ctype="tests-pass", label="t", event_indices=(0,), + pathspecs=("src",), declared_at_index=0) + r = grade(claim, sealed_tree, s.events(), repo, s.blobs.root) + assert r.grade == "stale" + assert r.reason == "evidence tree equals sealed tree but command was not self-stable" + + +def test_witnessed_failure_dominates_a_witnessed_success(repo: Path): + """[failed, success] grades FAILED, on the failure, not on the success. + + Reachable only through the library API today: the CLI's ``--event`` is + ``type=int``, not ``action="append"``, so every claim the CLI can build + binds exactly one event (0 of 846 corpus claims are multi-index). This is a + Low-severity honesty fix, not a headline. + """ + s = _session(repo) + run_wrapped([sys.executable, "-c", "import sys; sys.exit(1)"], s, repo) # 0: fails + run_wrapped([sys.executable, "-c", "import sys; sys.exit(0)"], s, repo) # 1: passes + claim = Claim(ctype="tests-pass", label="t", event_indices=(0, 1), declared_at_index=1) + + r = grade(claim, gp.commit_tree(repo, "HEAD"), s.events(), repo, s.blobs.root) + assert r.grade == "failed" + assert r.supporting_event_index == 0 + assert r.exit_code == 1 + assert r.reason == "the recorded command exited 1 — claim is not backed" + + +# --- every single-event rung of the ladder still fires exactly as before ------- + + +def _case_tree_exact(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + return s, Claim(ctype="tests-pass", label="t", event_indices=(0,), + declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +def _case_scope_exact(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + (repo / "src").mkdir() + (repo / "src" / "a.py").write_text("x = 1\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "add src") + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + (repo / "src" / "a.py").write_text("x = 2\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit src") + return s, Claim(ctype="tests-pass", label="t", event_indices=(0,), + pathspecs=("src",), declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +def _case_stale(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + (repo / "calc.py").write_text("def add(a, b):\n return a + b # later\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit") + return s, Claim(ctype="tests-pass", label="t", event_indices=(0,), + declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +def _case_unknown(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + run_wrapped([sys.executable, "-c", "print(2)"], s, repo) + return s, Claim(ctype="command-succeeded", label="c", event_indices=(1,), + declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +def _case_failed(repo: Path) -> tuple[Session, Claim, str]: + s = _session(repo) + run_wrapped([sys.executable, "-c", "import sys; sys.exit(1)"], s, repo) + return s, Claim(ctype="tests-pass", label="t", event_indices=(0,), + declared_at_index=0), gp.commit_tree(repo, "HEAD") + + +LADDER = { + "tree-exact": (_case_tree_exact, "self-stable command ran against the sealed tree"), + "scope-exact": (_case_scope_exact, "all 1 change(s) within declared pathspecs"), + "stale": (_case_stale, "tree moved since evidence: 1 path(s) differ"), + "unknown": (_case_unknown, + "retroactive binding: claim declared at index 0 precedes its backing event 1"), + "failed": (_case_failed, "the recorded command exited 1 — claim is not backed"), +} + + +@pytest.mark.parametrize("expected_grade", sorted(LADDER)) +def test_single_event_claims_are_unaffected(repo: Path, expected_grade: str): + """The hoist and the fail-closed delta must move no single-event grade. + + Reason strings are asserted verbatim, not by substring: downstream + validators read them, so a reworded rung is a break even when the grade is + right. + """ + build, expected_reason = LADDER[expected_grade] + s, claim, sealed_tree = build(repo) + r = grade(claim, sealed_tree, s.events(), repo, s.blobs.root) + assert r.grade == expected_grade + assert r.reason == expected_reason diff --git a/tests/test_interrupt_capture.py b/tests/test_interrupt_capture.py new file mode 100644 index 0000000..bc50791 --- /dev/null +++ b/tests/test_interrupt_capture.py @@ -0,0 +1,409 @@ +"""P2.3 — the recorder records the flight when the flight is interrupted. + +Two halves, and they meet at the same sentence: a lost event must cost one +event, never the ledger. + + - capture/CLI: SIGINT or SIGTERM during a wrapped command appends an event + that says what was NOT witnessed (no exit code, coverage "unobserved") + instead of guessing, keeps the output drained so far, and exits on the + conventional code with no traceback. + - ledger: an append is fsynced, a torn FINAL record is readable as a lost + event rather than as tamper, and `append` refuses to write past the tear + rather than burying it mid-log where it would break the chain forever. + +The signal legs drive the real CLI in a real process. Calling a handler +directly would test the handler, not the delivery; the delivery is the part +v0.1 got wrong. +""" + +from __future__ import annotations + +import json +import os +import signal +import subprocess +import sys +import threading +import time +from pathlib import Path + +import pytest + +from didrun import gitplumbing +from didrun.capture import CaptureInterrupted, run_wrapped +from didrun.claims import Claim, grade +from didrun.ledger import Event, LedgerError, Session, canonical_json + +# POSIX signal semantics. Windows has no SIGKILL and no process-group Ctrl-C +# with these mechanics; docs/COMPAT.md already scopes Tier-0 capture there. +pytestmark = pytest.mark.skipif( + os.name == "nt", reason="POSIX signal delivery semantics" +) + + +def _session(repo: Path, readonly: bool = False) -> Session: + return Session(repo / ".didrun", readonly=readonly) + + +def _wrapper_event(argv=("echo", "hi"), exit_code=0, **kw) -> Event: + return Event( + argv=tuple(argv), + cwd="/tmp", + env_fingerprint="abc123", + observed_via="wrapper", + coverage="complete", + exit_code=exit_code, + **kw, + ) + + +_CLI_BOOT = "import sys; from didrun.cli import main; sys.exit(main(sys.argv[1:]))" + +# Prints a marker, flushes, publishes its pid, then blocks. The pid file is +# written LAST and via a rename, so its existence proves both that the child is +# running and that the marker has already been handed to the pipe: the test +# never sends a signal into a race with the child's startup. +_STARTED_THEN_SLEEP = ( + "import os, sys, time\n" + "sys.stdout.write('MARKER-STARTED\\n')\n" + "sys.stdout.flush()\n" + "tmp = sys.argv[1] + '.tmp'\n" + "open(tmp, 'w').write(str(os.getpid()))\n" + "os.replace(tmp, sys.argv[1])\n" + "time.sleep(120)\n" +) + + +def _wait_for(predicate, timeout: float, interval: float = 0.02) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(interval) + return predicate() + + +def _start_wrapped_sleeper(repo: Path, tmp_path: Path): + """Spawn `didrun run -- ` and wait until it is running. + + Returns ``(proc, pid_file, child_argv)``. The child script lives outside the + repo so it cannot move a tree digest, and its source carries the marker text + so no assertion about the marker can pass off didrun's own summary line. + """ + script = tmp_path / "started_then_sleep.py" + script.write_text(_STARTED_THEN_SLEEP, encoding="utf-8") + pid_file = tmp_path / "child.pid" + child_argv = [sys.executable, str(script), str(pid_file)] + proc = subprocess.Popen( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "run", "--", *child_argv], + cwd=str(repo), + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + if not _wait_for(pid_file.exists, 60): + proc.kill() + proc.communicate(timeout=30) + pytest.fail("the wrapped child never started") + return proc, pid_file, child_argv + + +def _interrupt(repo: Path, tmp_path: Path, signum: int): + """Run a blocking child under didrun, signal DIDRUN, collect the outcome.""" + proc, _pid_file, child_argv = _start_wrapped_sleeper(repo, tmp_path) + proc.send_signal(signum) + out, err = proc.communicate(timeout=120) + return proc.returncode, out, err, child_argv + + +# --- 1 + 4: the event is recorded, and the exit code is conventional ---------- + + +@pytest.mark.parametrize( + "signame,expected_code", + [("SIGINT", 130), ("SIGTERM", 143)], +) +def test_an_interrupt_records_an_unobserved_event_and_exits_cleanly( + repo: Path, tmp_path: Path, signame: str, expected_code: int +): + signum = getattr(signal, signame) + code, out, err, child_argv = _interrupt(repo, tmp_path, signum) + + assert code == expected_code, err.decode("utf-8", "replace") + # v0.1's whole report was a traceback. The line that replaces it has to be + # both present and machine-greppable. + assert b"Traceback" not in err + assert b"KeyboardInterrupt" not in err + assert f"interrupted by {signame}".encode("ascii") in err + + entries = list(_session(repo, readonly=True).entries()) + assert len(entries) == 1, "the flight recorder lost the flight" + ev = entries[0].event + assert ev.observed_via == "wrapper" + assert ev.coverage == "unobserved" + assert ev.exit_code is None, "nobody witnessed an exit; recording one is a lie" + assert ev.argv == tuple(child_argv), "the in-flight argv is what was recorded" + # The recorded index is what the operator was told to look at. + assert f"recorded event {entries[0].index}".encode("ascii") in err + + ok, broken = _session(repo, readonly=True).verify_chain() + assert (ok, broken) == (True, None) + + +# --- 2: partial output survives (this is what P2.2 unlocked) ------------------ + + +def test_partial_output_survives_the_interrupt(repo: Path, tmp_path: Path): + """subprocess.run structurally could not do this. + + Its bare `except:` kills the child and re-raises with the buffers still + bound inside communicate(), where nothing can reach them. The pump drains + into a caller-owned accumulator, so the bytes the child had already written + are still there after the signal. + """ + _code, _out, _err, _argv = _interrupt(repo, tmp_path, signal.SIGINT) + + s = _session(repo, readonly=True) + ev = list(s.entries())[0].event + assert ev.stdout_blob is not None + assert b"MARKER-STARTED" in s.blobs.get(ev.stdout_blob) + + +# --- 3: it grades `unknown`, with no ladder change --------------------------- + + +def test_a_claim_bound_to_an_interrupted_event_grades_unknown( + repo: Path, tmp_path: Path +): + """The design criterion for the schema-free shape. + + `_supporting_event` skips the event (exit_code != 0) and `_witnessed_failure` + skips it (exit_code is None), so the correct grade falls out of the existing + ladder. The other half of this test — that claims.py was not edited — is a + `git diff --name-only` check in the unit's verification, because a comment + is not a check. + """ + _code, _out, _err, _argv = _interrupt(repo, tmp_path, signal.SIGINT) + + s = _session(repo, readonly=True) + events = [e.event for e in s.entries()] + claim = Claim( + ctype="command-succeeded", + label="the interrupted command", + event_indices=(0,), + declared_at_index=0, + ) + result = grade( + claim, + gitplumbing.commit_tree(repo, "HEAD"), + events, + repo, + s.blobs.root, + ) + assert result.grade == "unknown" + assert result.exit_code is None + # Never `failed`: an interrupted command was not caught lying. + assert "no witnessed successful command" in result.reason + + +# --- 5: the child dies, the parent lives — the normal path is untouched ------ + + +def test_killing_the_child_still_takes_the_normal_path(repo: Path, tmp_path: Path): + """A SIGKILLed child is a witnessed (negative) exit, not an interrupt. + + The interrupt path must not widen to swallow it: `-9` is evidence about the + command, and downgrading it to `unobserved` would lose a real observation. + """ + proc, pid_file, _argv = _start_wrapped_sleeper(repo, tmp_path) + os.kill(int(pid_file.read_text()), signal.SIGKILL) + _out, err = proc.communicate(timeout=120) + + assert b"interrupted by" not in err + ev = list(_session(repo, readonly=True).entries())[0].event + assert ev.coverage == "complete" + assert ev.exit_code == -signal.SIGKILL + + +# --- 6: a torn tail costs one event, not the ledger -------------------------- + + +def _torn_log(tmp_path: Path, records: int = 3, keep: int = 2) -> tuple: + """A log of ``records`` events truncated mid-way through record ``keep``. + + Returns ``(session, prefix_len, surviving_hashes)``. + """ + s = Session(tmp_path / ".didrun") + for i in range(records): + s.append(_wrapper_event(argv=("cmd", str(i)))) + lines = s.log_path.read_bytes().splitlines(keepends=True) + prefix = b"".join(lines[:keep]) + hashes = [json.loads(line)["entry_hash"] for line in lines[:keep]] + # Cut at a byte offset inside the record, not at a line boundary, or the + # log is merely SHORT and parses cleanly. + s.log_path.write_bytes(prefix + lines[keep][:37]) + return Session(tmp_path / ".didrun"), len(prefix), hashes + + +def test_a_torn_final_record_is_readable_and_is_not_tamper(tmp_path: Path): + s, prefix_len, hashes = _torn_log(tmp_path) + + entries = list(s.entries()) + assert [e.index for e in entries] == [0, 1] + assert [e.entry_hash for e in entries] == hashes + assert s.tail_truncated is True + assert s.tail_truncated_offset == prefix_len + + verdict = s.verify_chain_detail() + assert verdict.status == "intact" + assert verdict.first_broken_index is None + assert "torn at byte" in verdict.reason + assert str(prefix_len) in verdict.reason + assert s.verify_chain() == (True, None) + + +def test_a_malformed_middle_record_is_still_a_hard_error(tmp_path: Path): + """The tolerance is for the LAST line only. + + A hole with records after it is corruption, not an interrupted write, and + quietly skipping it would let a reader present a log with a missing middle + as a whole one. + """ + s = Session(tmp_path / ".didrun") + for i in range(3): + s.append(_wrapper_event(argv=("cmd", str(i)))) + lines = s.log_path.read_bytes().splitlines(keepends=True) + s.log_path.write_bytes(lines[0] + b"{not json at all}\n" + lines[2]) + + reader = Session(tmp_path / ".didrun") + with pytest.raises(LedgerError) as exc: + list(reader.entries()) + assert "corruption inside the log" in str(exc.value) + assert reader.tail_truncated is False + + +# --- 6b: the recorder survives the tear, and never writes past it ------------ + + +def test_a_torn_tail_does_not_kill_the_recorder_and_is_never_written_past( + tmp_path: Path, +): + """Change 7 without change 8 fixes the reader and leaves the writer dead. + + `_last()` must read past the tear (or every subsequent append raises, and + one lost event still kills the ledger); `append()` must refuse to WRITE + past it (or the partial bytes end up mid-log, unparseable forever, and one + lost event becomes a permanently broken chain). + """ + s, prefix_len, hashes = _torn_log(tmp_path) + before = s.log_path.read_bytes() + + assert s._last() == (1, hashes[1]) + + with pytest.raises(LedgerError) as exc: + s.append(_wrapper_event(argv=("cmd", "after"))) + assert "torn final record" in str(exc.value) + assert str(prefix_len) in str(exc.value), "the operator needs the byte offset" + assert s.log_path.read_bytes() == before, "a refusal must not mutate evidence" + + # The operator's recovery, at the offset the refusal named. + with s.log_path.open("r+b") as fh: + fh.truncate(prefix_len) + + fresh = Session(tmp_path / ".didrun") + entry = fresh.append(_wrapper_event(argv=("cmd", "after"))) + assert entry.index == 2 + assert fresh.tail_truncated is False + # No unparseable line anywhere, in the middle or at the end. + for line in fresh.log_path.read_bytes().splitlines(): + json.loads(line) + assert fresh.verify_chain() == (True, None) + assert [e.index for e in fresh.entries()] == [0, 1, 2] + + +# --- 7: fsync changes durability, not bytes ---------------------------------- + + +def test_append_fsyncs_the_record_before_returning(tmp_path: Path, monkeypatch): + """The durability half of the change, pinned by the call. + + The byte-identity test below is a regression guard: it passes on the + unfixed code by construction, because the point of fsync is that it changes + nothing on disk. Only this one fails when the fsync goes away, so both are + here and they are not the same test. + """ + real_fsync = os.fsync + fsynced: list = [] + + def spy(fd): + fsynced.append(fd) + return real_fsync(fd) + + monkeypatch.setattr(os, "fsync", spy) + s = Session(tmp_path / ".didrun") + s.append(_wrapper_event()) + + assert fsynced, "append returned before the record was durable" + assert list(s.entries())[0].event.argv == ("echo", "hi") + + +def test_fsync_adds_no_bytes_to_the_log(tmp_path: Path): + """The on-disk line is exactly what every previous release wrote. + + Pinned against a reconstruction rather than a golden file so it fails if the + record's SHAPE moves, not only if fsync appended something. + """ + s = Session(tmp_path / ".didrun") + events = [_wrapper_event(argv=("cmd", str(i))) for i in range(3)] + entries = [s.append(ev) for ev in events] + + expected = b"".join( + canonical_json( + { + "index": entry.index, + "prev_hash": entry.prev_hash, + "entry_hash": entry.entry_hash, + "event": ev.to_dict(), + } + ) + + b"\n" + for entry, ev in zip(entries, events) + ) + assert s.log_path.read_bytes() == expected + + +# --- 8: handlers are borrowed, never kept ------------------------------------ + + +def test_signal_handlers_are_restored_after_a_normal_run(repo: Path): + before = { + sig: signal.getsignal(sig) for sig in (signal.SIGINT, signal.SIGTERM) + } + ev = run_wrapped([sys.executable, "-c", "pass"], _session(repo), repo) + assert ev.exit_code == 0 + after = {sig: signal.getsignal(sig) for sig in (signal.SIGINT, signal.SIGTERM)} + assert after == before + + +def test_run_wrapped_from_a_worker_thread_records_normally(repo: Path): + """`signal.signal` raises off the main thread, and run_wrapped is a library + entry point an embedder may call from a worker. The guard makes that a + silent skip, not a ValueError — and the recorded event is unchanged.""" + outcome: dict = {} + + def work() -> None: + try: + outcome["event"] = run_wrapped( + [sys.executable, "-c", "print('hi')"], _session(repo), repo + ) + except BaseException as exc: # recorded, never swallowed + outcome["error"] = exc + + worker = threading.Thread(target=work) + worker.start() + worker.join(120) + assert not worker.is_alive() + assert "error" not in outcome, repr(outcome.get("error")) + + ev = outcome["event"] + assert (ev.observed_via, ev.coverage, ev.exit_code) == ("wrapper", "complete", 0) + assert not isinstance(outcome.get("error"), CaptureInterrupted) diff --git a/tests/test_interrupt_records_the_flight.py b/tests/test_interrupt_records_the_flight.py new file mode 100644 index 0000000..1b2aea1 --- /dev/null +++ b/tests/test_interrupt_records_the_flight.py @@ -0,0 +1,121 @@ +"""A signal must never cost the event for a command that actually ran. + +Regression cover for the lost-flight window between the drain and the append. +Two distinct mechanisms put an event at risk once the child is reaped: + +1. A second interrupt landing while didrun is digesting the tree and appending + (the recorder's own Python signal handling). +2. A terminal Ctrl-C reaching the whole foreground process group, which kills + the ``git`` the after-digest shells out to -- even when the child had + already exited cleanly and the drain saw no signal at all. + +Both used to end the process with no record of a command whose side effects +are on disk. The event is worth more than the after-digest: `None` grades +`unknown`, which is honest, while no event at all says the command never ran. +""" + +import os +import signal +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from didrun import gitplumbing +from didrun.capture import run_wrapped +from didrun.ledger import Session + + +def _git(repo, *args): + subprocess.run(["git", *args], cwd=repo, check=True, capture_output=True) + + +@pytest.fixture() +def repo(tmp_path): + _git(tmp_path, "init", "-q", ".") + _git(tmp_path, "config", "user.email", "t@t.test") + _git(tmp_path, "config", "user.name", "t") + (tmp_path / "f.txt").write_text("x\n") + _git(tmp_path, "add", "-A") + _git(tmp_path, "commit", "-qm", "init") + return tmp_path + + +def test_a_failing_after_digest_still_records_the_event(repo, monkeypatch): + """The mechanism behind the process-group kill, made deterministic. + + Rather than racing a real signal, fail the digest the way a killed git + fails it. The event must still land, tree_after must be None, and coverage + must say the tree was not witnessed instead of claiming "complete". + """ + session = Session(repo / ".didrun") + + real = gitplumbing.tree_digest + calls = {"n": 0} + + def flaky(*a, **kw): + calls["n"] += 1 + if calls["n"] > 1: # tree_before succeeds, tree_after dies + raise gitplumbing.GitError("git rev-parse --git-path index failed (-2): ") + return real(*a, **kw) + + monkeypatch.setattr(gitplumbing, "tree_digest", flaky) + + event = run_wrapped([sys.executable, "-c", "print('ran')"], session, repo=repo) + + entries = list(session.entries()) + assert len(entries) == 1, "a witnessed command left no event" + assert event.exit_code == 0, "the command completed; its exit code is known" + assert event.tree_after is None, "an unwitnessed tree must be None, not a guess" + assert event.coverage == "observed-text-only", ( + "claiming 'complete' would assert the tree was witnessed when it was not" + ) + assert session.verify_chain()[0], "the chain must stay intact" + + +def test_second_interrupt_during_the_record_phase_does_not_lose_the_flight(repo): + """End to end, through the CLI, against the real double-Ctrl-C window. + + The first signal ends the child; the second lands while didrun is + digesting and appending. Before the fix this window ran ~30-140ms wide and + lost the event outright. + """ + didrun = Path(sys.executable).parent / "didrun" + if not didrun.exists(): # pragma: no cover - depends on install layout + pytest.skip("didrun console script not present in this environment") + + (repo / "child.py").write_text( + "import time\n" + "for i in range(40):\n" + " print(i, flush=True)\n" + " time.sleep(0.05)\n" + ) + + lost = [] + for gap_ms in (30, 60, 90, 120): + for stale in repo.glob(".didrun/session.log"): + stale.unlink() + proc = subprocess.Popen( + [str(didrun), "run", "--", sys.executable, "child.py"], + cwd=repo, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + time.sleep(0.6) + for delay in (0.0, gap_ms / 1000.0): + time.sleep(delay) + try: + os.killpg(proc.pid, signal.SIGINT) + except (ProcessLookupError, PermissionError): # pragma: no cover + pass + proc.wait(timeout=60) + + log = repo / ".didrun" / "session.log" + lines = [l for l in log.read_text().splitlines() if l.strip()] if log.exists() else [] + if not lines: + lost.append(gap_ms) + + assert not lost, f"interrupt lost the flight at gaps (ms): {lost}" diff --git a/tests/test_ledger.py b/tests/test_ledger.py index a9f09bc..b6194c0 100644 --- a/tests/test_ledger.py +++ b/tests/test_ledger.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import os import subprocess import sys import textwrap @@ -42,6 +43,42 @@ def test_blob_roundtrip_and_dedup(tmp_path: Path): assert store.get(d1) == b"hello" +def test_two_writers_of_one_blob_do_not_collide_on_a_temp_name( + tmp_path: Path, monkeypatch +): + """P2.4 — concurrent recorders store the same blob constantly. + + An empty stderr is one digest every event shares, so two `didrun run` + processes writing at once is the normal case, not the exotic one. This is + the crash, driven deterministically instead of hoped for: writer A is + suspended at its rename, writer B completes the SAME blob inside that + window, and A resumes. With a shared ``.tmp`` name B's rename has + already consumed A's temp and A dies with FileNotFoundError. + + Measured before the fix on an eight-way concurrent `didrun run`: 2 of 6 + attempts lost a process to exactly this traceback. + """ + root = tmp_path / "obj" + store = BlobStore(root) + data = b"identical bytes from both writers" + real_replace = os.replace + interleaved = [] + + def replace_with_a_second_writer_inside(src, dst): + if not interleaved: + interleaved.append(True) + BlobStore(root).put(data) + return real_replace(src, dst) + + monkeypatch.setattr(os, "replace", replace_with_a_second_writer_inside) + + assert store.put(data) == sha256_hex(data) + assert interleaved == [True], "the second writer never ran; the test is vacuous" + assert store.get(sha256_hex(data)) == data + # No temp residue survives a completed pair of writes. + assert [p.name for p in root.iterdir()] == [sha256_hex(data)] + + def test_blob_corruption_detected(tmp_path: Path): store = BlobStore(tmp_path / "obj") d = store.put(b"payload") diff --git a/tests/test_ledger_permissions.py b/tests/test_ledger_permissions.py new file mode 100644 index 0000000..d886fea --- /dev/null +++ b/tests/test_ledger_permissions.py @@ -0,0 +1,266 @@ +"""P2.1 — the ledger root is private, and read-only commands do not create one. + +The criterion here is deliberately about the ROOT, not about files. The ledger's +``objects/`` directory is populated by git, twice per ``didrun run``, through the +``GIT_OBJECT_DIRECTORY`` redirect, at git's own modes — so "every file is 0600" +is not a property this package can hold. 0700 on the root is, and it is what +makes another uid unable to reach the files through the directory. +""" + +from __future__ import annotations + +import os +import stat +from pathlib import Path + +import pytest + +from didrun import gitplumbing +from didrun.cli import main +from didrun.ledger import Event, LedgerError, Session + + +def _mode(path: Path) -> int: + return stat.S_IMODE(path.stat().st_mode) + + +def _wrapper_event(argv=("echo", "hi"), exit_code=0, **kw) -> Event: + return Event( + argv=tuple(argv), + cwd="/tmp", + env_fingerprint="abc123", + observed_via="wrapper", + coverage="complete", + exit_code=exit_code, + **kw, + ) + + +@pytest.fixture +def umask(request): + """Run a test under a specific umask and restore it afterwards.""" + previous = os.umask(request.param) + try: + yield request.param + finally: + os.umask(previous) + + +# --- 1. the criterion: an existing 0755 ledger is re-tightened --------------- + +def test_existing_0755_ledger_is_retightened_on_open(tmp_path: Path): + # A ledger a v0.1 binary created: the directory already exists at 0755. + root = tmp_path / ".didrun" + root.mkdir() + os.chmod(root, 0o755) + assert _mode(root) == 0o755 + + Session(root) + + # mkdir(mode=0o700, exist_ok=True) does NOT re-tighten an existing + # directory, so an implementation that only passes mode= fails here. + assert _mode(root) == 0o700 + + +def test_fresh_ledger_root_is_private(tmp_path: Path): + root = tmp_path / ".didrun" + Session(root) + assert _mode(root) == 0o700 + + +# --- 2. umask independence, and the files this version creates --------------- + +@pytest.mark.parametrize("umask", [0o022, 0o002, 0o077], indirect=True) +def test_root_and_own_files_are_private_under_any_umask( + tmp_path: Path, umask: int, monkeypatch +): + from didrun import manifest as _manifest + from didrun.claims import Claim + + root = tmp_path / f".didrun-{umask:03o}" + session = Session(root) + session.append(_wrapper_event()) + _manifest.declare_claim( + session, + Claim(ctype="tests-pass", label="t", event_indices=(0,), declared_at_index=0), + ) + _manifest._record_seal(session, "c" * 40, "t" * 40, 1) + + assert _mode(root) == 0o700 + for name in ("session.log", "claims.jsonl", "seals.jsonl", ".gitignore"): + assert _mode(root / name) == 0o600, name + + # Deliberately NOT asserted: modes under objects/. git writes into that + # directory itself via GIT_OBJECT_DIRECTORY (gitplumbing.tree_digest), at + # git's modes, on every recorded command — so any assertion here would be + # about git's umask handling, not didrun's, and would be repaired-then-undone + # by the next `didrun run`. The root's 0700 is the control (red-team A2.4). + + +# --- 2b. the inherited-file bound, pinned rather than assumed ---------------- + +def test_inherited_v01_files_keep_their_modes_only_the_root_is_repaired( + tmp_path: Path, +): + root = tmp_path / ".didrun" + root.mkdir() + log = root / "session.log" + gitignore = root / ".gitignore" + log.write_bytes(b"") + gitignore.write_text("*\n", encoding="ascii") + os.chmod(log, 0o644) + os.chmod(gitignore, 0o644) + os.chmod(root, 0o755) + gitignore_before = gitignore.read_bytes() + mtime_before = gitignore.stat().st_mtime_ns + + Session(root) + + # This is the DELIBERATE BOUND of Session.harden, not an oversight: the + # root is the control, the files are not retro-fixed. O_CREAT's mode + # applies at creation only, and .gitignore is written only when absent, so + # a ledger an older binary created keeps 0644 on those files forever. + # Nobody may read the 0600 assertions in the test above as a general + # "the files are 0600" guarantee. + assert _mode(root) == 0o700 + assert _mode(log) == 0o644 + assert _mode(gitignore) == 0o644 + assert gitignore.read_bytes() == gitignore_before + assert gitignore.stat().st_mtime_ns == mtime_before + + +# --- the two guards on the chmod -------------------------------------------- + +def test_a_ledger_owned_by_another_uid_is_warned_about_not_chmodded( + tmp_path: Path, monkeypatch, capsys +): + root = tmp_path / ".didrun" + root.mkdir() + os.chmod(root, 0o755) + # Cannot create a foreign-owned directory without root, so move the OTHER + # side of the comparison: this process reports a euid the directory is not + # owned by. The branch under test is identical. + monkeypatch.setattr(os, "geteuid", lambda: root.stat().st_uid + 1) + + Session(root) + + assert _mode(root) == 0o755, "chmodded a directory this process does not own" + err = capsys.readouterr().err + assert "not tightening" in err and "0700" in err + + +def test_missing_geteuid_does_not_crash_the_windows_path(tmp_path: Path, monkeypatch): + # os.geteuid does not exist on Windows. The ownership check is skipped + # there rather than raising AttributeError on every Session open. + root = tmp_path / ".didrun" + root.mkdir() + os.chmod(root, 0o755) + monkeypatch.delattr(os, "geteuid", raising=False) + + Session(root) + + # POSIX chmod still works here, so the mode moves; the point of the test is + # that the absent attribute is a skipped check, not an exception. + assert _mode(root) == 0o700 + + +# --- 3. a read-only open creates nothing ------------------------------------ + +def test_readonly_session_creates_no_ledger(tmp_path: Path): + root = tmp_path / ".didrun" + session = Session(root, readonly=True) + assert not root.exists() + # And it still answers read questions about the ledger that is not there. + assert list(session.entries()) == [] + assert session.events() == [] + assert session.verify_chain() == (True, None) + assert not root.exists() + + +def test_readonly_session_refuses_to_write(tmp_path: Path): + session = Session(tmp_path / ".didrun", readonly=True) + with pytest.raises(LedgerError): + session.append(_wrapper_event()) + with pytest.raises(LedgerError): + session.blobs.put(b"payload") + assert not (tmp_path / ".didrun").exists() + + +def test_cmd_verify_on_a_repo_with_no_ledger_creates_nothing(repo: Path, capsys): + ledger = repo / ".didrun" + assert not ledger.exists() + + rc = main(["--repo", str(repo), "verify"]) + + assert rc == 0 + assert not ledger.exists(), "a read-only verify manufactured a ledger" + assert sorted(p.name for p in repo.iterdir()) == [".git", "calc.py"] + + +def test_cmd_show_session_on_a_repo_with_no_ledger_creates_nothing(repo: Path): + rc = main(["--repo", str(repo), "show", "--session"]) + assert rc == 0 + assert not (repo / ".didrun").exists() + + +# --- 4. a read-only open re-modes nothing ----------------------------------- + +def test_readonly_verify_does_not_remode_an_existing_ledger(repo: Path): + ledger = repo / ".didrun" + ledger.mkdir() + os.chmod(ledger, 0o755) + + rc = main(["--repo", str(repo), "verify"]) + + assert rc == 0 + # Pointing a read-only command at someone else's archive must not touch it. + assert _mode(ledger) == 0o755 + + +# --- 5. the chmod moves no tree digest -------------------------------------- + +def test_chmod_is_digest_neutral(repo: Path): + ledger = repo / ".didrun" + ledger.mkdir() + os.chmod(ledger, 0o755) + before = gitplumbing.tree_digest(repo) + assert before is not None + + Session(ledger) + + assert _mode(ledger) == 0o700 + after = gitplumbing.tree_digest(repo) + assert after == before + + +# --- 6. the new file-open path still appends a verifiable chain -------------- + +def test_append_and_chain_survive_the_private_open(tmp_path: Path): + root = tmp_path / ".didrun" + session = Session(root) + for i in range(4): + session.append(_wrapper_event(argv=("cmd", str(i)))) + ok, broke = session.verify_chain() + assert ok and broke is None + assert [e.index for e in session.entries()] == [0, 1, 2, 3] + + # Re-opening an existing ledger appends, never truncates. + reopened = Session(root) + reopened.append(_wrapper_event(argv=("cmd", "4"))) + assert len(list(reopened.entries())) == 5 + assert reopened.verify_chain() == (True, None) + assert _mode(root / "session.log") == 0o600 + + +def test_blob_temp_mode_does_not_leak_through_the_rename(tmp_path: Path): + # os.replace preserves the SOURCE mode, so a loose 0644 temp would become a + # 0644 blob. Under a permissive umask that is exactly what a plain + # write_bytes produced. + previous = os.umask(0o000) + try: + session = Session(tmp_path / ".didrun") + digest = session.blobs.put(b"secret output") + finally: + os.umask(previous) + assert _mode(session.blobs.root / digest) == 0o600 + assert session.blobs.get(digest) == b"secret output" diff --git a/tests/test_redact_render.py b/tests/test_redact_render.py index ee042db..458d11d 100644 --- a/tests/test_redact_render.py +++ b/tests/test_redact_render.py @@ -43,7 +43,8 @@ def test_redact_replaces_with_marker(): # --- render helpers ---------------------------------------------------------- -def _report(results, all_verified=False, worst="stale"): +def _report(results, all_verified=False, worst="stale", chain="intact", + chain_index=None, chain_why=""): @dataclass class R: commit: str = "abcdef123456" @@ -53,6 +54,13 @@ class R: coverage: dict = None secrets_override: bool = False notes_skipped: int = 0 + # Mirrors VerifyReport's chain fields. Carried as real attributes rather + # than left off: render reads them directly, so a surface that starts + # depending on a chain state this fake cannot express fails here loudly + # instead of silently rendering the default. + chain_status: str = chain + chain_broken_index: object = chain_index + chain_reason: str = chain_why @property def worst_status(self): @@ -62,6 +70,16 @@ def worst_status(self): def all_verified(self): return all_verified + # Mirrors VerifyReport: both human surfaces report how many verdicts + # were checked against the recorded entry the seal named. + @property + def evidence_bound_count(self): + return sum(1 for r in self.results if r.evidence_bound) + + @property + def total(self): + return len(self.results) + return R(results=results, coverage={"by_coverage": {"complete": 1}}) diff --git a/tests/test_seal_publication.py b/tests/test_seal_publication.py index ec4db17..9d60cf6 100644 --- a/tests/test_seal_publication.py +++ b/tests/test_seal_publication.py @@ -27,7 +27,9 @@ # Keys `to_json()` emits that a note of this manifest version never carried. # Additive fields land here as a DELIBERATE one-line diff, never silently. -_ADDITIVE_KEYS_BY_MANIFEST_VERSION = {1: frozenset()} +# v2 adds no TOP-LEVEL key: the evidence binding lives inside each claim entry, +# so the manifest's own key set is unchanged from v1. +_ADDITIVE_KEYS_BY_MANIFEST_VERSION = {1: frozenset(), 2: frozenset()} def _session(repo: Path) -> Session: diff --git a/tests/test_streaming_capture.py b/tests/test_streaming_capture.py new file mode 100644 index 0000000..7898331 --- /dev/null +++ b/tests/test_streaming_capture.py @@ -0,0 +1,349 @@ +"""P2.2 — the Tier-0 capture pump streams instead of buffering to completion. + +The criterion is byte-identity of what is RECORDED: the drained blobs must equal +what ``subprocess.run(capture_output=True)`` would have returned, for every shape +of output. Everything else here (tee, heartbeat, ``show --output``) is display, +and the tests pin that display never becomes the default and never carries the +child's bytes into a place the buffered path did not. +""" + +from __future__ import annotations + +import re +import subprocess +import sys +import threading +import time +from pathlib import Path + +import pytest + +from didrun.capture import run_wrapped +from didrun.ledger import Session, sha256_hex + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +# Spawning the CLI as a real process is the only way to see what a caller's +# terminal sees; `-c` is used rather than a path so no absolute path is embedded. +_CLI_BOOT = "import sys; from didrun.cli import main; sys.exit(main(sys.argv[1:]))" + + +def _cli_argv(repo: Path, *args: str) -> list: + return [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), *args] + + +def _run_cli(repo: Path, *args: str, timeout: float = 120) -> subprocess.CompletedProcess: + return subprocess.run( + _cli_argv(repo, *args), cwd=str(repo), capture_output=True, timeout=timeout + ) + + +def _child_script(tmp_path: Path, name: str, source: str) -> list: + """A child whose ARGV carries none of the text it prints. + + ``didrun run``'s summary line echoes argv, so a `-c` child whose source + contains the marker would let a marker-absence assertion pass (or fail) for + the wrong reason. The script lives outside the repo so it cannot move a tree + digest either. + """ + path = tmp_path / name + path.write_text(source, encoding="utf-8") + return [sys.executable, str(path)] + + +def _wait_for(predicate, timeout: float, interval: float = 0.05) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(interval) + return predicate() + + +# --- 1. the criterion: recorded blobs are byte-identical ---------------------- + +_NOTHING = "pass" +_LARGE = ( + "import sys\n" + "for _ in range(1024):\n" + " sys.stdout.buffer.write(b'y' * 1024)\n" + "sys.stdout.buffer.flush()\n" +) +_INTERLEAVED = ( + "import sys\n" + "for i in range(200):\n" + " sys.stdout.buffer.write(b'out-%d\\n' % i)\n" + " sys.stdout.buffer.flush()\n" + " sys.stderr.buffer.write(b'err-%d\\n' % i)\n" + " sys.stderr.buffer.flush()\n" +) +# NULs and lone 0xfe/0xff: bytes no text decoder round-trips. +_BINARY = ( + "import sys\n" + "sys.stdout.buffer.write(bytes(range(256)) * 8)\n" + "sys.stdout.buffer.flush()\n" +) +_STDERR_ONLY = ( + "import sys\n" + "sys.stderr.buffer.write(b'only stderr, repeatedly\\n' * 500)\n" + "sys.stderr.buffer.flush()\n" +) + + +@pytest.mark.parametrize( + "name,script", + [ + ("empty", _NOTHING), + ("large", _LARGE), + ("interleaved", _INTERLEAVED), + ("binary", _BINARY), + ("stderr-only", _STDERR_ONLY), + ], +) +def test_streamed_blobs_are_byte_identical_to_the_buffered_path( + repo: Path, name: str, script: str +): + """The pump changes WHEN bytes are seen, never WHICH bytes are recorded.""" + argv = [sys.executable, "-c", script] + reference = subprocess.run(argv, cwd=str(repo), capture_output=True) + + s = _session(repo) + ev = run_wrapped(argv, s, repo) + + assert ev.stdout_blob == sha256_hex(reference.stdout), f"{name}: stdout digest" + assert ev.stderr_blob == sha256_hex(reference.stderr), f"{name}: stderr digest" + assert s.blobs.get(ev.stdout_blob) == reference.stdout + assert s.blobs.get(ev.stderr_blob) == reference.stderr + assert ev.exit_code == reference.returncode + # The persisted shape is unchanged: this is still a complete wrapper event. + assert (ev.observed_via, ev.coverage) == ("wrapper", "complete") + + +# --- 2. liveness -------------------------------------------------------------- + +_EARLY_THEN_SLEEP = ( + "import sys, time\n" + "sys.stdout.write('MARKER-EARLY\\n')\n" + "sys.stdout.flush()\n" + "time.sleep(5)\n" +) + + +def test_tee_surfaces_output_before_the_child_exits(repo: Path, tmp_path: Path): + """--tee is progressive, not a replay at the end: the marker is readable + while the child is still running.""" + sink = tmp_path / "tee.out" + child = _child_script(tmp_path, "early_then_sleep.py", _EARLY_THEN_SLEEP) + with sink.open("wb") as fh: + proc = subprocess.Popen( + _cli_argv(repo, "run", "--tee", "--", *child), + cwd=str(repo), + stdout=fh, + stderr=subprocess.DEVNULL, + ) + try: + seen = _wait_for(lambda: b"MARKER-EARLY" in sink.read_bytes(), timeout=3.0) + # Read the liveness fact at the same instant, not after the wait. + still_running = proc.poll() is None + finally: + rc = proc.wait(timeout=60) + + assert seen, "the child's output never reached the terminal while it ran" + assert still_running, "the marker only appeared after the child had exited" + assert rc == 0 + + +def test_default_run_tees_nothing_and_prints_only_the_summary(repo: Path, tmp_path: Path): + """No flags: stdout carries the summary line and not one child byte. + External gates parse this stream.""" + child = _child_script( + tmp_path, + "noisy.py", + "import sys\nprint('MARKER-CHILD')\nprint('MARKER-ERR', file=sys.stderr)\n", + ) + proc = _run_cli(repo, "run", "--", *child) + + assert proc.returncode == 0 + lines = [ln for ln in proc.stdout.decode().splitlines() if ln.strip()] + assert len(lines) == 1, f"expected only the summary line, got {lines!r}" + assert lines[0].startswith("recorded wrapper event") + assert b"MARKER-CHILD" not in proc.stdout + assert b"MARKER-CHILD" not in proc.stderr + assert b"MARKER-ERR" not in proc.stdout + assert b"MARKER-ERR" not in proc.stderr + + +# --- 3. no deadlock ----------------------------------------------------------- + +_BOTH_PIPES_BIG = ( + "import sys\n" + "for _ in range(16):\n" + " sys.stdout.buffer.write(b'o' * 65536)\n" + " sys.stderr.buffer.write(b'e' * 65536)\n" + "sys.stdout.buffer.flush()\n" + "sys.stderr.buffer.flush()\n" +) + + +def test_pump_does_not_deadlock_when_both_pipes_fill(repo: Path): + """1 MiB down each pipe before the child exits — far past the OS pipe buffer. + A pump that drained one stream to EOF first would hang here forever, so the + join timeout is the assertion.""" + s = _session(repo) + argv = [sys.executable, "-c", _BOTH_PIPES_BIG] + box: dict = {} + + def go() -> None: + box["event"] = run_wrapped(argv, s, repo) + + worker = threading.Thread(target=go, daemon=True) + worker.start() + worker.join(120) + + assert not worker.is_alive(), "the capture pump deadlocked on a full pipe" + ev = box["event"] + assert s.blobs.get(ev.stdout_blob) == b"o" * 65536 * 16 + assert s.blobs.get(ev.stderr_blob) == b"e" * 65536 * 16 + + +# --- 4. heartbeat ------------------------------------------------------------- + +_HEARTBEAT_RE = re.compile( + r"^didrun: \d+s \S+ \(\d+ args\) stdout=\d+B stderr=\d+B$" +) + +_NOISY_THEN_SLEEP = ( + "import sys, time\n" + "sys.stdout.write('MARKER-CONTENT-OUT\\n')\n" + "sys.stderr.write('MARKER-CONTENT-ERR\\n')\n" + "sys.stdout.flush(); sys.stderr.flush()\n" + "time.sleep(3)\n" +) + + +def test_heartbeat_is_bounded_and_carries_no_content(repo: Path, tmp_path: Path): + child = _child_script(tmp_path, "noisy_then_sleep.py", _NOISY_THEN_SLEEP) + proc = _run_cli(repo, "run", "--heartbeat", "1", "--", *child) + + assert proc.returncode == 0 + err = proc.stderr.decode() + beats = [ln for ln in err.splitlines() if ln.startswith("didrun: ")] + assert len(beats) >= 2, f"expected repeated heartbeats over a 3s child, got {beats!r}" + for line in beats: + assert _HEARTBEAT_RE.match(line), f"heartbeat off-shape: {line!r}" + # Not one byte of what the child printed may ride out on the progress line. + assert "MARKER-CONTENT-OUT" not in err + assert "MARKER-CONTENT-ERR" not in err + assert b"MARKER-CONTENT-OUT" not in proc.stdout + # And the heartbeat stays off stdout, which external gates parse. + assert "didrun: " not in proc.stdout.decode() + + +def test_heartbeat_is_off_by_default(repo: Path): + child = [sys.executable, "-c", "import time; time.sleep(2)"] + proc = _run_cli(repo, "run", "--", *child) + assert proc.returncode == 0 + assert proc.stderr == b"", f"unrequested output on stderr: {proc.stderr!r}" + + +# --- 5. show --event N --output ---------------------------------------------- + +# Synthetic, structurally identical to a real AWS key; matches redact's +# aws-access-key pattern. Never a live credential. +_FAKE_KEY = "AKIAZZ7EXAMPLE9QTEST" + + +def test_show_output_round_trips_the_recorded_bytes(repo: Path): + s = _session(repo) + ev = run_wrapped([sys.executable, "-c", _BINARY], s, repo) + recorded = s.blobs.get(ev.stdout_blob) + + proc = _run_cli(repo, "show", "--event", "0", "--output") + assert proc.returncode == 0 + assert proc.stdout == recorded + + +def test_show_output_reads_the_stderr_stream(repo: Path): + s = _session(repo) + ev = run_wrapped([sys.executable, "-c", _STDERR_ONLY], s, repo) + + proc = _run_cli(repo, "show", "--event", "0", "--output", "--stream", "stderr") + assert proc.returncode == 0 + assert proc.stdout == s.blobs.get(ev.stderr_blob) + + +def test_show_output_redacted_applies_the_redactor(repo: Path): + s = _session(repo) + run_wrapped([sys.executable, "-c", f"print({_FAKE_KEY!r})"], s, repo) + + raw = _run_cli(repo, "show", "--event", "0", "--output") + assert _FAKE_KEY.encode() in raw.stdout + + red = _run_cli(repo, "show", "--event", "0", "--output", "--redacted") + assert red.returncode == 0 + assert _FAKE_KEY.encode() not in red.stdout + assert "«redacted:aws-access-key»" in red.stdout.decode("utf-8") + + +def test_show_output_refuses_a_missing_blob_without_a_traceback(repo: Path): + s = _session(repo) + ev = run_wrapped([sys.executable, "-c", "print('gone')"], s, repo) + (s.blobs.root / ev.stdout_blob).unlink() + + proc = _run_cli(repo, "show", "--event", "0", "--output") + assert proc.returncode == 2 + assert b"Traceback" not in proc.stderr + assert b"didrun show:" in proc.stderr + + +def test_show_output_refuses_an_out_of_range_event(repo: Path): + s = _session(repo) + run_wrapped([sys.executable, "-c", "print('one')"], s, repo) + + proc = _run_cli(repo, "show", "--event", "7", "--output") + assert proc.returncode == 2 + assert b"Traceback" not in proc.stderr + + proc = _run_cli(repo, "show", "--output") + assert proc.returncode == 2 + assert b"--event" in proc.stderr + + +def test_show_output_does_not_create_a_ledger(repo: Path, tmp_path: Path): + """--output is a read-only command and stays one (P2.1's seam).""" + empty = tmp_path / "empty" + empty.mkdir() + proc = _run_cli(empty, "show", "--event", "0", "--output") + assert proc.returncode == 2 + # didrun's own refusal, not argparse's: a 2 from an unrecognised flag would + # satisfy the returncode and the missing directory for the wrong reason. + assert b"didrun show:" in proc.stderr + assert not (empty / ".didrun").exists() + + +# --- 6. exit-code fidelity ---------------------------------------------------- + + +@pytest.mark.parametrize("code", [0, 1, 42]) +def test_exit_codes_match_the_buffered_path(repo: Path, code: int): + argv = [sys.executable, "-c", f"import sys; sys.exit({code})"] + reference = subprocess.run(argv, cwd=str(repo), capture_output=True) + s = _session(repo) + ev = run_wrapped(argv, s, repo) + assert ev.exit_code == reference.returncode == code + + +def test_signal_death_records_the_same_code_as_the_buffered_path(repo: Path): + argv = [ + sys.executable, + "-c", + "import os, signal; os.kill(os.getpid(), signal.SIGTERM)", + ] + reference = subprocess.run(argv, cwd=str(repo), capture_output=True) + s = _session(repo) + ev = run_wrapped(argv, s, repo) + assert ev.exit_code == reference.returncode + assert ev.exit_code < 0, "a signal death must not be recorded as a clean exit" From 9231f7e3eba46dba888d05ac0b09144057068141 Mon Sep 17 00:00:00 2001 From: Drew Date: Wed, 29 Jul 2026 12:37:51 -0700 Subject: [PATCH 3/8] redact: score runs, not paths, and redact every field that blocks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This lands the v0.2 secrets and evidence work, plus the two defects an independent verification pass found in it. The detector tiering stands as designed: the six structured patterns block, the entropy sweep only notices. Measured over a real archive, the structured detectors had never fired once and 65 of 66 seals carried an override, so a gate that only ever produced false positives is a gate that gets switched off. A refusal is now scoped to the bytes that actually leave the machine, and findings carry a location instead of a bare count. The first defect was in the fix for that gate's false positives. Removing `/` from the entropy sweep's token class stopped a filesystem path being scored as one long token, and that part was measured. What was not measured is that `/` is also 1 of the 64 characters of base64: a credential whose slash-free runs are all shorter than the 24-character floor stopped being a candidate at all, and since replacement is driven off the findings, it stopped being redacted too. An AWS-shaped secret in a recorded argv went into refs/notes/didrun verbatim under a seal line reading "0 findings". Reproduced end to end before touching anything. So the separator is no longer decided in the character class. A run is taken over the class including `/`, and a per-run test decides whether that `/` is a separator or an alphabet member: a path is a sequence of names, base64 is not, and a name is mostly letters that are mostly lower case. One name is enough to call a run a path, which is the conservative direction — it can only return a run to segment scoring, so it cannot add a false positive the previous behaviour did not already have. Measured at exactly that: 0 new false positives over 46,403 real paths, with recall on slash-bearing base64 back from 74-87% to 97-99%. Both directions are now in MEASUREMENTS.md, and the sentence there that presented a one-directional measurement as two-directional is corrected rather than removed. The second defect was that blocking and redacting were different sets. Widening the scan to the export domain caught secrets in claim labels and delta paths and refused the seal over them — but nothing scrubbed those fields, so the override published them raw, under a refusal message that had already promised "export a redacted artifact anyway". Every exported claim string is now redacted by the same pass that reports it to the gate, so the two cannot drift apart again; the projection is declared per field in `redaction.fields`, and `scan_domain` says the fields were scanned. The seal still blocks exactly as before, which has its own test, because a fix that quietly opened the gate would be worse than the leak. Both fixes carry regression tests that were falsified: reverting each one turns the relevant tests red (8 and 4 respectively) and restoring it turns them green. Also here, from the same phase: the v2 environment fingerprint and its advisory-by-default drift comparison, claims bound to the recorded entry by chain hash so an archived or substituted ledger is visible rather than silent, interrupted flights recorded instead of lost, and the exported redaction projection made exactly reconstructible. Known and deliberately not closed, recorded in COMPAT.md rather than dropped: an HTML report recomputes a stale claim's file list against the working tree, so that list is not the note's redacted copy; the entropy sweep still misses roughly 1-3% of slash-bearing base64 by design and resists no attacker who knows the rule; and `reason`/`coverage` are exported unredacted because they carry only generated text today, which is a property of the generators and not an enforced invariant. Gates: 320 passed, 4 skipped. harness.recall VERDICT PASS, defect-class recall 100%, freeze INTACT. All changed files parse under Python 3.11. No runtime dependencies added. No private paths or internal names in shipped files. Gate runs are UNRECEIPTED - this repo keeps no didrun ledger. --- README.md | 25 +- docs/COMPAT.md | 207 +++++++- docs/MEASUREMENTS.md | 145 ++++++ docs/TRUST_MODEL.md | 40 +- src/didrun/capture.py | 142 +++++- src/didrun/claims.py | 22 + src/didrun/cli.py | 28 +- src/didrun/manifest.py | 480 ++++++++++++++++-- src/didrun/redact.py | 676 ++++++++++++++++++++++++-- src/didrun/render.py | 51 +- tests/compat/test_corpus_replay.py | 11 +- tests/test_capture_claims_manifest.py | 11 +- tests/test_credential_recall.py | 399 +++++++++++++++ tests/test_detector_tiers.py | 635 ++++++++++++++++++++++++ tests/test_env_fingerprint.py | 484 ++++++++++++++++++ tests/test_projection_contract.py | 342 +++++++++++++ tests/test_redact_render.py | 33 +- tests/test_seal_publication.py | 13 +- 18 files changed, 3630 insertions(+), 114 deletions(-) create mode 100644 tests/test_credential_recall.py create mode 100644 tests/test_detector_tiers.py create mode 100644 tests/test_env_fingerprint.py create mode 100644 tests/test_projection_contract.py diff --git a/README.md b/README.md index d92f30b..5e19445 100644 --- a/README.md +++ b/README.md @@ -76,15 +76,28 @@ didrun verify --strict # exit 0 only if every claim is recorded-exact ``` The `--strict` exit code is the CI seam: wire `didrun verify --strict` into your -pipeline and a drifted or unbacked claim fails the build. For humans, generate a -self-contained HTML evidence report: +pipeline and a drifted or unbacked claim fails the build. + +Every verdict also reports how the environment compared with the one the seal +recorded — `env: N match / M drifted / K incomparable`. Drift is advisory by +default, because a changed environment is a fact about the machine verifying, +not about whether the recorded command ran; `didrun verify --require-env-match +--strict` makes it a refusal. Evidence sealed by an older didrun is +*incomparable*, never drifted, and never refuses. + +For humans, generate a self-contained HTML evidence report: ```bash didrun verify --html evidence.html ``` -The report is a single file with zero external assets — it opens offline, prints -cleanly, and is safe to attach to a PR (secrets are redacted from every export). +The report is a single file with zero external assets — it opens offline and +prints cleanly. Redaction covers the sealed note: every exported claim string — +the `argv_preview`, the label, the pathspecs and the changed paths — is +redacted, and the projection is declared in the manifest. The HTML report +carries no argv at all and renders the note's redacted label, but a stale +claim's file list is recomputed against your working tree at verify time and is +shown as it is on disk — read it before you attach it to a PR. ## Use it with your coding agent @@ -134,8 +147,8 @@ correct or that the command meaningfully tested anything. - `didrun run -- ` — record a wrapped execution (complete capture). - `didrun claim [--path ]` — declare a structured claim. -- `didrun seal` — compile a commit-bound evidence manifest (redacts secrets; blocks by default if found). -- `didrun verify [--strict] [--html ]` — check claims against recorded evidence. +- `didrun seal` — compile a commit-bound evidence manifest (redacts secrets; refuses when a structured secret is found in what it is about to publish, warns about the rest). +- `didrun verify [--strict] [--require-env-match] [--html ]` — check claims against recorded evidence. - `didrun show [--session]` — show the verdict, or the recorded session history. ## Capture is tiered and honest about coverage diff --git a/docs/COMPAT.md b/docs/COMPAT.md index 0939db7..d4769ff 100644 --- a/docs/COMPAT.md +++ b/docs/COMPAT.md @@ -84,16 +84,179 @@ note it *can* parse but whose version is too new is not skipped: it aborts the resolution. The fallback returns the first note whose sealed tree matches; it does not currently detect a second match. +## The exported redaction projection is declared, not inferred + +Each claim entry in a v0.2 note carries an additive `redaction` block: + +```json +"redaction": {"projection_version": 1, "detector_set_version": 2, + "applied": [{"argv_index": 2, "start": 10, "end": 59, + "detector_id": "env-secret-assignment", + "withheld": false}]} +``` + +`applied` names the exact source spans that were replaced in `argv_preview`, in the +source's own coordinates. It exists because the projection used to be readable only by +inference, and inference was wrong: which tokens the entropy sweep flagged turned on +letter frequency, so the same argv shape exported differently from one seal to the next +and a downstream validator had to keep a matrix of didrun's historical behaviour. A +consumer now reads the contract. `projection_version` bumps when the rule turning +findings into replaced spans changes; `detector_set_version` bumps when the detectors do. + +**The marker string alone is not authenticating.** `«redacted:openai-key»` is plain text +that a recorded argv is free to contain, and didrun does not escape a pre-existing +marker. A consumer that wants to know whether a preview was really redacted — and where +— reads `redaction.applied`; it must not pattern-match the preview. Keyed markers would +make the marker itself unforgeable and are deliberately not built: they need a per-ledger +key store, and a per-ledger key destroys the cross-ledger comparability that motivated +them. + +`withheld: true` marks the honest-absence path: the element was dropped whole rather than +rewritten, and the preview shows `«withheld:»` in its place. It is never an +empty string, because an empty preview and a suppressed one are different facts. + +This block is additive and read with `.get()`, so it does not bump `MANIFEST_VERSION` and +older notes that lack it are read exactly as before. **v0.2 projections differ from +v0.1's for any input with overlapping findings**: v0.1 replaced overlapping spans +end-first, which deleted the text after the outermost span, so an argv like +`run --key MY_TOKEN= && echo done` exported with `&& echo done` silently gone. +v0.2 merges findings into disjoint maximal spans first, and the result is exactly +reconstructible — substituting each marker back with the source span `applied` names +reproduces the input byte for byte. + +**`detector_set_version` is 2 from v0.2's detector work onward.** The entropy sweep no +longer decides in the character class whether `/` is a path separator; it decides per +run, scoring a filesystem path in segments and a base64 blob whole, and every detector +now declares a tier. That is a change to what the detectors match, which is exactly what +the number is for — a consumer comparing a `1` projection with a `2` one is told the +detectors differ instead of having to infer it. `projection_version` stays 1: the +span-partition algorithm did not change. + +### The projection covers the claim fields, not only argv + +A v0.2 `redaction` block carries a second, additive list beside `applied`: + +```json +"redaction": {"projection_version": 1, "detector_set_version": 2, + "applied": [{"argv_index": 3, "start": 9, "end": 49, ...}], + "fields": [{"field": "claim.label", "start": 17, "end": 57, + "detector_id": "github-token", "withheld": false}]} +``` + +`applied` is unchanged and stays argv-addressed by `argv_index`; `fields` is addressed +by field name, because a label has no argv index. Field names in v0.2 are `claim.label`, +`claim.pathspecs[i]` and `delta[i].path`. A reader that does not know the key sees +exactly the argv projection it always did. + +This exists because blocking and redacting were not the same set. A structured secret in +a claim label refused the seal — and the refusal says *"re-run with `--allow-secrets` to +export a redacted artifact anyway"* — but on the override the label was serialized into +`refs/notes/didrun` verbatim, a ref that gets pushed. The claim fields are now scrubbed +by the same pass that reports them to the gate, so the two cannot drift apart again. +`scan_domain` says so: it reads `claim-bound-events+claim-fields+manifest` rather than +`claim-bound-events+manifest`. A stored note carrying the older string is still read +normally; the string is a description of that seal, not a compatibility gate. + +## The manifest records what the secrets pass scanned + +A v0.2 manifest carries an additive top-level `secrets` block beside `secrets_override`: + +```json +"secrets": {"scan_domain": "claim-bound-events+claim-fields+manifest", "events_scanned": 1, + "events_total": 7, "bytes_scanned": 4213, "detector_set_version": 2, + "findings_by_tier": {"block": 0, "notice": 3}, + "blocked": false, "overridden": false} +``` + +It is additive, defaults to `{}`, and is read with `.get()`, so it does **not** bump +`MANIFEST_VERSION` and a stored note that lacks it reads exactly as before. The compat +harness's per-version additive-key allowlist carries `secrets` for both v1 and v2 — the +deliberate one-line diff that keeps the note round-trip leg honest instead of red across +every stored note at once. + +`secrets_override` is untouched: same name, same type, same meaning — *this seal would +have been refused and the operator overrode it*. It is deliberately not replaced by a +dict, because a v0.1 binary reads it with `.get("secrets_override", False)` and a dict +there is truthy, which would make every clean v0.2 seal print "sealed with +--allow-secrets" on an older reader. What `secrets` adds is the domain behind the +boolean: `false` used to be the same word for "nothing was found" and "almost nothing was +scanned", and `events_scanned` / `events_total` now say which. + +**What blocks changed direction in v0.2.** A refusal is scoped to the export domain — +the argv of each claim-bound event plus the serialized manifest, which carries the claim +labels and delta paths that were previously published entirely unscanned. Findings +confined to the local output blobs, and every `notice`-tier finding, warn on stderr and +do not stop the seal. So some previously-refused seals now pass and some previously +passing ones now refuse; both directions are measured in MEASUREMENTS.md. + ## The environment fingerprint is versioned in band -`env_fingerprint()` returns `v:<16 hex chars>`. The prefix exists because the digest's -*shape* does not change when its key set does — two bare digests over different key sets -look comparable and are not. +`env_fingerprint()` returns `v2:<16 hex gate>:path=<8 hex>`. The prefix exists because the +digest's *shape* does not change when its key set does — two bare digests over different +key sets look comparable and are not. A value with **no prefix was produced by v0.1** and is **incomparable** with a versioned one. It is not evidence that the environment drifted, and nothing may report it as such; `capture.fingerprint_version()` returns `None` for it so a consumer can tell the two -apart. +apart. `capture.fingerprint_gate()` returns `None` for the same values, so a consumer +cannot compare one by accident. + +### v2: what the gate binds, and what it deliberately does not + +The **gate** — the only portion any comparison reads — is a digest of `LANG`, `SHELL`, +`TZ`, `VIRTUAL_ENV` and the **process umask**. Two of v0.1's keys were dropped: + +- **`PATH`** varies with terminal tab, `direnv`, `nix develop`, tmux and homebrew + shellenv ordering, and flips on any venv activation `VIRTUAL_ENV` already records. A + gate that refuses in a new shell is ignored within a week. PATH is instead recorded as + its own truncated digest in the `path=` suffix: drift in it is **advisory**, reported + as a suffix on a verdict line, and can never turn a match into drift. +- **`PWD`** is not the child's cwd. A wrapped command always runs with `cwd=`, and + `Event.cwd` records that same resolved path; `PWD` is the caller's shell variable, + which can be stale and can be absent when didrun is spawned by a launcher. Once the + fingerprint is compared, gating on it would mean two byte-identical executions of the + same command against the same repo disagree because the operator was standing + somewhere else. + +`umask` is added because it is the fact whose absence made two green wrapper events under +different umasks indistinguishable in the receipt. It is read **once per process** +(`capture.process_umask`) and reused: reading a umask is a read-modify-write on +process-global state, so a per-event read would open that window on every wrapped command. + +**The `path=` digest is a deviation, stated plainly.** The design review asked for `PATH` +to be recorded *unhashed in the event body*. A raw `PATH` there would enter the v1 chain +preimage — which is frozen — and would sit one redaction pass away from a published note, +against the export-scoped secrets rule. A digest in the existing free-form `str` field +gives the same drift visibility at zero schema cost and leaks less. What it costs is +legibility: a reader learns *that* PATH changed, never *how*. + +### The comparison verify makes, and what it may not do + +`verify` computes this process's fingerprint once and compares the **gate** against the +one each claim's `evidence` block recorded. Four outcomes: + +| `env_status` | when | +|---|---| +| `match` | same fingerprint version, equal gates | +| `env-drift-since-seal` | same fingerprint version, different gates | +| `incomparable` | different fingerprint versions, or a bare v0.1 digest | +| `not-recorded` | the note bound no fingerprint for the claim | + +**Only `env-drift-since-seal` can ever refuse, and only under `--require-env-match`.** +By default drift is reported in the verdict and does not change `all_verified`, because +a changed environment is a fact about this machine now, not about whether the recorded +command ran. `incomparable` never refuses under either mode — it is the state every note +published before v0.2 lands in, and refusing on it would convert "this note predates the +current key set" into a failure. + +**No archived evidence changes.** `Event`'s field set is untouched, so no stored +`entry_hash` moves; archived events keep their v0.1 and v1 fingerprint strings verbatim; +and a note with no `evidence` block is compared not at all. Note also that this makes +green **harder** to reach, not easier: anyone measuring whether v0.2 reduced failed +attempts must be told that `--require-env-match` adds a way to fail. A fingerprint that +can never refuse is not binding anything. + +## Tree-digest semantics changed in v0.2 (release-note level) ## Tree-digest semantics changed in v0.2 (release-note level) @@ -184,6 +347,36 @@ no Windows leg exercises it. `--redacted`). It is a reader over the existing store, adds no format, and re-hashes on read, so a corrupted blob is a refusal rather than bytes presented as the record. +## Known, not closed: the redaction domain + +Found while fixing the two defects above, judged real, and deliberately left open rather +than fixed quietly in the same change. Each is a bounded statement about where redaction +stops, not a plan. + +1. **An HTML report's file list is recomputed, not read from the note.** `verify` + regrades a claim against the working tree, so a `stale` claim's changed paths in + `didrun verify --html` come from the local repository at verify time — they are not + the note's redacted copy. The note itself is redacted; the HTML file is an export and + this path is not. It predates the claim-field work and is unchanged by it. Redacting + at render time is the obvious fix and is not taken here because it also degrades the + local terminal's `stale` drill-down, which is the one thing that grade exists to show; + choosing between them needs its own measurement. README.md carries the caveat in the + sentence that describes the report. + +2. **The entropy sweep misses roughly 1–3% of slash-bearing base64 credentials.** A + credential whose own slashes chop it into pieces all shorter than the 24-character + floor, *and* leave one piece looking like a name, is scored as a path and missed. That + is the measured price of biasing the path test toward paths (MEASUREMENTS.md), and it + is a `notice`-tier detector either way. Nothing in the rule resists an attacker who + knows it — the rule is published here, and satisfying it deliberately is trivial. + +3. **`reason` and `coverage` are exported unredacted.** Both are didrun's own generated + text — grade names, counts, path *counts* — and carry no operator or repository + string today. That is a property of the current generators, not an enforced + invariant: a future `reason` that interpolates a path or a label would leave the + redacted set without anything failing. The whole-blob manifest scan still *blocks* on + such a field, so the failure mode would be a blocked seal, not a silent leak. + ## Two things v0.2 does not close Stated here because a trust tool's silence about its own limits is the same defect as an @@ -199,6 +392,12 @@ overclaim. richest secret carrier on a developer machine into the chain preimage. A declared clean-environment mode is the honest shape for this, and it is not built. + **The v2 environment fingerprint does not close this and must not be described as + closing it.** It binds five facts (`LANG`, `SHELL`, `TZ`, `VIRTUAL_ENV`, umask) and + records a PATH digest. `GOFLAGS` is in none of them, and adding it would only move the + line — the list above has no end. What the fingerprint buys is that a *change* in + those five facts between seal and verify is visible instead of silent. + 2. **The ledger has no retention or purge.** Captured stdout and stderr are stored as content-addressed blobs under `.didrun/objects`. A credential that was scrubbed out of git history survives there as a loose object until the ledger directory is removed by diff --git a/docs/MEASUREMENTS.md b/docs/MEASUREMENTS.md index 1118e46..52443c7 100644 --- a/docs/MEASUREMENTS.md +++ b/docs/MEASUREMENTS.md @@ -93,6 +93,151 @@ Also measured: git notes do **not** follow `commit --amend`/rebase by default (`notes.rewriteRef` defaults exclude custom refs) — which is why manifests dual-bind to commit id **and** tree id, with tree-fallback lookup. +## Secrets detection: why the detectors are tiered (v0.2) + +Measured **2026-07-29** over a real archive of **66** sealed `refs/notes/didrun` +notes (**884** graded claims, **22,062** exported argv elements, **602,975** +characters of exported argv), read through `git notes list` + `git cat-file` +only. Counts, never tokens. + +**What the detectors actually caught, in production:** + +| Detector | Redaction markers in the 66 exported notes | +|---|---| +| `high-entropy` (entropy sweep) | **7,482**, in 60 of 66 notes | +| `aws-access-key` | 0 | +| `github-token` | 0 | +| `slack-token` | 0 | +| `openai-key` | 0 | +| `pem-private-key` | 0 | +| `env-secret-assignment` | 0 | + +The six structured detectors fired **zero times**. Every finding the tool has +ever blocked on came from the entropy sweep — and **65 of the 66 notes record +`secrets_override: true`**, i.e. the gate was overridden on 98.5% of real seals. +A blocking gate that fires only false positives is a gate that gets switched +off, which is the failure mode the tiering exists to prevent. This asymmetry is +what the two-tier split rests on, so it is a gate on the change and not a +footnote: had the structured tier fired even once here, the split would have +needed a human decision. + +**The root cause was one character.** The entropy sweep's token class included +`/`, so an entire absolute filesystem path was a single token — and a path +accumulates distinct characters much faster than any one of its segments does. +Removing `/` from the class, over real absolute paths on this machine: + +| Path population | Paths | Entropy findings before | After | +|---|---|---|---| +| Repository file paths (project A) | 556 | 135 | **0** | +| Repository file paths (project B) | 47 | 9 | **0** | +| Homebrew Cellar directories (depth ≤ 6) | 2,039 | 35 | **0** | +| **Total** | **2,642** | **179** | **0** | + +The 24-character floor and the 4.2 bits/char threshold are unchanged, so the +before/after is attributable to the candidate — what gets scored — alone. The +population sits right on the knife edge — median 4.110 and 4.088 bits/char for +the two repository populations, against a 4.2 threshold — which is why the same +tool produced wildly different finding counts on runs that differed only in +where the repository was checked out. + +**That table is one-directional, and reading it as two-directional was a +mistake that cost a real credential class.** It measures what stopped firing on +paths. It does not measure what stopped firing on secrets — and `/` is not only +a path separator, it is 1 of the 64 characters of base64. Dropping it from the +token class meant a base64 credential whose slash-free runs are all shorter +than the 24-character floor stopped being a *candidate at all*: not re-scored, +not demoted to `notice`, invisible. Replacement is driven off the findings, so +invisible also meant unredacted — published verbatim into `refs/notes/didrun` +under a seal line reading "0 findings". + +So the separator is no longer decided by the character class. A run is taken +over the class **including** `/`, and a per-run test decides whether that `/` is +a separator (score the segments) or an alphabet member (score the run whole). +The test is that a filesystem path is a sequence of **names** and base64 is not: +a segment counts as a name if it is ≥3 characters, ≥60% letters, and ≥80% of +those letters are lower case. That admits `usr`, `site-packages` and CamelCase +like `CoreServices` (10/12) or `Frameworks` (9/10); base64 is ~50% lower case by +construction and clears it only by accident. **One** name is enough to call the +whole run a path — the conservative direction, since it can only return a run to +segment scoring. + +Measured over the same population plus a wider one (system framework and +shared-library trees, a virtualenv's site-packages, `/usr/share`), and over +seeded credential populations of 20,000 samples each: + +| Path population | Paths | pre-v0.2 | `/` dropped | now | +|---|---|---|---|---| +| The 2,642 above | 2,642 | 179 | **0** | **0** | +| Wide (adds system/library trees) | 46,403 | 9,278 | **248** | **248** | + +| Credential population (detected) | pre-v0.2 | `/` dropped | now | +|---|---|---|---| +| AWS secret-access-key shape (40 chars of `[A-Za-z0-9+/]`) | 100.0% | 83.6% | **98.9%** | +| `base64(24 bytes)` = 32 chars | 99.3% | 74.6% | **97.1%** | +| `base64(32 bytes)` = 44 chars | 100.0% | 86.8% | **99.0%** | +| `base64(64 bytes)` = 88 chars | 100.0% | 99.6% | **99.9%** | + +The false-positive column is **identical** to the `/`-dropped column on both +populations — the path win is kept in full, at zero measured cost — and the +recall lost by dropping the character is mostly recovered. The 1–3% still +missing is a credential whose own slashes chop it into sub-floor pieces *and* +leave one piece looking like a name; that is the price of biasing the test +toward paths, and it is the honest side to lose on for a detector that is +`notice`-tier precisely because it is a poor discriminator. None of this is +evasion-resistant, and it is not offered as such: an attacker who knows the rule +can satisfy it deliberately. + +The entropy sweep was **demoted to `notice`, not deleted**: it is the only +detector that catches an unstructured credential, and length is not secrecy in +either direction. Per-character entropy over an *n*-character token is capped at +log2(*n*), so ~18 distinct characters clears 4.2 — and, measured the other way, +a lowercase-base36 credential sits near 4.05 and does **not** clear it. The +threshold is a poor discriminator in both directions; that is an argument for +not blocking on it, not for trusting it. + +**Both directions of the blocking change**, replayed over the same 66 notes: +under the new rule **0** would block (the 65 overridden seals become clean +seals with a warning), and **0** newly block — no claim label and no delta path +in the archive carries a structured-secret shape, even though those fields are +exported and were never scanned before. + +## Environment fingerprint: what the v2 gate keys actually do (v0.2) + +The v2 gate binds `LANG`, `SHELL`, `TZ`, `VIRTUAL_ENV` and the process umask, +and drops `PATH` and `PWD` (both were in v0.1's key set). The fingerprint is now +compared across time, so every key in it is a potential *false*-drift source and +the drop needs a number rather than a shrug. + +Measured on one machine (macOS 26.5, arm64), sampling each key across five shell +contexts and comparing each against the value the current process holds. Counts +only — no value is recorded here or anywhere in the repo: + +| Key | Contexts disagreeing with the current process (of 5) | +|---|---| +| `SHELL`, `TZ`, `VIRTUAL_ENV` | 0 | +| `LANG` | 1 (login zsh) | +| `PATH` | 1 (login bash) | + +Contexts: `zsh -lc`, `zsh -ic`, `bash -lc`, `bash -c`, `sh -c`. + +**Read this honestly: it is a weak result and it does not on its own justify +dropping `PATH`.** On this machine `PATH` was no noisier than `LANG`, which is +kept. The measurement bounds the claim rather than proving it — one machine, one +day, no `direnv`, no `nix develop`, no tmux, no second homebrew prefix, which are +the conditions under which `PATH` reordering is reported to be routine. It also +records a real cost of the keys that were kept: `LANG` differs between login and +non-login zsh here, so a seal from one and a verify from the other reports drift. +That is a true statement about the environment rather than a bug, and it is why +drift is advisory unless `--require-env-match` is passed. + +The load-bearing argument for the two drops is structural, not statistical: +`PATH` flips on any venv activation that `VIRTUAL_ENV` already records, and +`PWD` is not the child's cwd at all — a wrapped command always runs with +`cwd=`, so gating on `PWD` would make two byte-identical executions of the +same command against the same repo disagree because the operator was standing in +a different directory. `PATH` is still recorded, as a truncated digest beside the +gate, so its drift stays visible without being a refusal. See docs/COMPAT.md. + ## Known digest blind spots (by design, documented rather than hidden) Ignored files are invisible to every git-based digest (`tree-exact` does not diff --git a/docs/TRUST_MODEL.md b/docs/TRUST_MODEL.md index 5032fd0..5b7597d 100644 --- a/docs/TRUST_MODEL.md +++ b/docs/TRUST_MODEL.md @@ -61,14 +61,46 @@ echoed: tokens, `.env` contents, connection strings. Treat `.didrun/` and any exported bundle as **secret-bearing**: - `.didrun/` is self-ignored (a `*` gitignore inside it) so it is never committed. -- At `seal`/export, didrun scans argv and captured output for likely secrets and - **blocks export by default** if it finds any (override with `--allow-secrets`, - which is logged into the manifest, and still redacts the exported artifact). +- At `seal`/export, didrun scans two domains and **tiers** what it finds. The six + structured detectors are `block`: they fire on shape and name what they found. + The entropy sweep is `notice`: it fires on "this looked random", which over a + 24-character run is a statement about length far more than about secrecy. +- **A seal is refused only for a `block`-tier finding in the bytes about to be + published** — the argv of each claim-bound event and the serialized manifest, + claim labels and delta paths included. Override with `--allow-secrets`, which + is logged into the manifest; the exported artifact is redacted either way. +- **What blocks and what is redacted are the same fields.** Every string in a + published claim entry that carries operator- or repository-authored text (the + argv preview, the label, the pathspecs, the changed paths) is replaced by a + marker before the note is written, and the spans are declared in + `redaction.applied` / `redaction.fields`. This is stated because it was once + false in the worst direction: a token in a claim label refused the seal and + was then published verbatim on the override, under a refusal message that had + already promised a redacted artifact. Blocking is not redacting. +- Everything else is **reported loudly and does not stop the seal**: every + `notice`-tier finding, and any `block`-tier finding confined to the recorded + output blobs, which stay in a gitignored local ledger and are never published. + Refusing over bytes that do not leave stopped work without protecting anything. +- The manifest records **what was scanned** (`secrets`: the domain, the events + scanned out of the events present, the bytes, the detector-set version, and + the findings per tier), so `secrets_override: false` is a checkable statement + instead of the same word for "nothing found" and "almost nothing looked at". +- Every finding carries a location: source kind, event index, argv index or line, + byte offset, and a **locality** fingerprint. The fingerprint deliberately does + not digest the matched token — an unkeyed digest of a short high-entropy string + would be a brute-force oracle for the thing just redacted. - Redaction applies to what *leaves* the machine (bundles, HTML reports). The raw - local ledger is left intact and stays local. + local ledger is left intact and stays local. One documented exception: an HTML + report's file list for a `stale` claim is **recomputed against your working + tree** at verify time rather than read from the note, so it is not the note's + redacted copy — see docs/COMPAT.md. The scanner is a safety net tuned to bound false positives, not a guarantee. Do not rely on it to catch a novel secret format; keep secrets out of command output. +How weak the entropy tier is in **both** directions is measured, not asserted — +see the detection tables in MEASUREMENTS.md. Those tables also bound the other +direction: the entropy sweep misses roughly 1–3% of slash-bearing base64 +credentials by design, and nothing in it resists an attacker who knows the rule. ## Honest capture limits diff --git a/src/didrun/capture.py b/src/didrun/capture.py index d51ff37..5977562 100644 --- a/src/didrun/capture.py +++ b/src/didrun/capture.py @@ -32,17 +32,50 @@ from .ledger import Event, Session from . import gitplumbing -# Environment variables whose presence/identity is safe to fingerprint. We hash -# a sorted allowlist rather than the raw environment so the fingerprint never -# leaks a secret (redaction of captured *output* is a separate pass; see redact.py). -_ENV_FINGERPRINT_KEYS = ("PATH", "SHELL", "LANG", "PWD", "VIRTUAL_ENV") +# The GATE: the environment facts whose identity is allowed to change a verdict. +# We hash a declared allowlist rather than the raw environment so the +# fingerprint never leaks a secret (redaction of captured *output* is a separate +# pass; see redact.py) — every value here is digested, never stored. +# +# IN, because each of them changes what a build produces: the locale (LANG), the +# shell a recipe may re-enter (SHELL), the active virtualenv (VIRTUAL_ENV), the +# timezone (TZ, which moves every date a build stamps), and the process umask +# (below — not an environment variable, but the same kind of fact, and the one +# whose absence made two green wrapper events under different umasks +# indistinguishable in the receipt). +# +# OUT, deliberately, and both were in v0.1's key set: +# +# PATH varies with terminal tab, direnv, `nix develop`, tmux and homebrew +# shellenv ordering, and flips on any venv activation VIRTUAL_ENV +# already records. A gate that refuses in a new shell is ignored within +# a week. It is recorded as its own digest instead (see +# env_fingerprint), so PATH drift stays inspectable without gating. +# +# PWD is NOT the child's cwd. run_wrapped always spawns with cwd=str(repo) +# and Event.cwd records that same resolved path; PWD is the caller's +# shell-maintained variable, which can be stale and can be absent when +# didrun is spawned by a launcher. While nothing read the fingerprint +# this was inert. The moment it is compared, PWD is a false-drift +# source: two byte-identical executions of the same command against the +# same repo, invoked from different directories, would disagree. +_ENV_FINGERPRINT_KEYS = ("LANG", "SHELL", "TZ", "VIRTUAL_ENV") # The fingerprint's preimage is versioned IN BAND, because the digest's shape # does not change when its key set does: two 16-hex strings over different key # sets are silently incomparable, which reads as "the environment drifted" when # the truth is "these were produced by different didrun versions". A bare digest # with no prefix was produced by v0.1 and is incomparable, not drifted. -FINGERPRINT_VERSION = 1 +# +# 2: the key set changed (umask and TZ in, PATH and PWD out) AND the string +# grew a `:path=` suffix. A v1 and a v2 value are never compared — see +# manifest._env_comparison and docs/COMPAT.md. +FINGERPRINT_VERSION = 2 + +# The marker introducing the non-gated PATH digest in a v2 fingerprint. +_PATH_MARKER = "path=" + +_UMASK: Optional[int] = None # The env var that gates Tier-2 traps. The trap does nothing unless this is set, # and it is set ONLY by `didrun run`, so an installed trap never records an @@ -50,14 +83,54 @@ DIDRUN_ACTIVE_ENV = "DIDRUN_SESSION" -def env_fingerprint(env: Optional[dict] = None) -> str: +def process_umask() -> int: + """This process's umask, read ONCE and reused for every event. + + There is no read-only way to read a umask: the read is a read-modify-write + (``os.umask(0)`` then restore), and between the two calls this process's + umask is 0 — process-global state, not thread-local, so a sibling thread + forking in that window hands its child the wrong one. Reading once per + process opens that window once, before the first child is spawned, instead + of once per wrapped command. The umask is a property of the process, not of + the command, so re-reading it per event would buy nothing anyway. + + ``run_wrapped`` primes this before spawning, so the value recorded on an + event is the umask the child actually inherited. + """ + global _UMASK + if _UMASK is None: + old = os.umask(0) + os.umask(old) + _UMASK = old + return _UMASK + + +def env_fingerprint(env: Optional[dict] = None, umask: Optional[int] = None) -> str: + """Digest the environment facts that can change what a build produces. + + Returns ``v2:<16 hex gate>:path=<8 hex>``. The GATE is the only part a + comparison may read (``fingerprint_gate``); the ``path=`` digest records + PATH so its drift stays INSPECTABLE without becoming a gate — see + ``_ENV_FINGERPRINT_KEYS`` for why, and docs/COMPAT.md for why a digest and + not the raw string. + + Every input is hashed and none is stored: the fingerprint must never become + the place where a secret-bearing environment variable lands verbatim in a + note. That is also why the whole environment is not fingerprinted, and it is + the reason an inherited-environment false green (``GOFLAGS=-exec=…``) is a + documented NON-guarantee rather than something this closes. + """ env = env if env is not None else os.environ - parts = [] - for key in _ENV_FINGERPRINT_KEYS: - val = env.get(key, "") - parts.append(f"{key}={val}") - digest = hashlib.sha256("\n".join(parts).encode("utf-8", "replace")).hexdigest()[:16] - return f"v{FINGERPRINT_VERSION}:{digest}" + umask = process_umask() if umask is None else umask + parts = [f"{key}={env.get(key, '')}" for key in _ENV_FINGERPRINT_KEYS] + # Named like the env parts, so the preimage stays one readable list and a + # future key called "umask" could not collide with it silently. + parts.append(f"umask={umask:04o}") + gate = hashlib.sha256("\n".join(parts).encode("utf-8", "replace")).hexdigest()[:16] + path_digest = hashlib.sha256( + env.get("PATH", "").encode("utf-8", "replace") + ).hexdigest()[:8] + return f"v{FINGERPRINT_VERSION}:{gate}:{_PATH_MARKER}{path_digest}" def fingerprint_version(value: str) -> Optional[int]: @@ -66,8 +139,11 @@ def fingerprint_version(value: str) -> Optional[int]: ``None`` means the value came from a didrun that predates the prefix, so it is INCOMPARABLE with a versioned one — a consumer must not read a mismatch between the two as environment drift. + + A v2 value carries a second colon (the ``:path=`` suffix); only the leading + ``v`` field is read here, so both shapes parse. """ - prefix, sep, _digest = value.partition(":") + prefix, sep, _rest = value.partition(":") if not sep or not prefix.startswith("v"): return None digits = prefix[1:] @@ -76,6 +152,36 @@ def fingerprint_version(value: str) -> Optional[int]: return int(digits) +def fingerprint_gate(value: str) -> Optional[str]: + """The comparable portion of a fingerprint, or None when there is none. + + ``None`` is INCOMPARABLE and a consumer must never report it as drift: it + means a bare v0.1 digest with no version prefix, or a value whose gate field + is empty. The ``:path=`` suffix is excluded on purpose — PATH is recorded to + be looked at, never to be gated on. + """ + if fingerprint_version(value) is None: + return None + _prefix, _sep, rest = value.partition(":") + gate, _sep2, _suffix = rest.partition(":") + return gate or None + + +def fingerprint_path_digest(value: str) -> Optional[str]: + """The non-gated PATH digest carried by a v2 fingerprint, or None. + + Advisory only, in every consumer. A v1 fingerprint carries none, and a + difference here is reported as PATH drift, never as environment drift. + """ + if fingerprint_version(value) is None: + return None + _prefix, _sep, rest = value.partition(":") + _gate, sep, suffix = rest.partition(":") + if not sep or not suffix.startswith(_PATH_MARKER): + return None + return suffix[len(_PATH_MARKER):] or None + + # --- Tier 0: the output pump -------------------------------------------------- # # v0.1 ran the child through subprocess.run(capture_output=True), which buffers @@ -530,6 +636,12 @@ def run_wrapped( repo = Path(repo or os.getcwd()) ledger_objects = session.blobs.root # digest objects live beside blobs + # Read the umask BEFORE the child is spawned, so the value the event records + # is the one the child inherited, and so the read-modify-write window this + # opens (see process_umask) is never concurrent with a fork. Cached from + # here on: the fingerprint below reuses it rather than re-reading. + process_umask() + tree_before = gitplumbing.tree_digest(repo, ledger_objects) submod = gitplumbing.submodule_dirty(repo) @@ -610,6 +722,10 @@ def pty_transcript(argv: list[str], session: Session, repo: Optional[Path] = Non """ import pty + # Before pty.spawn, for the same reason run_wrapped primes it before Popen: + # the recorded umask must be the one the child inherited. + process_umask() + repo = Path(repo or os.getcwd()) chunks: list[bytes] = [] diff --git a/src/didrun/claims.py b/src/didrun/claims.py index e79eda7..c7ed628 100644 --- a/src/didrun/claims.py +++ b/src/didrun/claims.py @@ -54,6 +54,21 @@ # is never admitted to the set --strict accepts. GRADE_WITNESS_UNAVAILABLE = "witness-unavailable" +# The environment comparison verify makes ACROSS TIME — between the fingerprint +# the seal recorded and the one this process computes. These are NOT grades: a +# drifted environment does not make a recorded command un-run, so they never +# enter the grading ladder and never reach `worst_status`. +# +# The two refusals are load-bearing. `incomparable` is what a fingerprint from a +# different FINGERPRINT_VERSION (or a bare v0.1 digest) gets — different key +# sets are not measuring the same thing, and calling that drift would make every +# archived note look like the environment moved. `not-recorded` is a note that +# bound no fingerprint at all, which is a different fact again. +ENV_MATCH = "match" +ENV_DRIFT = "env-drift-since-seal" +ENV_INCOMPARABLE = "incomparable" +ENV_NOT_RECORDED = "not-recorded" + class ClaimError(Exception): """A claim declaration violated an invariant.""" @@ -108,6 +123,11 @@ class GradeResult: facts: they say whether this verdict was checked against the specific recorded entry the seal named, and what that seal graded. ``grade()`` never sets them, so a sealed note carries them at their defaults. + + ``env_status`` / ``env_reason`` are verify-side in the same way, and they + are deliberately NOT in ``to_dict``: the seal has nothing to compare against + itself, so writing them would put a permanently-default key in every + published note. """ claim: Claim @@ -118,6 +138,8 @@ class GradeResult: exit_code: Optional[int] = None evidence_bound: bool = False sealed_grade: Optional[str] = None + env_status: str = ENV_NOT_RECORDED + env_reason: str = "" def to_dict(self) -> dict: return { diff --git a/src/didrun/cli.py b/src/didrun/cli.py index d402138..e419de5 100644 --- a/src/didrun/cli.py +++ b/src/didrun/cli.py @@ -122,9 +122,16 @@ def cmd_seal(args) -> int: print(f"didrun seal: {exc}", file=sys.stderr) return 2 verified = sum(1 for c in m.claims if c["grade"] in ("tree-exact", "scope-exact")) + # The tier split is reported even when nothing blocked: "0 findings" and + # "12 findings, none of them blocking" are different facts about the same + # green seal, and only one of them is worth a second look. + by_tier = (m.secrets or {}).get("findings_by_tier", {}) + blocking = by_tier.get("block", 0) + noticed = by_tier.get("notice", 0) print( f"sealed manifest for {m.commit[:12]} (tree {m.tree[:12]}) " - f"{verified}/{len(m.claims)} claims recorded-exact" + f"{verified}/{len(m.claims)} claims recorded-exact " + f"{blocking + noticed} findings ({blocking} block / {noticed} notice)" + (" [--allow-secrets]" if m.secrets_override else "") ) return 0 @@ -134,7 +141,12 @@ def cmd_verify(args) -> int: repo = Path(args.repo or os.getcwd()) session = _session(repo, readonly=True) try: - report = _manifest.verify(session, repo, commitish=args.commit or "HEAD") + report = _manifest.verify( + session, + repo, + commitish=args.commit or "HEAD", + require_env_match=args.require_env_match, + ) except _manifest.ManifestError as exc: # Evidence this binary cannot read is a graded refusal, not a crash. 2, # not --strict's 1: "could not read the manifest" is a different fact @@ -198,8 +210,7 @@ def _show_output(session: Session, args) -> int: print(f"didrun show: {exc}", file=sys.stderr) return 2 if args.redacted: - text, _findings = redact.redact(data) - data = text.encode("utf-8", "replace") + data = redact.redact(data).text.encode("utf-8", "replace") out = getattr(sys.stdout, "buffer", None) if out is not None: out.write(data) @@ -302,6 +313,15 @@ def build_parser() -> argparse.ArgumentParser: pv = sub.add_parser("verify", help="verify a commit's claims against recorded evidence") pv.add_argument("--commit", help="commit to verify (default: HEAD)") pv.add_argument("--strict", action="store_true", help="exit nonzero unless all claims are recorded-exact (CI mode)") + pv.add_argument( + "--require-env-match", + action="store_true", + help=( + "treat environment drift since the seal as a refusal, not a note " + "(with --strict, a drifted claim exits nonzero). Fingerprints from " + "an older didrun are incomparable and never count as drift" + ), + ) pv.add_argument("--html", help="also write an HTML report to this path") pv.add_argument("--quiet", action="store_true", help="with --html, suppress the terminal verdict") pv.set_defaults(func=cmd_verify) diff --git a/src/didrun/manifest.py b/src/didrun/manifest.py index 2970a97..ca411da 100644 --- a/src/didrun/manifest.py +++ b/src/didrun/manifest.py @@ -11,19 +11,39 @@ Verify resolves a manifest by commit id, falls back to tree id, recomputes the grades, and returns a structured report with CI-friendly exit semantics. It is a pre-merge / PR-head tool: squash-merged trees grade stale/unknown by design. + +The note is the only artifact that leaves the machine, so the redaction it +carries is declared rather than inferred: each claim entry gets a `redaction` +block naming the projection version, the detector-set version, and the exact +source spans that were replaced. A consumer reads that block. The marker string +alone is NOT authenticating — a recorded argv can contain the literal text +`«redacted:openai-key»` and nothing escapes it — so "is this a real redaction or +did the command print that?" is answered by `redaction.applied`, never by +pattern-matching the preview. Keyed markers would close the forgery gap; they +are out of scope (see docs/COMPAT.md). """ from __future__ import annotations import json import subprocess +import sys from dataclasses import dataclass, field from pathlib import Path from typing import Optional from .ledger import Session, canonical_json, _append_lock, _open_private_append -from .claims import GRADE_WITNESS_UNAVAILABLE, Claim, GradeResult, grade -from . import gitplumbing, redact +from .claims import ( + ENV_DRIFT, + ENV_INCOMPARABLE, + ENV_MATCH, + ENV_NOT_RECORDED, + GRADE_WITNESS_UNAVAILABLE, + Claim, + GradeResult, + grade, +) +from . import capture, gitplumbing, redact # 2: a claim may carry an `evidence` block naming the recorded entry that backed # it, and verify CHECKS that binding instead of regrading by index alone. The @@ -46,6 +66,11 @@ class Manifest: claims: list[dict] # graded results (serialized) coverage: dict secrets_override: bool = False + # What the secrets pass actually looked at. `secrets_override: false` used + # to be unfalsifiable — it is the same word for "nothing was found" and + # "almost nothing was scanned". This says which. Additive, defaults to {}, + # read with .get(): no MANIFEST_VERSION bump (docs/COMPAT.md). + secrets: dict = field(default_factory=dict) def to_json(self) -> bytes: return canonical_json( @@ -56,6 +81,7 @@ def to_json(self) -> bytes: "claims": self.claims, "coverage": self.coverage, "secrets_override": self.secrets_override, + "secrets": self.secrets, } ) @@ -83,6 +109,7 @@ def from_json(cls, data: bytes) -> "Manifest": claims=d["claims"], coverage=d["coverage"], secrets_override=d.get("secrets_override", False), + secrets=d.get("secrets", {}), ) @@ -128,8 +155,12 @@ def seal( ) -> Manifest: """Compile, redact, and attach a manifest for ``commitish``. - Raises ``redact.SecretsBlocked`` if secrets are found and ``allow_secrets`` - is False. On override, the exported artifact is still redacted and the + Raises ``redact.SecretsBlocked`` if a BLOCK-tier detector fires on the + EXPORT domain and ``allow_secrets`` is False. Notice-tier findings, and + block-tier findings confined to the local output blobs, warn loudly on + stderr and do not stop the seal — the blobs stay in a gitignored ledger and + are not published, so refusing over them stopped work without protecting + anything. On override, the exported artifact is still redacted and the override is recorded in the manifest. """ repo = Path(repo) @@ -164,18 +195,57 @@ def seal( _gc_durable_copy(session, results) - # Secrets pass over argv + backing output blobs before anything is exported. - findings = _scan_for_secrets(session, results) - if findings and not allow_secrets: - raise redact.SecretsBlocked(findings) - + # Secrets pass. The export domain is compiled FIRST so the scan runs over + # the bytes that would actually be published — claim labels and delta paths + # included, which the argv+blobs scan never covered. Nothing is written + # until this clears: a blocked seal must leave the watermark and the note + # exactly where they were, which is what makes retrying one free. + # + # The two fields absent here are the two this pass GENERATES — + # `secrets_override` and `secrets` — and they are the scan's own counters, + # versions and booleans, never operator- or command-authored text. Every + # field that carries content from outside is in these bytes, and a field + # added later is in them too, without anyone remembering to add it. + # + # Compiling the export domain also REDACTS it, and the findings that come + # back are what the gate blocks on. A field cannot be blocked-but-published + # (the label bug) or published-but-unscanned, because one pass does both. + redacted = [ + _redact_result(r, session, entries, claim_index=i) + for i, r in enumerate(results) + ] + claims_payload = [rc.payload for rc in redacted] + field_findings = [f for rc in redacted for f in rc.findings] + field_bytes = sum(rc.bytes_scanned for rc in redacted) + coverage = _coverage_statement(session) + export_bytes = canonical_json( + { + "version": MANIFEST_VERSION, + "commit": commit, + "tree": tree, + "claims": claims_payload, + "coverage": coverage, + } + ) + scan = _scan_for_secrets( + session, results, export_bytes, field_findings, field_bytes + ) + if scan.blocking and not allow_secrets: + raise redact.SecretsBlocked(scan.blocking) + _warn_unblocked_secrets(scan) + + # `secrets_override` keeps its exact v0.1 meaning: "this seal would have + # been refused and the operator overrode it". The set that would refuse is + # the blocking set, which is what `findings` denoted before tiering existed. + overridden = bool(scan.blocking) and allow_secrets manifest = Manifest( version=MANIFEST_VERSION, commit=commit, tree=tree, - claims=[_redact_result(r, session, entries) for r in results], - coverage=_coverage_statement(session), - secrets_override=bool(findings) and allow_secrets, + claims=claims_payload, + coverage=coverage, + secrets_override=overridden, + secrets=scan.to_dict(overridden), ) # Publication and the watermark are one atomic pair. A published note with @@ -232,6 +302,12 @@ class VerifyReport: chain_status: str = "absent" # intact | broken | unverifiable | empty | absent chain_broken_index: Optional[int] = None # set only when status is "broken" chain_reason: str = "" + # Whether environment drift REFUSES rather than reports. Off by default: + # drift is a statement about this machine now, not about whether the + # recorded command ran, so making it fail --strict silently would flip green + # units red on routine noise. The flag is recorded on the report so a reader + # can tell which of the two verdicts they are looking at. + require_env_match: bool = False @property def chain_faulted(self) -> bool: @@ -266,11 +342,43 @@ def all_verified(self) -> bool: # Membership is deliberately NOT widened. `witness-unavailable` reports # a grade this run did not check, so admitting it here would make # --strict pass on evidence nobody has seen. + # + # Environment drift subtracts from this set ONLY under + # --require-env-match. `incomparable` never does, under either mode: it + # is the state every archived v1 note lands in, and refusing on it would + # turn "this note predates the current fingerprint" into a failure. if self.chain_faulted: return False - return bool(self.results) and all( - r.grade in ("tree-exact", "scope-exact") for r in self.results - ) + if not self.results: + return False + for r in self.results: + if r.grade not in ("tree-exact", "scope-exact"): + return False + if self.require_env_match and r.env_status == ENV_DRIFT: + return False + return True + + @property + def env_counts(self) -> dict: + """How the sealed environment fingerprints compared, by state. + + Always all four keys, including zeros: "0 drifted" and "not checked" + are different facts, and a counter that disappears when it is zero + cannot say the first one. + """ + counts = { + ENV_MATCH: 0, + ENV_DRIFT: 0, + ENV_INCOMPARABLE: 0, + ENV_NOT_RECORDED: 0, + } + for r in self.results: + counts[r.env_status] = counts.get(r.env_status, 0) + 1 + return counts + + @property + def env_drift_count(self) -> int: + return self.env_counts[ENV_DRIFT] @property def evidence_bound_count(self) -> int: @@ -287,12 +395,24 @@ def total(self) -> int: return len(self.results) -def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyReport: +def verify( + session: Session, + repo: Path, + commitish: str = "HEAD", + require_env_match: bool = False, +) -> VerifyReport: """Resolve and re-grade a manifest for ``commitish``. Resolves by commit id, then by tree id (survives amend/rebase). Degrades a claim to unknown ("referenced object gc'd") rather than erroring. + The sealed environment fingerprint is compared against this process's, once + per verify. That comparison is ACROSS TIME, which is the only axis on which + it says anything: comparing the fingerprints within a claim's own events is + vacuous, because every claim didrun's CLI can declare binds exactly one + event and one event trivially agrees with itself. Drift is reported and does + not change the verdict unless ``require_env_match`` is set. + A claim the note bound to a specific recorded entry is only regraded when the live ledger still holds that entry; otherwise the sealed grade is reported as ``witness-unavailable`` and --strict refuses it. This detects @@ -325,6 +445,7 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor chain_status=chain_status, chain_broken_index=chain_index, chain_reason=chain_reason, + require_env_match=require_env_match, ) # Re-grade the MANIFEST'S OWN claims against its sealed tree (deterministic @@ -334,12 +455,24 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor entries = list(session.entries()) events = [e.event for e in entries] ledger_objects = session.blobs.root + # Once per verify, not once per claim: it is a property of this process, and + # reading the umask inside it is a read-modify-write (capture.process_umask). + current_fingerprint = capture.env_fingerprint() results: list[GradeResult] = [] for c in manifest.claims: claim = Claim.from_dict(c["claim"]) - results.append( - _verify_claim(c, claim, manifest, entries, events, repo, ledger_objects) + result = _verify_claim( + c, claim, manifest, entries, events, repo, ledger_objects ) + # Assigned around the ladder, like witness-unavailable: the environment + # comparison is a fact about this run, not a grade the evidence earns. + # It is set on EVERY path, including witness-unavailable — the sealed + # fingerprint is in the note whether or not the ledger still holds the + # entry it names. + result.env_status, result.env_reason = _env_comparison( + c.get("evidence"), current_fingerprint + ) + results.append(result) return VerifyReport( commit=manifest.commit, tree=manifest.tree, @@ -351,6 +484,89 @@ def verify(session: Session, repo: Path, commitish: str = "HEAD") -> VerifyRepor chain_status=chain_status, chain_broken_index=chain_index, chain_reason=chain_reason, + require_env_match=require_env_match, + ) + + +def _env_comparison(evidence, current: str) -> tuple[str, str]: + """Compare the fingerprint the seal recorded against this process's. + + Four outcomes, and three of them are refusals to compare. The refusals are + the reason this is safe to add to a tool with sealed evidence already in the + world: a note that bound no fingerprint, a bare v0.1 digest, and a + fingerprint written under a different FINGERPRINT_VERSION are all cases + where the two strings are not measurements of the same thing. Reporting any + of them as drift would make every note published before this unit look like + the environment moved. + + Only the GATE portion is compared. A `path=` difference is appended as an + advisory to whatever the gate said — never promoted to drift, and never able + to turn a match into one. + """ + if not isinstance(evidence, dict): + return ( + ENV_NOT_RECORDED, + "this note bound no environment fingerprint to the claim", + ) + sealed = evidence.get("env_fingerprint") + if not isinstance(sealed, str) or not sealed: + return ( + ENV_NOT_RECORDED, + "the sealed evidence block carries no environment fingerprint", + ) + + sealed_version = capture.fingerprint_version(sealed) + current_version = capture.fingerprint_version(current) + if sealed_version is None: + return ( + ENV_INCOMPARABLE, + "the sealed fingerprint is an unversioned v0.1 digest, which is " + "incomparable with this one — not evidence that anything drifted", + ) + if current_version is None or sealed_version != current_version: + return ( + ENV_INCOMPARABLE, + f"the sealed fingerprint is version {sealed_version} and this didrun " + f"computes version {current_version}; different key sets are " + "incomparable, not drifted", + ) + + sealed_gate = capture.fingerprint_gate(sealed) + current_gate = capture.fingerprint_gate(current) + if sealed_gate is None or current_gate is None: + return ( + ENV_INCOMPARABLE, + "a fingerprint carries no readable gate portion to compare", + ) + + advisory = _path_advisory(sealed, current) + if sealed_gate == current_gate: + return ENV_MATCH, advisory + drifted = ( + f"environment changed since seal (sealed gate {sealed_gate}, " + f"now {current_gate})" + ) + return ENV_DRIFT, f"{drifted}; {advisory}" if advisory else drifted + + +def _path_advisory(sealed: str, current: str) -> str: + """PATH drift, as text. Never a gate, never drift, never a verdict. + + PATH varies with terminal tab, direnv, `nix develop`, tmux and homebrew + shellenv ordering, so gating on it produces a refusal that fires in a new + shell and is ignored within a week. It is recorded as its own digest so the + variation stays visible to a reader who is looking for it. + + Digests, not values: what changed is not said, because saying it would put a + developer machine's PATH in a published note. + """ + sealed_path = capture.fingerprint_path_digest(sealed) + current_path = capture.fingerprint_path_digest(current) + if not sealed_path or not current_path or sealed_path == current_path: + return "" + return ( + f"PATH differs (sealed path={sealed_path}, now path={current_path}) — " + "recorded, not gated" ) @@ -711,31 +927,237 @@ def _resolve_manifest( return None, "none", skipped -def _scan_for_secrets(session: Session, results: list[GradeResult]) -> list[redact.Finding]: - findings: list[redact.Finding] = [] +# What the secrets pass looked at, published so `secrets_override: false` is a +# checkable statement rather than the same word for "nothing found" and +# "almost nothing looked at". `claim-fields` is the label, the pathspecs and +# the changed paths: they used to ride into the scan only inside the +# serialized manifest, where they blocked a seal and were published anyway. +SCAN_DOMAIN = "claim-bound-events+claim-fields+manifest" + + +@dataclass +class SecretsScan: + """What the secrets pass looked at, and what it found where. + + Two domains, and the distinction is the whole point. ``export`` is what is + about to leave the machine: the argv of each claim-bound event and the + serialized manifest bytes themselves. ``local`` is what stays: the stdout + and stderr blobs, which live in a gitignored ledger and are never published. + Blocking a seal over bytes that never leave, while publishing claim labels + and delta paths entirely unscanned, was the gate pointed at the wrong half. + """ + + export: list # findings in bytes about to be written + local: list # findings in blobs that stay on this machine + events_scanned: int + events_total: int + bytes_scanned: int + + @property + def blocking(self) -> list: + """The findings that stop a seal: block-tier, in the export domain.""" + return [f for f in self.export if f.tier == redact.TIER_BLOCK] + + @property + def non_blocking(self) -> list: + """Everything found that did not stop the seal — reported, not silent.""" + return [f for f in self.export if f.tier != redact.TIER_BLOCK] + list(self.local) + + @property + def all_findings(self) -> list: + return self.export + self.local + + def to_dict(self, overridden: bool) -> dict: + by_tier = {tier: 0 for tier in redact.TIERS} + for f in self.all_findings: + by_tier[f.tier] = by_tier.get(f.tier, 0) + 1 + return { + "scan_domain": SCAN_DOMAIN, + "events_scanned": self.events_scanned, + "events_total": self.events_total, + "bytes_scanned": self.bytes_scanned, + "detector_set_version": redact.DETECTOR_SET_VERSION, + "findings_by_tier": by_tier, + "blocked": bool(self.blocking), + "overridden": overridden, + } + + +def _scan_for_secrets( + session: Session, + results: list[GradeResult], + export_bytes: bytes, + field_findings: Optional[list] = None, + field_bytes: int = 0, +) -> SecretsScan: + """Scan both domains, tagging every finding with where it is. + + Three contributions make up the export domain, and they are deliberately + not the same mechanism: + + * the argv of each claim-bound event, scanned RAW here and redacted into + the published `argv_preview` separately; + * ``field_findings`` — the claim fields (label, pathspecs, changed paths), + which arrive already located from the pass that redacted them, so what + blocks and what is scrubbed cannot drift apart; + * ``export_bytes``, the serialized manifest as it would be published, + scanned WHOLE rather than field by field. It is the backstop: a field a + later version adds and forgets to redact is still in these bytes, so it + still blocks. The redacted fields are already markers by the time they + get here, so nothing is counted twice. + + The scan covers each claim's supporting event once, not once per claim, so + ``events_scanned`` counts events rather than claim-event pairs. + """ + export: list[redact.Finding] = list(field_findings or ()) + local: list[redact.Finding] = [] events = session.events() + seen: set = set() + # Starts at the claim-field bytes rather than 0: those bytes were scanned, + # by the redaction pass, and a published counter that omitted them would be + # the same "checkable statement that is not true" this block exists to fix. + bytes_scanned = field_bytes + for r in results: idx = r.supporting_event_index - if idx is None or not (0 <= idx < len(events)): + if idx is None or not (0 <= idx < len(events)) or idx in seen: continue + seen.add(idx) ev = events[idx] - findings.extend(redact.scan(" ".join(ev.argv))) - for blob in (ev.stdout_blob, ev.stderr_blob): + for argv_index, arg in enumerate(ev.argv): + bytes_scanned += len(arg.encode("utf-8", "replace")) + export.extend( + redact.scan( + arg, + redact.FindingSource( + "argv", event_index=idx, argv_index=argv_index + ), + ) + ) + for stream, blob in (("stdout", ev.stdout_blob), ("stderr", ev.stderr_blob)): if blob and session.blobs.has(blob): - findings.extend(redact.scan(session.blobs.get(blob))) - return findings + data = session.blobs.get(blob) + bytes_scanned += len(data) + local.extend( + redact.scan(data, redact.FindingSource(stream, event_index=idx)) + ) + + bytes_scanned += len(export_bytes) + export.extend(redact.scan(export_bytes, redact.FindingSource("manifest"))) + return SecretsScan( + export=export, + local=local, + events_scanned=len(seen), + events_total=len(events), + bytes_scanned=bytes_scanned, + ) + + +def _warn_unblocked_secrets(scan: SecretsScan) -> None: + """Say, loudly, what was found and not blocked on. -def _redact_result(r: GradeResult, session: Session, entries: list) -> dict: + Silence here would make the tiering a suppression list with extra steps. + Notice-tier findings and block-tier findings confined to local blobs do not + stop a seal, but the operator is told they exist and where they are. + """ + findings = scan.non_blocking + if not findings: + return + notice = sum(1 for f in findings if f.tier == redact.TIER_NOTICE) + local_block = len(findings) - notice + print( + f"didrun seal: WARNING: {len(findings)} secret finding(s) did not block " + f"this seal ({notice} notice-tier, {local_block} block-tier in local " + f"output blobs that are not exported):", + file=sys.stderr, + ) + for line in redact.enumerate_findings(findings): + print(line, file=sys.stderr) + + +@dataclass(frozen=True) +class RedactedClaim: + """One exported claim entry, plus what redacting it found. + + ``findings`` is not diagnostic output: it is the export gate's input for + every field of this claim that is not argv. Producing the two together is + the only reason the gate cannot disagree with the publication again. + ``bytes_scanned`` keeps the manifest's published byte count honest about + the fields this pass added to the domain. + """ + + payload: dict + findings: list + bytes_scanned: int = 0 + + +def _redact_result( + r: GradeResult, session: Session, entries: list, claim_index: int = 0 +) -> RedactedClaim: d = r.to_dict() - # Redact argv in the exported claim view. - d["claim"]["argv_preview"] = redact.scrub_argv( + # Redact argv in the exported claim view, and DECLARE the projection that + # produced it. The marker string alone is not authenticating — a recorded + # argv may contain the literal marker text — so a consumer distinguishes a + # real redaction from literal text by reading `redaction.applied`, which + # names the source spans that were replaced. Additive, read with .get(): + # no MANIFEST_VERSION bump (docs/COMPAT.md). + argv_redaction = redact.redact_argv( list(_event_argv(session, r.supporting_event_index)) ) + d["claim"]["argv_preview"] = argv_redaction.argv + + # The rest of the published claim. argv used to be the only field scrubbed + # here, which made the seal's own refusal message ("re-run with + # --allow-secrets to export a redacted artifact anyway") false for every + # other field: a token in a claim label BLOCKED the seal and was then + # published verbatim into refs/notes/didrun on the override. These are the + # exported strings that carry operator- or repository-authored text, and + # each one is redacted by the same pass that reports it. + fields: list[dict] = [] + findings: list = [] + scanned = 0 + + def _field(name: str, kind: str, value: str) -> str: + nonlocal scanned + scanned += len(value.encode("utf-8", "replace")) + redaction = redact.redact_field( + name, + value, + redact.FindingSource(kind, claim_index=claim_index), + ) + fields.extend(redaction.applied) + findings.extend(redaction.findings) + return redaction.text + + d["claim"]["label"] = _field("claim.label", "label", d["claim"]["label"]) + d["claim"]["pathspecs"] = [ + _field(f"claim.pathspecs[{i}]", "pathspec", spec) + for i, spec in enumerate(d["claim"]["pathspecs"]) + ] + d["delta"] = [ + { + "status": change["status"], + "path": _field(f"delta[{i}].path", "delta-path", change["path"]), + } + for i, change in enumerate(d["delta"]) + ] + + d["redaction"] = { + "projection_version": redact.PROJECTION_VERSION, + "detector_set_version": redact.DETECTOR_SET_VERSION, + "applied": argv_redaction.applied, + # Additive and separate from `applied`, which stays argv-addressed by + # `argv_index` exactly as v0.2 consumers already read it. Entries here + # are addressed by field name instead, because a label has no argv + # index. A reader that does not know this key sees the same argv + # projection it always did. + "fields": fields, + } block = _evidence_block(entries, r.supporting_event_index) if block is not None: d["evidence"] = block - return d + return RedactedClaim(payload=d, findings=findings, bytes_scanned=scanned) def _evidence_block(entries: list, idx: Optional[int]) -> Optional[dict]: diff --git a/src/didrun/redact.py b/src/didrun/redact.py index e4d34cc..a1fc2a2 100644 --- a/src/didrun/redact.py +++ b/src/didrun/redact.py @@ -5,27 +5,210 @@ turn a trust tool into a leak ("didrun leaked our staging token" is category-killing; see docs/TRUST_MODEL.md). -This pass runs at seal/export: it scans argv and manifest-bound blobs, BLOCKS -export by default when a likely secret is found, and always redacts what reaches -an exported/rendered artifact. The raw *local* ledger is left intact (it is -gitignored and local); only what leaves the machine is scrubbed. +This pass runs at seal/export: it scans argv, the bytes about to be published, +and the manifest-bound output blobs. The raw *local* ledger is left intact (it +is gitignored and local); only what leaves the machine is scrubbed. What BLOCKS +is scoped to the export domain for the same reason: stopping a seal over bytes +that never leave the machine, while publishing claim labels unscanned, is the +gate pointed at the wrong half of the tool (see manifest.py's `_scan_for_secrets`). + +Blocking and redacting cover the SAME fields, and saying so is not a tautology +— they did not, and the gap was a leak. Every string in the published claim +entry that carries operator- or repository-authored text (the argv preview, the +label, the pathspecs, the changed paths) is replaced by a marker before it is +serialized. A field that only blocked was published raw the moment anyone +passed --allow-secrets, under a refusal message that had already promised +"export a redacted artifact anyway". So ``redact_field`` returns findings along +with text: for those fields, being scrubbed and being reported to the gate are +one act, not two passes that can drift apart. The heuristic is deliberately tuned to bound false positives — a benign 40-char git hash must NOT trip it, or the tool blocks everything and gets disabled. + +Redaction is a SPAN PARTITION, not a splice. ``scan`` produces overlapping +findings routinely (``OPENAI_API_KEY=sk-…`` matches two patterns at once), and +replacing overlapping spans one at a time silently deletes whatever followed the +outermost one — the exported record then shows a different command than the one +that ran, which is an evidence-integrity bug, not a cosmetic one. ``redact`` +therefore merges findings into disjoint maximal spans and rebuilds the string by +walking that partition, so the output is exactly reconstructible: substituting +each marker back with its recorded source span reproduces the input byte for +byte. + +The marker string alone is NOT authenticating: a recorded argv can contain the +literal text ``«redacted:openai-key»``, and nothing here escapes it. What a +consumer reads instead is the applied partition, which ``seal`` publishes into +the manifest as ``redaction.applied``. Keyed markers would close the forgery +gap and are deliberately not built (they need a per-ledger key store, and a +per-ledger key destroys the cross-ledger stability that motivated them). + +Detectors are TIERED, and the two tiers say different things. The six +structured patterns fire on shape and name what they found: ``block``. The +entropy sweep fires on "this looked random", which over a 24+ character run is +a statement about length far more than about secrecy — per-character entropy is +capped at log2(n), so roughly 18 distinct characters clears the threshold. It +is ``notice``: reported, never blocking, and never deleted, because it is the +only detector that catches an unstructured credential. + +Every finding carries its own locality (``source``, ``line``, and the offset +inside that source), because "346 likely secret(s) found" with no address is +not something an operator can act on. """ from __future__ import annotations +import hashlib import math import re from dataclasses import dataclass +from typing import Optional + +# What a finding does to an export. `block` is a structured detector that named +# what it found; `notice` is reported and warned about, never blocking. +TIER_BLOCK = "block" +TIER_NOTICE = "notice" +TIERS = (TIER_BLOCK, TIER_NOTICE) + +# Where a finding is. `argv`/`stdout`/`stderr` address a recorded event; +# `label`, `pathspec` and `delta-path` address individual exported claim fields; +# `manifest` addresses the serialized bytes about to be published. +SOURCE_KINDS = ( + "argv", + "stdout", + "stderr", + "label", + "pathspec", + "delta-path", + "manifest", +) + +# Source kinds that address ONE exported field rather than a stream of bytes. +# A line number inside them would always be 1, which is noise, not locality. +_LINELESS_KINDS = ("argv", "label", "pathspec", "delta-path") + + +@dataclass(frozen=True) +class FindingSource: + """Where a finding is, in the coordinates of the text that was scanned. + + ``event_index`` is the ledger index of the event whose bytes were scanned + (None when the scanned text is not an event's). ``claim_index`` is the + seal-relative claim whose field was scanned (None for anything that is not + a claim field): a label belongs to a claim, not to an event, and without it + two claims carrying a secret at the same offset are one location. + ``argv_index`` is the argv element (None for anything that is not argv). + ``byte_offset`` is the offset of the finding inside that text, so + ``text[byte_offset:]`` begins at it. The detectors only match ASCII, so + across a match it is a byte count; in non-ASCII surroundings it counts + decoded characters. + """ + + kind: str + event_index: Optional[int] = None + argv_index: Optional[int] = None + byte_offset: int = 0 + claim_index: Optional[int] = None + + def __post_init__(self) -> None: + if self.kind not in SOURCE_KINDS: + raise ValueError(f"unknown finding source kind: {self.kind!r}") @dataclass(frozen=True) class Finding: + """One detector hit. + + The first three fields are the original positional triple, in their + original order, so every existing construction still works unchanged. + Everything tier- and locality-related is additive with a default. + """ + kind: str span: tuple[int, int] preview: str # already-masked preview, never the raw secret + detector_id: str = "" + detector_version: int = 1 + tier: str = TIER_NOTICE + source: Optional[FindingSource] = None + line: Optional[int] = None # 1-based within the scanned text; None for argv + fingerprint: str = "" + + +@dataclass(frozen=True) +class AppliedSpan: + """One disjoint span that was actually replaced, in SOURCE coordinates. + + This is the exported projection's unit. ``kind`` is the marker text that + took its place; ``detector_id`` names the detector whose extent won the + merge. ``withheld`` marks the honest-absence path: the element was dropped + whole rather than rewritten, so the span is the element, not a token. + """ + + start: int + end: int + kind: str + detector_id: str + withheld: bool = False + + +@dataclass(frozen=True) +class RedactionResult: + text: str + findings: list[Finding] + applied: list[AppliedSpan] + + +@dataclass(frozen=True) +class ArgvRedaction: + """Scrubbed argv plus the projection that produced it. + + ``applied`` is the wire shape ``seal`` writes into the manifest, one entry + per replaced span, addressed by ``argv_index``. + """ + + argv: list[str] + applied: list[dict] + + +@dataclass(frozen=True) +class FieldRedaction: + """One scrubbed exported field, its projection, and what it found. + + ``findings`` is the part that is not a convenience: it is what makes + "redacted" and "reported to the export gate" the same act for a field, + rather than two passes that can disagree. + """ + + text: str + findings: list[Finding] + applied: list[dict] + + +# The exported projection's contract version. Bump PROJECTION_VERSION when the +# RULE that turns findings into replaced spans changes (merge policy, marker +# placement, the withheld path). Bump DETECTOR_SET_VERSION when the set of +# detectors or what they match changes. A consumer reads these instead of +# maintaining a matrix of didrun's historical behaviour. +# +# 2: the entropy sweep decides per run whether `/` is a path separator or a +# base64 alphabet member, instead of scoring every run the same way, and every +# detector now declares a tier. That is a change to what the detectors match, +# which is exactly what this number is for — leaving it at 1 would re-create +# the "read the changelog and infer it" problem the version exists to remove. +# +# It stays at 2 rather than going to 3: version 2 has never been released, so +# there is no consumer anywhere that saw the intermediate "tokenizer simply +# dropped `/`" behaviour. Bumping to 3 would publish a version number for a +# detector set that never shipped, which is the opposite of what these numbers +# are for. +# +# The span-partition ALGORITHM did not change, so PROJECTION_VERSION stays 1. +# What the projection COVERS did widen (claim fields, not just argv), and that +# is readable directly off the note: argv spans are in `redaction.applied` and +# field spans are in `redaction.fields`. A consumer checks for the key rather +# than consulting a version matrix. +PROJECTION_VERSION = 1 +DETECTOR_SET_VERSION = 2 # High-confidence structured patterns. These fire on shape, not entropy. @@ -33,7 +216,11 @@ class Finding: ("aws-access-key", re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b")), ("github-token", re.compile(r"\bgh[pousr]_[A-Za-z0-9]{36,}\b")), ("slack-token", re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{10,}\b")), - ("openai-key", re.compile(r"\bsk-[A-Za-z0-9]{20,}\b")), + # `[-_]` inside the body, or `sk-proj-…` (and every other prefixed key + # OpenAI has shipped since) matches nothing: the internal hyphen ends the + # body under `[A-Za-z0-9]`. Leading \b stays, so a `sk-` inside a word + # (`risk-...`) is still not a key. + ("openai-key", re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b")), ("pem-private-key", re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----")), # KEY=value / TOKEN=value / SECRET=value / PASSWORD=value assignments in argv. ( @@ -45,14 +232,71 @@ class Finding: ] _REDACTION_MARKER = "«redacted:{kind}»" +# Redact-or-withhold, never mangle: when a finding cannot be expressed as a +# whole-span replacement, the element is dropped whole rather than rewritten +# into something that reads authoritative and is wrong. +_WITHHELD_MARKER = "«withheld:{kind}»" + +# Detector specificity for merge tie-breaks: a structured pattern names what it +# found, the entropy sweep only names that something looked random. +_ENTROPY_KIND = "high-entropy" # Entropy heuristic bounds. A long, high-entropy, non-hex token that isn't a git # hash is suspicious. Git SHAs are hex (entropy ~4.0 bits/char over [0-9a-f]), # so requiring >4.2 bits/char over a larger alphabet excludes them. +# +# The length floor and the threshold are deliberately UNCHANGED from v0.1: one +# variable at a time, or the measurement stops being interpretable. What moved +# is the CANDIDATE — what gets scored — and it moved twice. +# +# `/` was in the token class in v0.1, and that single character was the +# detector's dominant false-positive source: with `/` included, an entire +# absolute filesystem path is one token, and a path accumulates distinct +# characters faster than any single segment does (179 of 2,642 real paths +# cleared the threshold whole; none clears it per segment). +# +# But `/` is also 1 of the 64 characters of base64, so dropping it outright +# stopped scoring credentials as well as paths: a base64 secret whose +# slash-free runs are all shorter than the floor became invisible — not +# demoted, not re-scored, INVISIBLE, and therefore also unredacted, because +# replacement is driven off the findings. Measured, that cost 15.7% of +# AWS-secret-access-key-shaped strings and 25.3% of base64(24 bytes). +# +# So the separator is not decided by the tokenizer at all. A run is taken over +# the class INCLUDING `/`, and `_looks_like_path` decides whether the `/` in it +# is a separator (score the segments) or an alphabet member (score the run). +# See docs/MEASUREMENTS.md for both directions of that decision. _ENTROPY_MIN_LEN = 24 _ENTROPY_MIN_BITS_PER_CHAR = 4.2 _HEXISH = re.compile(r"^[0-9a-fA-F]+$") -_TOKENISH = re.compile(r"[A-Za-z0-9_\-+/=]{%d,}" % _ENTROPY_MIN_LEN) +# One SEGMENT: the credential alphabet with the path separator removed. This is +# the unit a path is scored in. +_TOKENISH = re.compile(r"[A-Za-z0-9_\-+=]{%d,}" % _ENTROPY_MIN_LEN) +# One RUN: the same class plus `/`, which is the unit a base64 credential is +# scored in. A path and a base64 blob are the SAME run shape — they are told +# apart by what the segments look like, below, never by the character class. +_RUNISH = re.compile(r"[A-Za-z0-9_\-+/=]{%d,}" % _ENTROPY_MIN_LEN) + +# What tells a filesystem path from a base64 credential, given that both are +# written in [A-Za-z0-9+/]: a path is a sequence of NAMES, and base64 is not. +# A name is mostly letters and those letters are mostly lower case — `usr`, +# `homebrew`, `site-packages`, and CamelCase like `CoreServices` (10/12) or +# `Frameworks` (9/10) all clear 0.8, while base64 is ~50% lower case by +# construction and clears it only by accident. +# +# ONE name is enough to call the whole run a path. That direction is the +# conservative one: it can only move a run from whole-scoring back to the +# segment-scoring that ships today, so it cannot introduce a false positive +# that today's code does not already have — measured at exactly 0 new ones over +# 46,403 real paths. It costs recall instead (~1-3%), and that is the honest +# side to lose on for a notice-tier heuristic. +_NAME_MIN_LEN = 3 +_NAME_LETTER_FRACTION = 0.6 +_NAME_LOWERCASE_FRACTION = 0.8 + +# Per-detector revisions. Distinct from DETECTOR_SET_VERSION: this says "this +# detector's own matcher changed", the other says "the set changed". +_DETECTOR_VERSIONS = {_ENTROPY_KIND: 2} def _shannon_bits_per_char(s: str) -> float: @@ -69,11 +313,118 @@ def _mask(kind: str) -> str: return _REDACTION_MARKER.format(kind=kind) -def scan(data) -> list[Finding]: +def _withheld_mask(kind: str) -> str: + return _WITHHELD_MARKER.format(kind=kind) + + +def _tier(kind: str) -> str: + """Structured patterns block; the entropy sweep only notices.""" + return TIER_NOTICE if kind == _ENTROPY_KIND else TIER_BLOCK + + +def _is_name(segment: str) -> bool: + """Is this `/`-delimited segment a NAME (a directory or file name)? + + Mostly letters, and those letters mostly lower case. This is the whole + path-vs-base64 discriminator; see the constants above for why it is stated + over case rather than over length or slash density (both of those + distributions overlap — measured). + """ + if len(segment) < _NAME_MIN_LEN: + return False + letters = [c for c in segment if c.isalpha()] + if len(letters) < _NAME_LETTER_FRACTION * len(segment): + return False + lowercase = sum(1 for c in letters if c.islower()) + return lowercase >= _NAME_LOWERCASE_FRACTION * len(letters) + + +def _looks_like_path(run: str) -> bool: + """Is the `/` in this run a separator (True) or an alphabet member (False)?""" + return any(_is_name(segment) for segment in run.split("/")) + + +def _entropy_qualifies(token: str) -> bool: + """The unchanged v0.1 test, applied to whatever candidate is handed to it.""" + if len(token) < _ENTROPY_MIN_LEN: + return False + if _HEXISH.match(token): + return False # git hashes and hex digests are not secrets + return _shannon_bits_per_char(token) >= _ENTROPY_MIN_BITS_PER_CHAR + + +def _entropy_candidates(run: str, base: int): + """Yield ``(offset, token)`` for every unit of ``run`` that gets scored. + + Three cases, and the third is the one that carries the fix: + + * no `/` at all — the run is the token, exactly as it has always been. + * `/` as a separator — score the segments, which is what keeps a filesystem + path from being scored as one 90-character token. + * `/` as an alphabet member — score the run whole, which is what catches a + base64 credential that its own slashes chop into sub-floor pieces. If the + whole run does NOT clear the bar, fall through to the segments anyway, so + a long low-entropy prefix cannot dilute a high-entropy tail below the + threshold and hide it. + """ + if "/" not in run: + yield base, run + return + if not _looks_like_path(run) and _entropy_qualifies(run): + yield base, run + return + for m in _TOKENISH.finditer(run): + yield base + m.start(), m.group(0) + + +def locality_fingerprint(detector_id: str, source: FindingSource) -> str: + """A stable handle on WHERE a finding is. Never on WHAT it matched. + + The token is deliberately not digested: an unkeyed digest of a short + high-entropy token is a brute-force oracle for the very thing that was just + redacted, and publishing one in a manifest would undo the redaction for + anyone willing to spend a GPU-hour. The consequence is that this moves when + the content around it moves — it identifies a location, not a secret, and + it is for reporting, never for suppression (there is no suppression). + """ + material = ( + f"{detector_id}\0{source.kind}\0{source.event_index}\0{source.byte_offset}" + f"\0{source.claim_index}" + ) + return hashlib.sha256(material.encode("utf-8")).hexdigest()[:16] + + +def _locate(kind: str, span: tuple[int, int], text: str, source: Optional[FindingSource]): + """The (source, line, fingerprint) triple for one hit.""" + if source is None: + return None, None, "" + here = FindingSource( + kind=source.kind, + event_index=source.event_index, + argv_index=source.argv_index, + byte_offset=source.byte_offset + span[0], + claim_index=source.claim_index, + ) + # A single exported field (an argv element, a label, a pathspec, a changed + # path) has no lines; a blob does, and the line is what makes a finding in + # 4 MB of test output reachable. + line = ( + None + if here.kind in _LINELESS_KINDS + else text.count("\n", 0, span[0]) + 1 + ) + return here, line, locality_fingerprint(kind, here) + + +def scan(data, source: Optional[FindingSource] = None) -> list[Finding]: """Return findings for likely secrets in ``data`` (str or bytes). Structured patterns first, then a bounded entropy sweep that explicitly skips hex-only tokens (git hashes) to keep false positives low. + + ``source`` is optional and additive: pass one and every finding comes back + located (source kind, event/argv index, line, offset, fingerprint); omit it + and the findings are exactly the shape they have always been. """ if isinstance(data, bytes): text = data.decode("utf-8", "replace") @@ -82,57 +433,308 @@ def scan(data) -> list[Finding]: findings: list[Finding] = [] matched_spans: list[tuple[int, int]] = [] + def _make(kind: str, span: tuple[int, int]) -> Finding: + here, line, fingerprint = _locate(kind, span, text, source) + return Finding( + kind=kind, + span=span, + preview=_mask(kind), + detector_id=kind, + detector_version=_DETECTOR_VERSIONS.get(kind, 1), + tier=_tier(kind), + source=here, + line=line, + fingerprint=fingerprint, + ) + for kind, pat in _PATTERNS: for m in pat.finditer(text): - findings.append(Finding(kind=kind, span=m.span(), preview=_mask(kind))) + findings.append(_make(kind, m.span())) matched_spans.append(m.span()) - for m in _TOKENISH.finditer(text): - tok = m.group(0) - if _HEXISH.match(tok): - continue # git hashes and hex digests are not secrets - if any(s <= m.start() < e for s, e in matched_spans): - continue # already covered by a structured pattern - if _shannon_bits_per_char(tok) >= _ENTROPY_MIN_BITS_PER_CHAR: - findings.append( - Finding(kind="high-entropy", span=m.span(), preview=_mask("high-entropy")) - ) + for m in _RUNISH.finditer(text): + for offset, tok in _entropy_candidates(m.group(0), m.start()): + if any(s <= offset < e for s, e in matched_spans): + continue # already covered by a structured pattern + if _entropy_qualifies(tok): + findings.append(_make(_ENTROPY_KIND, (offset, offset + len(tok)))) return findings -def redact(data) -> tuple[str, list[Finding]]: - """Return (redacted_text, findings). Matches are replaced by stable markers.""" +def _specificity(kind: str) -> int: + """Lower is more specific. A structured detector names what it found.""" + return 1 if kind == _ENTROPY_KIND else 0 + + +def _preferred(a: Finding, b: Finding) -> Finding: + """Which of two overlapping findings labels the merged span. + + Outermost extent first — the wider finding is the one whose replacement + covers the other's bytes. On an exact tie, the more specific detector, so a + structured match is never relabelled as generic entropy. Ties beyond that + keep the earlier finding, which makes the choice deterministic rather than + dependent on pattern iteration order. + """ + a_width = a.span[1] - a.span[0] + b_width = b.span[1] - b.span[0] + if b_width > a_width: + return b + if b_width == a_width and _specificity(b.kind) < _specificity(a.kind): + return b + return a + + +def merge_findings(findings) -> list[AppliedSpan]: + """Collapse findings into DISJOINT MAXIMAL spans, left to right. + + ``scan`` emits overlapping findings by construction: the entropy sweep's + dedup tests start containment only, and the structured patterns are never + deduped against each other, so `MY_TOKEN=ghp_…` yields both an + `env-secret-assignment` span and a `github-token` span ending at the same + offset. Replacing those one at a time deletes the text after the outer span. + Merging first is what makes replacement a partition. + """ + ordered = sorted(findings, key=lambda f: (f.span[0], -f.span[1])) + merged: list[AppliedSpan] = [] + winners: list[Finding] = [] # the finding whose kind currently labels merged[i] + for f in ordered: + start, end = f.span + if merged and start < merged[-1].end: + prev = merged[-1] + winner = _preferred(winners[-1], f) + winners[-1] = winner + merged[-1] = AppliedSpan( + start=prev.start, + end=max(prev.end, end), + kind=winner.kind, + detector_id=winner.detector_id or winner.kind, + ) + else: + merged.append( + AppliedSpan( + start=start, + end=end, + kind=f.kind, + detector_id=f.detector_id or f.kind, + ) + ) + winners.append(f) + return merged + + +def redact(data, source: Optional[FindingSource] = None) -> RedactionResult: + """Replace every finding with a marker, as a partition of the input. + + The output is built by walking the merged spans left to right and + concatenating alternating kept slices and markers — never by index + arithmetic on a string that is changing length underneath the offsets. + + ``source`` is optional and additive, and it matters for one reason beyond + reporting: a caller that both redacts a field and needs the findings for + the export gate gets a LOCATED set here, so the same pass that removes the + secret is the one that reports it. Redacting and scanning in two places is + what let a claim label block a seal and be published anyway. + """ if isinstance(data, bytes): text = data.decode("utf-8", "replace") else: text = data - findings = scan(text) - # Replace from the end so earlier spans keep their offsets. - ordered = sorted(findings, key=lambda f: f.span[0], reverse=True) - out = text - for f in ordered: - s, e = f.span - out = out[:s] + _mask(f.kind) + out[e:] - return out, findings + findings = scan(text, source) + applied = merge_findings(findings) + parts: list[str] = [] + cursor = 0 + for span in applied: + parts.append(text[cursor:span.start]) + parts.append(_mask(span.kind)) + cursor = span.end + parts.append(text[cursor:]) + return RedactionResult(text="".join(parts), findings=findings, applied=applied) + + +def reconstruct(source: str, result: RedactionResult) -> str: + """Rebuild ``source`` from the redacted text plus the applied partition. + + Kept slices come from the OUTPUT and replaced slices from the source, so a + deletion, a stale offset or a misplaced marker all fail to reproduce the + input. Markers are located arithmetically, never searched for: a recorded + argv is allowed to contain the literal marker text. + """ + rebuilt: list[str] = [] + out_cursor = 0 + src_cursor = 0 + for span in result.applied: + kept = span.start - src_cursor + if kept < 0: + raise ValueError("applied spans are not disjoint and ascending") + rebuilt.append(result.text[out_cursor:out_cursor + kept]) + out_cursor += kept + marker = _mask(span.kind) + if result.text[out_cursor:out_cursor + len(marker)] != marker: + raise ValueError("no marker at the offset the projection names") + out_cursor += len(marker) + rebuilt.append(source[span.start:span.end]) + src_cursor = span.end + rebuilt.append(result.text[out_cursor:]) + return "".join(rebuilt) + + +def _projection_entry(argv_index: int, span: AppliedSpan) -> dict: + return { + "argv_index": argv_index, + "start": span.start, + "end": span.end, + "detector_id": span.detector_id, + "withheld": span.withheld, + } + + +def redact_argv(argv) -> ArgvRedaction: + """Redact each argv element and report the projection that was applied. + + Post-condition per element: the redaction must reconstruct the element + exactly. It cannot fail once ``merge_findings`` has run — but if it ever + does, the element is WITHHELD whole rather than exported in a mangled form. + An honest absence beats a corrupted string that reads authoritative. + """ + scrubbed: list[str] = [] + applied: list[dict] = [] + for index, arg in enumerate(argv): + result = redact(arg) + source = arg.decode("utf-8", "replace") if isinstance(arg, bytes) else arg + try: + faithful = reconstruct(source, result) == source + except ValueError: + faithful = False + if faithful: + scrubbed.append(result.text) + applied.extend(_projection_entry(index, s) for s in result.applied) + continue + kind = result.applied[0].kind if result.applied else "unknown" + scrubbed.append(_withheld_mask(kind)) + applied.append( + _projection_entry( + index, + AppliedSpan( + start=0, + end=len(source), + kind=kind, + detector_id=kind, + withheld=True, + ), + ) + ) + return ArgvRedaction(argv=scrubbed, applied=applied) def scrub_argv(argv) -> list[str]: """Redact secret-shaped tokens inside argv elements (e.g. FOO=secret).""" - scrubbed = [] - for arg in argv: - red, _ = redact(arg) - scrubbed.append(red) - return scrubbed + return redact_argv(argv).argv + + +def _field_entry(field: str, span: AppliedSpan) -> dict: + return { + "field": field, + "start": span.start, + "end": span.end, + "detector_id": span.detector_id, + "withheld": span.withheld, + } + + +def redact_field(field: str, value: str, source: Optional[FindingSource] = None): + """Redact ONE named exported field, and return what was found doing it. + + This is ``redact_argv``'s discipline (redact-or-withhold, never mangle; + declare the projection) for the exported claim fields that are not argv — + the label, the pathspecs, the changed paths. It returns the findings as + well as the text on purpose: those findings ARE the export-gate input for + this field, so the field cannot be blocked-but-published or + published-but-unscanned. Both of those were real. + """ + result = redact(value, source) + try: + faithful = reconstruct(value, result) == value + except ValueError: + faithful = False + if faithful: + return FieldRedaction( + text=result.text, + findings=result.findings, + applied=[_field_entry(field, s) for s in result.applied], + ) + kind = result.applied[0].kind if result.applied else "unknown" + return FieldRedaction( + text=_withheld_mask(kind), + findings=result.findings, + applied=[ + _field_entry( + field, + AppliedSpan( + start=0, + end=len(value), + kind=kind, + detector_id=kind, + withheld=True, + ), + ) + ], + ) + + +# How many findings a refusal enumerates before it collapses into a count. A +# refusal that prints 19,900 lines is as unusable as one that prints none. +ENUMERATION_CAP = 20 + + +def describe_finding(f: Finding) -> str: + """One located line for a finding. Carries the masked preview, never the token.""" + src = f.source + if src is None: + where = "unlocated" + offset = "-" + else: + where = src.kind + if src.claim_index is not None: + where += f" of claim {src.claim_index}" + if src.event_index is not None: + where += f" of event {src.event_index}" + if src.argv_index is not None: + where += f" arg {src.argv_index}" + offset = str(src.byte_offset) + return ( + f"{f.detector_id or f.kind} [{f.tier}] at {where} " + f"line {'-' if f.line is None else f.line} offset {offset}: " + f"{f.preview} fingerprint {f.fingerprint or '-'}" + ) + + +def enumerate_findings(findings, cap: int = ENUMERATION_CAP) -> list[str]: + """Located lines for ``findings``, capped, with the remainder as a count.""" + lines = [f" - {describe_finding(f)}" for f in findings[:cap]] + remainder = len(findings) - cap + if remainder > 0: + lines.append(f" ... and {remainder} more finding(s) not listed") + return lines class SecretsBlocked(Exception): - """Raised when export is blocked because secrets were found.""" + """Raised when export is blocked because secrets were found. + + The message ENUMERATES: a finding an operator cannot locate is a finding + they can only respond to with --allow-secrets, which is the failure mode + that gets a blocking gate switched off. + """ - def __init__(self, findings: list[Finding]) -> None: + def __init__(self, findings: list[Finding], cap: int = ENUMERATION_CAP) -> None: self.findings = findings kinds = ", ".join(sorted({f.kind for f in findings})) - super().__init__( - f"export blocked: {len(findings)} likely secret(s) found ({kinds}). " + lines = [ + f"export blocked: {len(findings)} likely secret(s) found in the " + f"exported artifact ({kinds}).", + ] + lines.extend(enumerate_findings(findings, cap)) + lines.append( "Re-run with --allow-secrets to export a redacted artifact anyway " "(the override is logged into the manifest)." ) + super().__init__("\n".join(lines)) diff --git a/src/didrun/render.py b/src/didrun/render.py index 3e90a4a..1a45e19 100644 --- a/src/didrun/render.py +++ b/src/didrun/render.py @@ -20,6 +20,8 @@ import os from typing import Optional +from .claims import ENV_DRIFT, ENV_INCOMPARABLE, ENV_MATCH, ENV_NOT_RECORDED + # Grade -> (text token, ANSI color, severity marker for NO_COLOR, honest gloss). # The token NEVER overclaims: the strongest positive says TREE-EXACT ("the # evidence tree equals the sealed tree"), never "VERIFIED"/"PROVEN" — didrun @@ -104,6 +106,33 @@ def _skipped_notes(n: int) -> str: return f"{n} note{'' if n == 1 else 's'} skipped (unparseable)" +def env_summary_text(report) -> str: + """One line for how the sealed environment fingerprints compared. + + The three counters are always printed, zeros included: "0 drifted" is a + measurement and a missing counter is not. `not-recorded` is appended only + when it is nonzero — it is the shape of every note published before the + fingerprint was bound, and printing "0 not-recorded" on every modern verify + would be noise about a case that no longer occurs. + + Whether drift REFUSES is said here rather than left implied. The same three + numbers mean different things under --require-env-match, and a reader who + cannot tell which mode produced them cannot act on either. + """ + counts = report.env_counts + line = ( + f"env: {counts[ENV_MATCH]} match / {counts[ENV_DRIFT]} drifted / " + f"{counts[ENV_INCOMPARABLE]} incomparable" + ) + if counts[ENV_NOT_RECORDED]: + line += f" / {counts[ENV_NOT_RECORDED]} not-recorded" + if report.require_env_match: + line += " (drift refuses: --require-env-match)" + elif counts[ENV_DRIFT]: + line += " (advisory; --require-env-match makes it refuse)" + return line + + def chain_banner_text(report) -> str: """The chain sentence a reviewer must not miss, or "" when there is none. @@ -169,6 +198,7 @@ def render_verdict(report, width: int = 80) -> str: # How many of those verdicts were checked against the recorded entry the # seal named, rather than regraded against whatever ledger is on disk now. lines.append(f" {report.evidence_bound_count}/{total} claims evidence-bound") + lines.append(f" {env_summary_text(report)}") if report.secrets_override: lines.append(_c(" ! sealed with --allow-secrets (redacted export)", "33")) if report.notes_skipped: @@ -185,6 +215,14 @@ def render_verdict(report, width: int = 80) -> str: if r.exit_code is not None: detail = f"exit {r.exit_code} {detail}" lines.append(f" {marker} {_c(f'{token:<12}', color)} {label:<26} {detail}") + # The drifted rows say so individually. `incomparable` does not get a + # row of its own: it is the state of every note published before the + # fingerprint was bound, so a per-claim line would be a paragraph of + # boilerplate under an archive — the header counter already reports it. + if r.env_status == ENV_DRIFT: + lines.append( + f" {'':<12} {'':<26} {ENV_DRIFT}: {_sanitize(r.env_reason)}" + ) # Drill-down: show the delta for stale/scope, capped. for change in r.delta[:5]: lines.append(f" {'':<12} {'':<26} {change.status} {_sanitize(change.path)}") @@ -230,12 +268,19 @@ def esc(s: str) -> str: more = f"
  • … {len(r.delta) - 20} more
    • " if len(r.delta) > 20 else "" delta_html = f"
      {items}{more}
    " claim_label = r.claim.label if not isinstance(r.claim, dict) else r.claim.get("label", "") + # Same rule as the CLI table: drifted rows say so, incomparable ones are + # left to the header counter. + env_html = ( + f"{esc(ENV_DRIFT)}: {esc(r.env_reason)}" + if r.env_status == ENV_DRIFT + else "" + ) rows.append( f"" f"{esc(token)}" f"{esc(claim_label)}" f"{detail}" - f"{esc(exit_txt)}{delta_html}" + f"{esc(exit_txt)}{env_html}{delta_html}" f"" ) @@ -339,6 +384,9 @@ def esc(s: str) -> str: .detail {{ color:var(--dim); }} .detail .reason {{ display:block; }} .exit {{ display:inline-block; margin-top:2px; color:var(--mono); font-size:12px; }} + /* Advisory by default, so it is toned like the warn states rather than the + failure ones — drift does not make a recorded command un-run. */ + .envdrift {{ display:block; margin-top:4px; color:var(--warn); font-size:12px; }} .delta {{ margin:6px 0 0; padding-left:16px; color:var(--dim); font-size:12px; }} .delta li {{ overflow-wrap:anywhere; }} .delta .ch {{ display:inline-block; width:14px; color:var(--mono); font-weight:700; }} @@ -365,6 +413,7 @@ def esc(s: str) -> str: {chain_html}
    {esc(verified)}/{esc(total)} claims recorded-exact · {esc(report.evidence_bound_count)}/{esc(total)} claims evidence-bound · + {esc(env_summary_text(report))} · commit {esc(report.commit[:12])} · tree {esc(report.tree[:12])} · resolved-by {esc(report.resolved_by)} · coverage {coverage_html}
    diff --git a/tests/compat/test_corpus_replay.py b/tests/compat/test_corpus_replay.py index 6421f23..f590624 100644 --- a/tests/compat/test_corpus_replay.py +++ b/tests/compat/test_corpus_replay.py @@ -298,10 +298,15 @@ def test_leg1_chain_recompute(replay_source): # coded per manifest version so a new manifest field is a deliberate one-line # diff here and never a silent pass. # v2 introduced no top-level manifest key — the evidence binding lives inside -# each claim entry — so its allowlist is v1's. +# each claim entry. +# `secrets` is P3.2's: the scan-domain record beside `secrets_override`. It is +# additive with a v1-reproducing default ({}), so a stored note of EITHER +# version legitimately lacks it, and both allowlists carry it. This entry is +# the deliberate one-line diff the criterion exists to force — without it the +# round-trip leg would go red across every stored note at once. MANIFEST_ADDITIVE_KEYS = { - 1: frozenset({"secrets_override"}), - 2: frozenset({"secrets_override"}), + 1: frozenset({"secrets_override", "secrets"}), + 2: frozenset({"secrets_override", "secrets"}), } diff --git a/tests/test_capture_claims_manifest.py b/tests/test_capture_claims_manifest.py index 9e13add..6e92fc5 100644 --- a/tests/test_capture_claims_manifest.py +++ b/tests/test_capture_claims_manifest.py @@ -316,7 +316,10 @@ def test_cli_refuses_a_future_note_as_a_message_not_a_traceback(repo: Path, caps def test_env_fingerprint_carries_its_version_in_band(): - assert re.fullmatch(r"v1:[0-9a-f]{16}", capture.env_fingerprint({"PATH": "/bin"})) + # The shape is pinned in tests/test_env_fingerprint.py, which owns the v2 + # contract; what this asserts is that the version is READABLE off the wire + # and that an unversioned v0.1 digest still reports as unversioned. + assert capture.fingerprint_version(capture.env_fingerprint({"PATH": "/bin"})) == 2 assert capture.fingerprint_version("v1:0123456789abcdef") == 1 # A bare v0.1 digest: incomparable with a versioned one, not drifted. assert capture.fingerprint_version("0123456789abcdef") is None @@ -324,14 +327,14 @@ def test_env_fingerprint_carries_its_version_in_band(): def test_env_fingerprint_is_deterministic_and_env_sensitive(): """Determinism is a trust-path invariant; sensitivity is the point of the - field. No hard-coded digest — the key set changes in a later unit.""" + field. No hard-coded digest — the key set has changed once already.""" env = {"PATH": "/usr/bin", "SHELL": "/bin/zsh", "LANG": "C"} assert capture.env_fingerprint(env) == capture.env_fingerprint(dict(env)) - assert capture.env_fingerprint({**env, "PATH": "/usr/local/bin"}) != capture.env_fingerprint(env) + assert capture.env_fingerprint({**env, "SHELL": "/bin/bash"}) != capture.env_fingerprint(env) def test_recorded_events_carry_the_versioned_fingerprint(repo: Path): """The prefix reaches the ledger, not just the helper's return value.""" s = _session(repo) ev = run_wrapped([sys.executable, "-c", "print(1)"], s, repo) - assert capture.fingerprint_version(ev.env_fingerprint) == 1 + assert capture.fingerprint_version(ev.env_fingerprint) == 2 diff --git a/tests/test_credential_recall.py b/tests/test_credential_recall.py new file mode 100644 index 0000000..4f33b0e --- /dev/null +++ b/tests/test_credential_recall.py @@ -0,0 +1,399 @@ +"""P3 remediation — the two directions the tokenizer change was measured in. + +The P3.2 work removed `/` from the entropy sweep's token class because an +absolute filesystem path was tokenizing as one 90-character token and scoring +like a credential. That fixed the false positives and was measured. What was +not measured is the other direction: `/` is also 1 of the 64 characters of +base64, so dropping it stopped scoring a real credential class as well as +paths. A base64 secret whose slash-free runs are all shorter than the +24-character floor became invisible — and because replacement is driven off +`scan`'s findings, invisible means UNREDACTED, published verbatim into a note +under a seal line reading "0 findings". + +So this file pins BOTH directions, and each fixture asserts the structural +property that makes it a faithful witness rather than a convenient one: + +* the credential fixtures assert that every one of their `/`-delimited + segments is under the floor, which is exactly the class that went missing — + a fixture with a 24-character segment would pass on segment scoring alone and + prove nothing; +* the path fixtures assert they fired under the pre-P3.2 sweep, so "still + silent" is a statement about a real false positive and not about a string + that was always quiet. + +`_looks_like_path` is what tells the two apart, and it is deliberately biased: +one name-like segment is enough to call a run a path. That direction can only +move a run back to the segment scoring that P3.2 ships, so it cannot introduce +a false positive P3.2 does not already have — measured at 0 new ones over +46,403 real paths — and it pays for that in recall instead. + +Every credential-shaped string here is synthetic and structural. Every path is +synthetic too, for the reason the P3.2 file gives: the strings this was +measured against are a home directory and a private project name, and a test +fixture is a shipped file. +""" + +from __future__ import annotations + +import json +import math +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import manifest as M +from didrun import redact +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + + +# --- the P3.2 sweep, reimplemented so before/after is checkable here ---------- + +# The token class exactly as P3.2 shipped it: `/` removed, so a run is only +# ever scored in segments. Written out rather than imported, so a change to +# redact.py cannot make these before/after assertions agree with it. +_P32_TOKENISH = re.compile(r"[A-Za-z0-9_\-+=]{24,}") +_PRE_P32_TOKENISH = re.compile(r"[A-Za-z0-9_\-+/=]{24,}") +_HEXISH = re.compile(r"^[0-9a-fA-F]+$") + + +def _bits_per_char(s: str) -> float: + counts: dict = {} + for ch in s: + counts[ch] = counts.get(ch, 0) + 1 + n = len(s) + return -sum((c / n) * math.log2(c / n) for c in counts.values()) + + +def _sweep(text: str, tokenizer) -> list: + """The entropy sweep under an arbitrary tokenizer, structured hits removed.""" + covered = [] + for _kind, pat in redact._PATTERNS: + covered.extend(m.span() for m in pat.finditer(text)) + hits = [] + for m in tokenizer.finditer(text): + tok = m.group(0) + if _HEXISH.match(tok): + continue + if any(s <= m.start() < e for s, e in covered): + continue + if _bits_per_char(tok) >= redact._ENTROPY_MIN_BITS_PER_CHAR: + hits.append(m.span()) + return hits + + +def _entropy_hits(text: str) -> list: + return [f for f in redact.scan(text) if f.kind == redact._ENTROPY_KIND] + + +# --- direction 1: base64 credentials with a separator in them ---------------- + +# Synthetic base64-alphabet credentials. Each one carries `/` at an offset that +# chops it into pieces that are ALL under the 24-character floor, which is the +# precise shape that stopped being a candidate at all. +SEPARATED_CREDENTIALS = [ + "Kq7Xm2Bv9Rt4Wn6Zc/8Yd3Fj5Hg1Lp0Qs", + "Zr4Nb8Kw2Vt6Mx1Qy/Jd7Fs3Gh5Lc9Pn0Rv2Bt6Wq", + "Tj5Hn9Cw3Xq7Bz1Mv/Ld4Fp8Gs2Kr6Yt0Nx/Vb3Zc7Qw", +] + + +@pytest.mark.parametrize("cred", SEPARATED_CREDENTIALS) +def test_credential_fixtures_are_faithful_witnesses(cred): + """Every segment under the floor — otherwise the fixture proves nothing. + + A fixture with one 24-character slash-free segment is caught by segment + scoring alone, so it would pass against the very code that lost this + class. The whole run must clear the bar and no segment may reach it. + """ + segments = cred.split("/") + assert len(segments) > 1, "fixture must contain a separator" + assert max(len(s) for s in segments) < redact._ENTROPY_MIN_LEN, ( + "fixture is caught by segment scoring and cannot witness the defect" + ) + assert len(cred) >= redact._ENTROPY_MIN_LEN + assert _bits_per_char(cred) >= redact._ENTROPY_MIN_BITS_PER_CHAR + + +@pytest.mark.parametrize("cred", SEPARATED_CREDENTIALS) +def test_separated_credentials_were_lost_by_p32_and_are_found_again(cred): + """The regression, stated as a before/after on each fixture.""" + assert _sweep(cred, _PRE_P32_TOKENISH), "fixture did not fire before P3.2" + assert not _sweep(cred, _P32_TOKENISH), ( + "fixture is not in the class P3.2 lost, so it cannot witness the fix" + ) + assert _entropy_hits(cred), "the credential class is still invisible" + + +@pytest.mark.parametrize("cred", SEPARATED_CREDENTIALS) +def test_a_found_credential_is_actually_replaced(cred): + """Detection is not the point; removal is. + + `redact` drives replacement off `scan`'s findings, so a class that stops + being a candidate stops being scrubbed too. This asserts the consequence + rather than the mechanism. + """ + result = redact.redact(cred) + assert cred not in result.text + assert result.text == redact._mask(redact._ENTROPY_KIND) + # And the projection still reconstructs, so nothing was mangled. + assert redact.reconstruct(cred, result) == cred + + +def test_a_separated_credential_inside_a_command_line_is_replaced(): + """The shape that actually leaks: a secret sitting in a recorded argv.""" + cred = SEPARATED_CREDENTIALS[1] + text = f"deploy --token {cred} --region eu-west-1" + scrubbed = redact.redact(text).text + assert cred not in scrubbed + assert "deploy --token" in scrubbed and "--region eu-west-1" in scrubbed + + +# --- direction 2: the paths P3.2 silenced stay silent ------------------------ + +# The same class the P3.2 measurement was taken over: ordinary build, cache and +# toolchain paths, every one of them a false positive under the pre-P3.2 sweep. +PATHS_THAT_FIRED_BEFORE_P32 = [ + "/opt/homebrew/Cellar/go/1.24.0/libexec/pkg/tool/darwin_arm64", + "/var/folders/qs/tmpbuild/example-service/cmd/gateway/internal/transport/grpc_handler.go", + "/Users/exampleuser/Library/Caches/build-tool/artifacts/darwin_arm64/release/binary", + "/home/runner/_work/pipeline-Qz4/artifacts/Linux_x64/Release/bin/toolchain", + "/opt/build/Jenkins/workspace/Nightly_x64/out/Release/obj/gen/protoc_wrapper", +] + +# Paths whose segments are CamelCase or capitalised rather than lower case — +# the shape that a naive "a path segment is lower case" rule misreads as a +# credential. They must stay quiet too. +MIXED_CASE_PATHS = [ + "/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks", + "/opt/toolchain/PrivateFrameworks/CoreFoundation/Modules/Darwin/Headers", +] + + +@pytest.mark.parametrize("path", PATHS_THAT_FIRED_BEFORE_P32 + MIXED_CASE_PATHS) +def test_paths_silenced_by_p32_are_still_silent(path): + assert _entropy_hits(path) == [], "a path is being scored as a credential again" + assert redact.redact(path).text == path, "a path was rewritten in an export" + + +@pytest.mark.parametrize("path", PATHS_THAT_FIRED_BEFORE_P32) +def test_path_fixtures_are_faithful_witnesses(path): + """These fired before P3.2, so "still silent" is about a real false positive.""" + assert _sweep(path, _PRE_P32_TOKENISH), ( + "fixture did not fire under the pre-P3.2 sweep and proves nothing" + ) + + +def test_the_separator_is_classified_not_tokenized_away(): + """The actual mechanism, pinned directly. + + P3.2 answered "is this `/` a separator?" in the character class, which + answers it the same way for every string. It is answered per run now: a + path is scored in segments, a base64 blob whole. The segment tokenizer + still admits no separator, which is what keeps a path from re-merging. + """ + assert "/" not in redact._TOKENISH.pattern + assert "/" in redact._RUNISH.pattern + assert redact._looks_like_path(PATHS_THAT_FIRED_BEFORE_P32[0]) + assert redact._looks_like_path(MIXED_CASE_PATHS[0]) + for cred in SEPARATED_CREDENTIALS: + assert not redact._looks_like_path(cred) + + +def test_the_threshold_and_the_length_floor_still_did_not_move(): + """The candidate moved; the test applied to it did not.""" + assert redact._ENTROPY_MIN_BITS_PER_CHAR == 4.2 + assert redact._ENTROPY_MIN_LEN == 24 + + +def test_a_low_entropy_prefix_cannot_hide_a_high_entropy_tail(): + """Whole-run scoring must not become a way to dilute a secret below the bar. + + A long repetitive prefix drags the whole run's per-character entropy under + the threshold. Scoring only the run would miss the tail that segment + scoring catches, so a run that does not clear the bar whole is still + scored in segments. + """ + run = "a" * 40 + "/" + "Kq7Xm2Bv9Rt4Wn6Zc8Yd3Fj5Hg1Lp0Qs" + assert _bits_per_char(run) < redact._ENTROPY_MIN_BITS_PER_CHAR, ( + "fixture does not dilute the run and cannot witness the fallback" + ) + assert _entropy_hits(run), "the high-entropy tail was diluted away" + + +def test_a_git_hash_is_still_not_a_secret(): + """The hex exclusion survives whole-run scoring.""" + assert _entropy_hits("a" * 0 + "0123456789abcdef0123456789abcdef01234567") == [] + + +# --- the exported claim fields are redacted, not just blocked on ------------- + +_GHP = "ghp_" + "G" * 36 + + +def _one_event(session: Session, repo: Path, code: str): + return run_wrapped([sys.executable, "-c", code], session, repo) + + +def _note_text(repo: Path) -> str: + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", "HEAD"], + cwd=str(repo), capture_output=True, text=True, + ) + assert proc.returncode == 0, "no note was published" + return proc.stdout + + +def test_a_secret_in_a_claim_label_is_redacted_in_the_published_note(repo: Path): + """The gate and the publication used to disagree. + + A token in a claim label blocked the seal — and the refusal says "re-run + with --allow-secrets to export a REDACTED artifact anyway". On the + override the label went into refs/notes/didrun verbatim: a ref that gets + pushed. Blocking is not redacting, and the message promised both. + """ + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label=f"suite green with {_GHP}", + event_indices=(0,), + declared_at_index=0, + ), + ) + # Still blocks: this fix does not open the gate, it makes the override honest. + with pytest.raises(redact.SecretsBlocked): + M.seal(s, repo) + + m = M.seal(s, repo, allow_secrets=True) + assert m.secrets_override is True + + published = _note_text(repo) + assert _GHP not in published, "the token was published verbatim" + assert json.loads(published)["claims"][0]["claim"]["label"] == ( + f"suite green with {redact._mask('github-token')}" + ) + + +def test_the_label_redaction_declares_its_projection(repo: Path): + """A marker is plain text; the projection is what a consumer can check.""" + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + label = f"suite green with {_GHP}" + M.declare_claim( + s, + Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo, allow_secrets=True) + fields = m.claims[0]["redaction"]["fields"] + entry = next(f for f in fields if f["field"] == "claim.label") + assert entry["detector_id"] == "github-token" + assert entry["withheld"] is False + # The declared span names the source coordinates that were replaced. + assert label[entry["start"]:entry["end"]] == _GHP + + +def test_a_secret_in_a_pathspec_is_redacted_in_the_published_note(repo: Path): + """Pathspecs are operator-authored too, and they are exported.""" + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label="scoped", + event_indices=(0,), + pathspecs=(f"src/{_GHP}",), + declared_at_index=0, + ), + ) + m = M.seal(s, repo, allow_secrets=True) + published = _note_text(repo) + assert _GHP not in published + assert redact._mask("github-token") in m.claims[0]["claim"]["pathspecs"][0] + + +def test_a_secret_in_a_changed_path_is_redacted_in_the_published_note(repo: Path): + """A changed path is repository-authored text and rides in every stale grade.""" + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="t", event_indices=(0,), declared_at_index=0), + ) + # Move the tree after the evidence, with a filename that carries a secret. + (repo / f"{_GHP}.txt").write_text("x\n") + subprocess.run(["git", "add", "-A"], cwd=str(repo), capture_output=True, check=True) + subprocess.run( + ["git", "commit", "-qm", "add file"], cwd=str(repo), capture_output=True, check=True + ) + + m = M.seal(s, repo, allow_secrets=True) + result = m.claims[0] + assert result["delta"], "fixture did not produce a delta and proves nothing" + published = _note_text(repo) + assert _GHP not in published, "a changed path was published verbatim" + assert any( + redact._mask("github-token") in change["path"] for change in result["delta"] + ) + + +def test_every_exported_claim_string_field_is_scrubbed(repo: Path): + """The invariant, not the four instances. + + Anything in the published claim entry that carries operator- or + repository-authored text goes through the redaction pass. This walks the + serialized note rather than naming fields, so a field added later that + forgets the pass fails here. + """ + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label=f"label {_GHP}", + event_indices=(0,), + pathspecs=(f"spec/{_GHP}",), + declared_at_index=0, + ), + ) + (repo / f"{_GHP}.txt").write_text("x\n") + subprocess.run(["git", "add", "-A"], cwd=str(repo), capture_output=True, check=True) + subprocess.run( + ["git", "commit", "-qm", "add file"], cwd=str(repo), capture_output=True, check=True + ) + M.seal(s, repo, allow_secrets=True) + assert _GHP not in _note_text(repo) + + +def test_the_scan_still_blocks_on_a_field_it_redacts(repo: Path): + """Redacting a field must not quietly stop it blocking. + + Scrubbing the label before the manifest bytes are serialized removes it + from the whole-blob scan. If the field pass did not report what it removed, + this seal would succeed silently — the opposite failure, and just as bad. + """ + s = Session(repo / ".didrun") + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label=f"label {_GHP}", + event_indices=(0,), + declared_at_index=0, + ), + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo) + message = str(exc.value) + assert "github-token" in message + # And it says WHERE, in claim coordinates: a label has no event and no line. + assert "label of claim 0" in message diff --git a/tests/test_detector_tiers.py b/tests/test_detector_tiers.py new file mode 100644 index 0000000..e845358 --- /dev/null +++ b/tests/test_detector_tiers.py @@ -0,0 +1,635 @@ +"""P3.2 — detector tiers, finding locality, and the export-domain block. + +Three things are pinned here, and they pull against each other on purpose. + +1. The tokenizer stops treating a whole filesystem path as one token. The + entropy sweep's token class used to include `/`, and a path accumulates + distinct characters far faster than any one of its segments does, so an + ordinary build path scored like a credential. `_old_entropy_findings` below + is a deliberate reimplementation of the pre-P3.2 sweep, so every "this no + longer fires" assertion is a measured BEFORE/AFTER inside one file rather + than a claim about code that no longer exists. + +2. Nothing in (1) may cost structured recall. The six shape-based detectors are + unchanged and every one of them is pinned firing at `block`, and the entropy + sweep is pinned still firing on a genuinely high-entropy token — it was + demoted to `notice`, never deleted. Deleting it would pass every "paths stop + firing" test in this file while removing the only detector that catches an + unstructured credential. + +3. Blocking follows the bytes that leave the machine. A secret in a claim label + is published and now blocks; a secret in a stdout blob stays in a gitignored + local ledger and now warns instead. + +Every credential-shaped string below is synthetic and structural — a shape the +detectors match, never a real token. Every path below is synthetic too: the +strings this work was measured against are a home directory and a private +project name, and a test fixture is a shipped file. The substitution is not +taken on faith — each synthetic path's entropy is asserted into the band its +group is supposed to occupy, so swapping the fixture for a differently-shaped +string fails here instead of quietly changing what is being tested. +""" + +from __future__ import annotations + +import json +import math +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import manifest as M +from didrun import redact +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +# --- the pre-P3.2 detector, reimplemented so before/after is checkable -------- + +# The token class as it stood before this prompt: `/` included, which is the +# entire defect. Written out rather than imported, so restoring the old class +# in redact.py cannot make these tests agree with it. +_OLD_TOKENISH = re.compile(r"[A-Za-z0-9_\-+/=]{24,}") +_HEXISH = re.compile(r"^[0-9a-fA-F]+$") + + +def _bits_per_char(s: str) -> float: + counts: dict = {} + for ch in s: + counts[ch] = counts.get(ch, 0) + 1 + n = len(s) + return -sum((c / n) * math.log2(c / n) for c in counts.values()) + + +def _old_entropy_findings(text: str) -> list: + """The entropy sweep exactly as it behaved before the tokenizer change.""" + covered = [] + for _kind, pat in redact._PATTERNS: + covered.extend(m.span() for m in pat.finditer(text)) + hits = [] + for m in _OLD_TOKENISH.finditer(text): + tok = m.group(0) + if _HEXISH.match(tok): + continue + if any(s <= m.start() < e for s, e in covered): + continue + if _bits_per_char(tok) >= redact._ENTROPY_MIN_BITS_PER_CHAR: + hits.append(m.span()) + return hits + + +def _max_token_bits(text: str, tokenizer=_OLD_TOKENISH) -> float: + toks = [m.group(0) for m in tokenizer.finditer(text) if not _HEXISH.match(m.group(0))] + return max((_bits_per_char(t) for t in toks), default=0.0) + + +# --- test 1: paths no longer tokenize whole ---------------------------------- + +# The band ordinary repository paths were measured to occupy. Paths in it were +# never the problem — they are the control group, and asserting the synthetics +# land in it is what makes the substitution faithful rather than convenient. +_ORDINARY_BAND = (3.94, 4.12) + +ORDINARY_PATHS = [ + "/home/runner/work/project-name/project-name/src/backend/services/authentication/handlers.py", + "/Users/exampleuser/src/example-project/pkg/adapters/outbound/persistence/migrations", +] + +# The knife edge: paths that scored between the ordinary band and the threshold. +# They did not fire, but a hair more variety in a directory name and they would +# have — which is the mechanism behind the observed 1-to-346 finding swing on +# runs that differed only in where the repo was checked out. +KNIFE_EDGE_PATHS = [ + "/Users/exampleuser/Documents/billing-reconciler/internal/messagebus/http/connection_lifecycle_test.go", +] + +# And the paths that actually fired: ordinary build and cache paths, every one +# of them a false positive, every one of them silent after the change. The +# homebrew path is used verbatim — it names no user and no private project. +DEEP_PATHS_THAT_FIRED = [ + "/opt/homebrew/Cellar/go/1.24.0/libexec/pkg/tool/darwin_arm64", + "/var/folders/qs/tmpbuild/example-service/cmd/gateway/internal/transport/grpc_handler.go", + "/Users/exampleuser/Library/Caches/build-tool/artifacts/darwin_arm64/release/binary", + "/Users/exampleuser/Documents/widget-platform-Qz4/internal/servicelayer/http/connection_lifecycle_test.go", +] + + +@pytest.mark.parametrize("path", ORDINARY_PATHS) +def test_ordinary_paths_are_faithful_substitutes_and_stay_quiet(path): + """The control group: in the measured band, silent before and after.""" + low, high = _ORDINARY_BAND + bits = _max_token_bits(path) + assert low <= bits <= high, ( + f"synthetic path is not a faithful substitute: {bits:.4f} bits/char is " + f"outside the measured ordinary-path band {_ORDINARY_BAND}" + ) + assert _old_entropy_findings(path) == [] + assert redact.scan(path) == [] + + +@pytest.mark.parametrize("path", KNIFE_EDGE_PATHS) +def test_knife_edge_paths_sit_between_the_band_and_the_threshold(path): + """Faithful to the near-miss class: above ordinary, still under the line.""" + bits = _max_token_bits(path) + assert _ORDINARY_BAND[1] < bits < redact._ENTROPY_MIN_BITS_PER_CHAR, ( + f"synthetic path is not a faithful knife-edge substitute: {bits:.4f}" + ) + assert _old_entropy_findings(path) == [] + assert redact.scan(path) == [] + + +@pytest.mark.parametrize("path", DEEP_PATHS_THAT_FIRED) +def test_deep_paths_fired_before_and_are_silent_after(path): + """The before/after difference, stated explicitly on each fixture.""" + bits = _max_token_bits(path) + assert bits >= redact._ENTROPY_MIN_BITS_PER_CHAR, ( + f"synthetic path is not a faithful substitute for a FIRING path: {bits:.4f}" + ) + before = _old_entropy_findings(path) + assert before, "fixture must fire under the old tokenizer or it proves nothing" + assert redact.scan(path) == [], "path still tokenizes whole" + + +def test_tokenizer_no_longer_admits_a_path_separator(): + """The one-character change, pinned directly. + + A path is now several tokens, not one, and the count is what changes the + entropy: the same characters spread over segments never reach the + threshold that the concatenation did. + """ + assert "/" not in redact._TOKENISH.pattern + path = DEEP_PATHS_THAT_FIRED[1] + old = _OLD_TOKENISH.findall(path) + new = redact._TOKENISH.findall(path) + assert any("/" in tok for tok in old), "fixture must span separators" + assert all("/" not in tok for tok in new) + # A path's segments are individually short, so the whole path no longer + # reaches the 24-character floor at all: it is not merely re-scored, it + # stops being a candidate. + assert max(len(t) for t in old) > max((len(t) for t in new), default=0) + + +def test_the_threshold_and_the_length_floor_did_not_move(): + """One variable at a time. If these move, the measurement above is void.""" + assert redact._ENTROPY_MIN_BITS_PER_CHAR == 4.2 + assert redact._ENTROPY_MIN_LEN == 24 + + +# --- test 2: structured recall is unchanged, and entropy still exists -------- + +STRUCTURED_SAMPLES = [ + ("aws-access-key", "AKIAIOSFODNN7EXAMPLE"), + ("github-token", "ghp_" + "A" * 36), + ("slack-token", "xoxb-1234567890-ABCDEFGHIJ"), + ("openai-key", "sk-AbCdEf0123456789XyZwVuTsQq"), + ("pem-private-key", "-----BEGIN RSA PRIVATE KEY-----"), + ("env-secret-assignment", "MY_SECRET=hunter2taffylongvalue"), +] + + +@pytest.mark.parametrize("kind,sample", STRUCTURED_SAMPLES) +def test_every_structured_detector_fires_at_block_tier(kind, sample): + findings = [f for f in redact.scan(sample) if f.kind == kind] + assert findings, f"{kind} lost recall" + assert all(f.tier == redact.TIER_BLOCK for f in findings) + assert all(f.detector_id == kind for f in findings) + + +def test_structured_detectors_all_declare_block_tier(): + """No structured pattern may quietly arrive at notice tier.""" + for kind, _pat in redact._PATTERNS: + assert redact._tier(kind) == redact.TIER_BLOCK + + +def test_entropy_sweep_still_fires_on_a_genuine_high_entropy_token(): + """DEMOTED, NOT DELETED. + + Deleting the sweep would satisfy every path assertion in this file while + removing the only detector that catches an unstructured credential. A + 24-character token drawn from a wide alphabet, with no `/` to be blamed on + the tokenizer, must still be reported. + """ + token = "aZ3kQ9mB7xT2vL5nR8wY4cF6" + assert "/" not in token and len(token) == redact._ENTROPY_MIN_LEN + findings = [f for f in redact.scan(token) if f.kind == "high-entropy"] + assert findings, "the entropy sweep was deleted, not demoted" + assert findings[0].tier == redact.TIER_NOTICE + + +def test_entropy_sweep_fires_inside_ordinary_surrounding_text(): + """The same token embedded in a command line is still found.""" + text = "curl -H 'X-Api: aZ3kQ9mB7xT2vL5nR8wY4cF6' https://example.invalid/v1" + assert any(f.kind == "high-entropy" for f in redact.scan(text)) + + +# --- test 3: locality -------------------------------------------------------- + +_BLOB = ( + "starting run\n" + "no secrets on this line\n" + "leaked ghp_" + "B" * 36 + " in output\n" + "done\n" +) + + +def test_findings_in_a_blob_carry_source_line_and_offset(): + source = redact.FindingSource("stdout", event_index=4) + findings = redact.scan(_BLOB, source) + hit = next(f for f in findings if f.kind == "github-token") + assert hit.source.kind == "stdout" + assert hit.source.event_index == 4 + assert hit.source.argv_index is None + assert hit.line == 3, "line must be 1-based within the scanned text" + # The offset indexes the text that was scanned, so a reader can go get it. + start = hit.source.byte_offset + assert _BLOB[start:start + (hit.span[1] - hit.span[0])] == _BLOB[hit.span[0]:hit.span[1]] + assert hit.fingerprint + + +def test_argv_findings_carry_an_argv_index_and_no_line(): + source = redact.FindingSource("argv", event_index=2, argv_index=3) + hit = redact.scan("MY_TOKEN=ghp_" + "C" * 36, source)[0] + assert hit.source.kind == "argv" + assert hit.source.argv_index == 3 + assert hit.line is None, "an argv element has no lines" + + +def test_findings_scanned_without_a_source_are_unlocated_not_wrong(): + """Back-compat: the original three-field call still works and lies about nothing.""" + hit = redact.scan("AKIAIOSFODNN7EXAMPLE")[0] + assert hit.source is None + assert hit.fingerprint == "" + assert redact.Finding(kind="k", span=(0, 1), preview="p").tier == redact.TIER_NOTICE + + +def test_fingerprint_is_stable_across_runs_over_identical_input(): + source = redact.FindingSource("stdout", event_index=4) + first = [f.fingerprint for f in redact.scan(_BLOB, source)] + second = [f.fingerprint for f in redact.scan(_BLOB, source)] + assert first == second + assert all(len(fp) == 16 for fp in first) + + +def test_fingerprint_locates_and_does_not_digest_the_token(): + """It is a LOCALITY digest, and that is a security property, not a shortcut. + + Digesting the token would publish an unkeyed hash of a short high-entropy + string next to the redaction that removed it — a brute-force oracle for the + thing just protected. So: two different secrets at the same location share a + fingerprint, and the same secret at two locations does not. + """ + source = redact.FindingSource("stdout", event_index=1) + a = redact.scan("ghp_" + "D" * 36, source)[0] + b = redact.scan("ghp_" + "E" * 36, source)[0] + assert a.fingerprint == b.fingerprint # same place, different token + + elsewhere = redact.FindingSource("stdout", event_index=2) + c = redact.scan("ghp_" + "D" * 36, elsewhere)[0] + assert a.fingerprint != c.fingerprint # same token, different place + + +def test_unknown_source_kind_is_refused(): + with pytest.raises(ValueError): + redact.FindingSource("stdin") + + +# --- test 4: the scope inversion is fixed ------------------------------------ + +_LABEL_TOKEN = "ghp_" + "F" * 36 + + +def _one_event(session: Session, repo: Path, code: str): + return run_wrapped([sys.executable, "-c", code], session, repo) + + +def _note_bytes(repo: Path): + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", "HEAD"], + cwd=str(repo), capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +def test_a_secret_in_a_claim_label_blocks_the_seal(repo: Path): + """The exported field that was never scanned. + + A claim label is operator-authored text that travels in the note to + wherever the note travels. Until now the gate never looked at it. + """ + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim( + ctype="tests-pass", + label=f"suite green with {_LABEL_TOKEN}", + event_indices=(0,), + declared_at_index=0, + ), + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo) + assert "github-token" in str(exc.value) + + +def test_a_secret_in_a_local_output_blob_warns_but_does_not_block(repo, capsys): + """The bytes that never leave: reported loudly, not a work stoppage. + + The token is assembled inside the child so it never appears in argv — argv + IS exported (as a redacted preview) and blocking on it is unchanged. + """ + s = _session(repo) + _one_event(s, repo, "print('gh' + 'p_' + 'B' * 36)") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="clean label", event_indices=(0,), + declared_at_index=0), + ) + m = M.seal(s, repo) + assert m.secrets["blocked"] is False + assert m.secrets_override is False + err = capsys.readouterr().err + assert "did not block" in err + assert "github-token" in err + assert "stdout of event 0" in err + + +def test_a_secret_in_argv_still_blocks_and_says_where(repo: Path): + """argv is exported, so argv still blocks — with an address this time.""" + s = _session(repo) + _one_event(s, repo, f"x = 'MY_TOKEN={_LABEL_TOKEN}'") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="clean", event_indices=(0,), declared_at_index=0), + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo) + message = str(exc.value) + assert "at argv of event 0 arg 2" in message + assert _LABEL_TOKEN not in message, "a refusal must not reprint the secret" + + +def test_a_deep_path_in_stdout_does_not_block_and_seals(repo, capsys): + """The regression the tokenizer change is for: a build path is not a secret.""" + s = _session(repo) + _one_event(s, repo, f"print({DEEP_PATHS_THAT_FIRED[0]!r})") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="build ok", event_indices=(0,), + declared_at_index=0), + ) + m = M.seal(s, repo) + assert m.secrets["findings_by_tier"] == {"block": 0, "notice": 0} + assert m.secrets["blocked"] is False + + +def test_allow_secrets_overrides_a_blocking_label(repo: Path): + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label=f"green {_LABEL_TOKEN}", event_indices=(0,), + declared_at_index=0), + ) + m = M.seal(s, repo, allow_secrets=True) + assert m.secrets_override is True + assert m.secrets["blocked"] is True + assert m.secrets["overridden"] is True + + +def test_allow_secrets_on_a_clean_seal_still_records_false(repo: Path): + """`secrets_override` keeps its exact meaning: an override actually happened.""" + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="clean", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo, allow_secrets=True) + assert m.secrets_override is False + assert m.secrets["overridden"] is False + + +# --- test 5: the scan domain is recorded ------------------------------------- + +def test_seal_records_what_was_actually_scanned(repo: Path): + """`secrets_override: false` used to be unfalsifiable — it was the same word + for "nothing was found" and "almost nothing was looked at".""" + s = _session(repo) + _one_event(s, repo, "print('one')") + _one_event(s, repo, "print('two')") + _one_event(s, repo, "print('three')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="first", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo) + secrets = m.secrets + # `claim-fields` is load-bearing in this string, not decoration: the label, + # the pathspecs and the changed paths are scanned as fields by the same + # pass that redacts them. Before that they reached the scan only inside the + # serialized manifest, which blocked a seal over a label and then published + # that label verbatim on --allow-secrets. + assert secrets["scan_domain"] == "claim-bound-events+claim-fields+manifest" + assert secrets["events_total"] == 3 + assert secrets["events_scanned"] == 1 + assert secrets["events_scanned"] <= secrets["events_total"] + assert secrets["bytes_scanned"] > 0 + assert secrets["detector_set_version"] == redact.DETECTOR_SET_VERSION + + +def test_the_secrets_block_survives_the_note_round_trip(repo: Path): + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="ok", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo) + body = _note_bytes(repo) + assert body is not None + parsed = M.Manifest.from_json(body.rstrip(b"\n")) + assert parsed.secrets == m.secrets + + +def test_a_note_without_a_secrets_block_reads_back_additively(): + """A stored v1/v2 note has no `secrets` key and must not become an error.""" + body = json.dumps( + { + "version": 1, + "commit": "0" * 40, + "tree": "1" * 40, + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + "secrets_override": False, + } + ).encode("ascii") + assert M.Manifest.from_json(body).secrets == {} + + +# --- change 4b: the detector-set version is declared, not inferred ----------- + +def test_detector_set_version_is_two_and_the_projection_version_is_not(repo: Path): + """Re-tiering every detector and changing what the entropy sweep matches IS + a detector-set change. The span-partition algorithm is untouched. + """ + assert redact.DETECTOR_SET_VERSION == 2 + assert redact.PROJECTION_VERSION == 1 + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="ok", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(s, repo) + assert m.claims[0]["redaction"]["detector_set_version"] == 2 + assert m.claims[0]["redaction"]["projection_version"] == 1 + assert m.secrets["detector_set_version"] == 2 + + +def test_a_v1_projection_is_distinguishable_from_a_v2_one(repo: Path): + """A consumer reads the declared version instead of inferring didrun's era.""" + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="ok", event_indices=(0,), declared_at_index=0), + ) + fresh = M.seal(s, repo).claims[0]["redaction"] + + stored_v1 = dict(fresh, detector_set_version=1) + assert stored_v1["projection_version"] == fresh["projection_version"] + assert stored_v1["detector_set_version"] != fresh["detector_set_version"] + + +# --- test 6: the refusal enumerates ------------------------------------------ + +def test_secrets_blocked_names_detector_source_and_line(): + source = redact.FindingSource("stdout", event_index=7) + findings = [f for f in redact.scan(_BLOB, source) if f.tier == redact.TIER_BLOCK] + message = str(redact.SecretsBlocked(findings)) + assert "github-token" in message + assert "stdout of event 7" in message + assert "line 3" in message + assert "fingerprint" in message + + +def test_secrets_blocked_caps_the_enumeration_and_counts_the_rest(): + """A refusal that prints 19,900 lines is as unusable as one that prints none.""" + text = "\n".join(f"line {i} ghp_" + "G" * 36 for i in range(50)) + findings = redact.scan(text, redact.FindingSource("manifest")) + assert len(findings) == 50 + message = str(redact.SecretsBlocked(findings)) + listed = [ln for ln in message.splitlines() if ln.startswith(" - ")] + assert len(listed) == redact.ENUMERATION_CAP + assert "and 30 more finding(s) not listed" in message + assert message.splitlines()[0].startswith("export blocked: 50 likely secret(s)") + + +# --- test 7: a blocked seal is free ------------------------------------------ + +def test_a_blocked_seal_advances_nothing(repo: Path): + """Currently unguarded, and it is what makes attempting a seal cheap: the + refusal must happen before the note is published and before the watermark + moves, or a blocked seal silently consumes the claims it refused. + """ + s = _session(repo) + _one_event(s, repo, "print('ok')") + M.declare_claim( + s, + Claim(ctype="tests-pass", label="clean unit", event_indices=(0,), + declared_at_index=0), + ) + M.seal(s, repo) + + seals = s.root / "seals.jsonl" + seals_before = seals.read_bytes() + note_before = _note_bytes(repo) + assert note_before is not None + + M.declare_claim( + s, + Claim(ctype="tests-pass", label=f"second unit {_LABEL_TOKEN}", + event_indices=(0,), declared_at_index=0), + ) + with pytest.raises(redact.SecretsBlocked): + M.seal(s, repo) + + assert seals.read_bytes() == seals_before, "the watermark moved on a refused seal" + assert _note_bytes(repo) == note_before, "a refused seal republished the note" + + # And the refusal is genuinely free: fixing the label and re-sealing works, + # with the refused claim still in scope rather than swallowed. + (s.root / "claims.jsonl").write_bytes( + b"\n".join( + line for line in (s.root / "claims.jsonl").read_bytes().splitlines() + if _LABEL_TOKEN.encode("ascii") not in line + ) + + b"\n" + ) + M.declare_claim( + s, + Claim(ctype="tests-pass", label="second unit", event_indices=(0,), + declared_at_index=0), + ) + m = M.seal(s, repo) + assert [c["claim"]["label"] for c in m.claims] == ["second unit"] + + +# --- test 8: the false-positive pin ------------------------------------------ + +# Realistic developer strings and the tier each is expected to reach. This is a +# table so that any future tuning of the detectors shows up as a diff on a +# reviewed list rather than as a number nobody looked at. `notice` entries are +# not endorsements — they are the entropy sweep's remaining reach, recorded so +# that a change to it is visible. +DEVELOPER_STRINGS = [ + ("commit 356a192b7913b04c54574d18c28d46e6395428ab passed", set()), + ("sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", set()), + ("https://example.invalid/example-org/example-repo/pull/1284#issuecomment-2098765432", set()), + ("npm WARN deprecated request@2.88.2: request has been deprecated", set()), + ("registry.invalid/library/postgres@sha256:1a2b3c4d5e6f70819a2b3c4d5e6f7081", set()), + ("pytest -q tests/test_capture_claims_manifest.py::test_grade_tree_exact", set()), + ("go test ./internal/servicelayer/... -run TestServiceLifecycle -count=1", set()), + ("--define=GOFLAGS=-mod=readonly -trimpath", set()), + ("cargo build --release --target aarch64-apple-darwin", set()), + ("kubectl -n production get pods -l app.kubernetes.io/name=gateway", set()), + ("registry.invalid/example-org/example-service:v1.24.0-rc3", set()), + ("uuid 3f2504e0-4f89-11d3-9a0c-0305e82c3301 assigned", set()), + ("git rebase --onto origin/main feature/long-running-branch-name", set()), + ("Traceback (most recent call last):\n File \"app/main.py\", line 42", set()), + ("ruff check --select E,F,W --ignore E501 src/ tests/", set()), + ("terraform plan -var-file=environments/production/main.tfvars", set()), + ("ORDINARY_PATH " + ORDINARY_PATHS[0], set()), + ("DEEP_PATH " + DEEP_PATHS_THAT_FIRED[0], set()), + # Still reported, and deliberately so: these are the shapes an unstructured + # credential actually takes, and the sweep is the only detector that sees them. + ("Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9", {"notice"}), + ("integrity sha512-abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOP==", {"notice"}), + # And the six that must never be demoted. + ("AKIAIOSFODNN7EXAMPLE", {"block"}), + ("ghp_" + "A" * 36, {"block"}), + ("xoxb-1234567890-ABCDEFGHIJ", {"block"}), + ("sk-AbCdEf0123456789XyZwVuTsQq", {"block"}), + ("-----BEGIN OPENSSH PRIVATE KEY-----", {"block"}), + ("DEPLOY_TOKEN=hunter2taffylongvalue", {"block"}), +] + + +@pytest.mark.parametrize("text,expected", DEVELOPER_STRINGS) +def test_developer_string_tiers_are_pinned(text, expected): + assert {f.tier for f in redact.scan(text)} == expected + + +def test_the_false_positive_table_covers_both_outcomes(): + """A table that only contained clean strings would pass if scan() returned [].""" + tiers = {t for _s, expected in DEVELOPER_STRINGS for t in expected} + assert tiers == {"block", "notice"} diff --git a/tests/test_env_fingerprint.py b/tests/test_env_fingerprint.py new file mode 100644 index 0000000..ac6303a --- /dev/null +++ b/tests/test_env_fingerprint.py @@ -0,0 +1,484 @@ +"""P3.3 — the environment fingerprint is bound to something and read by someone. + +Before this unit the fingerprint was written twice and read nowhere: two green +wrapper events recorded under different umasks were indistinguishable in the +receipt, and the key set gated on the two worst false-drift sources on a +developer machine (PATH, PWD) while omitting the one that voided real chains +(umask). + +The comparison that makes it load-bearing is ACROSS TIME — sealed fingerprint +versus this process's. Comparing the fingerprints inside a single claim would be +vacuous: every claim didrun's CLI can declare binds exactly one event, and one +event trivially agrees with itself, so ``return False`` would pass such a test on +100% of real claims. +""" + +from __future__ import annotations + +import os +import re +import subprocess +import sys +import textwrap +from pathlib import Path + +from didrun import capture +from didrun import cli +from didrun import gitplumbing as gp +from didrun import manifest as M +from didrun import render +from didrun.claims import ( + ENV_DRIFT, + ENV_INCOMPARABLE, + ENV_MATCH, + ENV_NOT_RECORDED, + Claim, +) +from didrun.ledger import Event, Session + +# A structurally equivalent stand-in for a real environment, so no test here +# depends on the machine it runs on. +_ENV = { + "LANG": "en_US.UTF-8", + "SHELL": "/bin/zsh", + "TZ": "UTC", + "VIRTUAL_ENV": "/opt/envs/example", + "PATH": "/usr/bin:/bin", +} + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _ok(repo: Path, session: Session): + return capture.run_wrapped([sys.executable, "-c", "print(1)"], session, repo) + + +def _sealed_green(repo: Path) -> Session: + """One recorded success, one claim over it, sealed on a clean tree.""" + s = _session(repo) + _ok(repo, s) + M.declare_claim( + s, Claim(ctype="tests-pass", label="suite", event_indices=(0,), declared_at_index=0) + ) + M.seal(s, repo) + return s + + +def _seal_with_fingerprint(repo: Path, fingerprint: str) -> Session: + """Seal a green claim over an event carrying ``fingerprint`` verbatim. + + The event is constructed rather than recorded because the point is a + fingerprint THIS didrun would never produce — a bare v0.1 digest, or a v1 + value from before the key set changed. It is appended through the real + ``Session.append`` so the chain covers it exactly as it covers a captured + event; nothing here edits a log. + """ + s = _session(repo) + tree = gp.tree_digest(repo, s.blobs.root) + s.append( + Event( + argv=("example-suite", "--quiet"), + cwd=str(repo), + env_fingerprint=fingerprint, + observed_via="wrapper", + coverage="complete", + exit_code=0, + started_at=1.0, + ended_at=2.0, + tree_before=tree, + tree_after=tree, + ) + ) + M.declare_claim( + s, Claim(ctype="tests-pass", label="suite", event_indices=(0,), declared_at_index=0) + ) + M.seal(s, repo) + return s + + +def _run_child(script: str, *args: str, env=None, cwd=None) -> str: + proc = subprocess.run( + [sys.executable, "-c", textwrap.dedent(script), *args], + capture_output=True, + text=True, + check=True, + env=env, + cwd=cwd, + ) + return proc.stdout.strip() + + +# --- test 1: the falsification test (criterion per red-team §3.1(3)) --------- + +def test_a_changed_TZ_drifts_the_verdict_and_refuses_under_require_env_match( + repo: Path, monkeypatch, capsys +): + """Seal green, verify under a different TZ, and the verdict must move. + + THE FALSIFICATION: a no-op implementation fails BOTH halves. If nothing + compares the fingerprint across time (the pre-P3.3 state, or a reader that + always answers "match"), the two verdict texts are identical and + --require-env-match --strict still exits 0. Passing requires the comparison + to exist AND to be able to refuse. + + TZ is the mover on purpose: it is in the gate, it is trivially settable, and + it is not PATH or PWD — both of which are now deliberately unable to do + this. + """ + monkeypatch.setenv("TZ", "UTC") + s = _sealed_green(repo) + + before = render.render_verdict(M.verify(s, repo)) + assert "1/1 claims recorded-exact" in before + assert f"env: 1 {ENV_MATCH} / 0 drifted / 0 {ENV_INCOMPARABLE}" in before + assert ENV_DRIFT not in before + + monkeypatch.setenv("TZ", "Asia/Tokyo") + after = render.render_verdict(M.verify(s, repo)) + assert after != before + assert ENV_DRIFT in after + assert "0 match / 1 drifted" in after + # Drift did not un-record the command: the grade is untouched. + assert "1/1 claims recorded-exact" in after + + # Advisory by default... + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + capsys.readouterr() + # ...and a refusal on request. + assert cli.main(["--repo", str(repo), "verify", "--require-env-match", "--strict"]) == 1 + out = capsys.readouterr().out + assert ENV_DRIFT in out + + +def test_drift_alone_does_not_change_the_grade_or_the_default_verdict( + repo: Path, monkeypatch +): + """`all_verified` keeps its default membership. Drift is a fact about this + machine now, not about whether the recorded command ran.""" + monkeypatch.setenv("TZ", "UTC") + s = _sealed_green(repo) + monkeypatch.setenv("TZ", "Asia/Tokyo") + + drifted = M.verify(s, repo) + assert drifted.env_drift_count == 1 + assert [r.grade for r in drifted.results] == ["tree-exact"] + assert drifted.all_verified is True + assert drifted.worst_status == "tree-exact" + + gated = M.verify(s, repo, require_env_match=True) + assert gated.all_verified is False + # Still not a grade: the claim is recorded-exact and says so. + assert [r.grade for r in gated.results] == ["tree-exact"] + + +# --- test 2: umask is bound -------------------------------------------------- + +_RECORD_UNDER_UMASK = """ + import os, sys + os.umask(int(sys.argv[3], 8)) + from pathlib import Path + from didrun.capture import run_wrapped + from didrun.ledger import Session + ev = run_wrapped( + [sys.executable, "-c", "pass"], Session(Path(sys.argv[2])), Path(sys.argv[1]) + ) + print(ev.env_fingerprint) +""" + + +def test_umask_is_bound_into_the_gate(repo: Path, tmp_path: Path): + """Two events recorded under umask 022 and 077 must differ in the gate. + + This is the case that voided three chains and was invisible in the receipt: + same argv, same tree, same exit code, two different file-permission + outcomes, one indistinguishable pair of green events. + + Recorded in child processes because the umask is read once per process and + reused (capture.process_umask) — which is itself the property under test: + the value must reach the ledger, not merely the helper's return value. + """ + prints = [ + _run_child( + _RECORD_UNDER_UMASK, str(repo), str(tmp_path / f"ledger-{mask}"), mask + ) + for mask in ("022", "077") + ] + assert all(capture.fingerprint_version(p) == 2 for p in prints) + assert capture.fingerprint_gate(prints[0]) != capture.fingerprint_gate(prints[1]) + # Only the umask moved: the non-gated PATH digest is unchanged. + assert capture.fingerprint_path_digest(prints[0]) == capture.fingerprint_path_digest( + prints[1] + ) + + +def test_umask_reaches_the_fingerprint_directly_too(): + """The same fact without a subprocess, so a failure localizes.""" + a = capture.env_fingerprint(_ENV, umask=0o022) + b = capture.env_fingerprint(_ENV, umask=0o077) + assert capture.fingerprint_gate(a) != capture.fingerprint_gate(b) + assert capture.env_fingerprint(_ENV, umask=0o022) == a + + +# --- test 3: PWD no longer drifts -------------------------------------------- + +_RECORD_FROM_CWD = """ + import sys + from pathlib import Path + from didrun.capture import run_wrapped + from didrun.ledger import Session + ev = run_wrapped( + [sys.executable, "-c", "pass"], Session(Path(sys.argv[2])), Path(sys.argv[1]) + ) + print(ev.env_fingerprint) +""" + + +def test_the_invocation_directory_does_not_drift_the_gate(repo: Path, tmp_path: Path): + """The same command against the same repo, invoked from two directories. + + PWD was in v0.1's key set and is NOT the child's cwd — run_wrapped always + spawns with cwd=str(repo). Gating on it would mean two byte-identical + executions disagree because the operator was standing somewhere else. + + Both the real child cwd and the PWD variable are moved, because a shell sets + the second and a launcher may not set it at all. + """ + here = tmp_path / "here" + there = tmp_path / "there" + here.mkdir() + there.mkdir() + prints = [ + _run_child( + _RECORD_FROM_CWD, + str(repo), + str(tmp_path / f"ledger-{d.name}"), + env={**os.environ, "PWD": str(d)}, + cwd=str(d), + ) + for d in (here, there) + ] + assert capture.fingerprint_gate(prints[0]) == capture.fingerprint_gate(prints[1]) + assert prints[0] == prints[1] + + +def test_PWD_is_not_in_the_preimage_at_all(): + assert "PWD" not in capture._ENV_FINGERPRINT_KEYS + a = capture.env_fingerprint({**_ENV, "PWD": "/home/example/one"}, umask=0o022) + b = capture.env_fingerprint({**_ENV, "PWD": "/home/example/two"}, umask=0o022) + assert a == b + + +# --- test 4: PATH is visible but not a gate ---------------------------------- + +def test_PATH_moves_the_recorded_digest_but_never_the_gate(): + """Two different PATHs: same gate, different `path=` suffix. + + PATH varies with terminal tab, direnv, `nix develop`, tmux and homebrew + shellenv ordering. Gating on it produces a refusal that fires on a new shell + and is ignored within a week — but dropping it entirely would lose a real + signal, so it is recorded beside the gate instead of inside it. + """ + assert "PATH" not in capture._ENV_FINGERPRINT_KEYS + a = capture.env_fingerprint({**_ENV, "PATH": "/usr/bin:/bin"}, umask=0o022) + b = capture.env_fingerprint({**_ENV, "PATH": "/opt/example/bin:/bin"}, umask=0o022) + assert capture.fingerprint_gate(a) == capture.fingerprint_gate(b) + assert capture.fingerprint_path_digest(a) != capture.fingerprint_path_digest(b) + + +def test_a_PATH_difference_is_an_advisory_and_never_drift(repo: Path, monkeypatch): + """A verdict must not call a PATH change environment drift.""" + monkeypatch.setenv("PATH", os.environ.get("PATH", "/usr/bin:/bin")) + s = _sealed_green(repo) + monkeypatch.setenv("PATH", "/opt/example/bin:" + os.environ.get("PATH", "")) + + report = M.verify(s, repo) + assert [r.env_status for r in report.results] == [ENV_MATCH] + assert report.env_drift_count == 0 + assert "PATH differs" in report.results[0].env_reason + assert "recorded, not gated" in report.results[0].env_reason + assert M.verify(s, repo, require_env_match=True).all_verified is True + + +# --- test 5: legacy is incomparable, not drifted ----------------------------- + +def test_a_bare_v0_1_fingerprint_is_incomparable_and_never_renders_as_drift( + repo: Path, +): + """The archived-evidence case, and the reason this unit is compat-safe. + + A v0.1 digest carries no version prefix, so it is not a measurement of the + same key set. Reporting a mismatch against it as drift would make every note + published before v0.2 look like the environment moved. + """ + s = _seal_with_fingerprint(repo, "0123456789abcdef") + report = M.verify(s, repo) + + assert [r.env_status for r in report.results] == [ENV_INCOMPARABLE] + assert "unversioned v0.1 digest" in report.results[0].env_reason + assert "not evidence that anything drifted" in report.results[0].env_reason + + text = render.render_verdict(report) + assert ENV_DRIFT not in text + assert f"0 drifted / 1 {ENV_INCOMPARABLE}" in text + assert ENV_DRIFT not in render.render_html(report) + + # Incomparable never refuses, under either mode. This is the assertion that + # would break the corpus if it stopped holding. + assert report.all_verified is True + assert M.verify(s, repo, require_env_match=True).all_verified is True + + +def test_a_v1_fingerprint_from_an_older_didrun_is_incomparable_too(repo: Path): + """Sealed evidence from P0.3-era didrun (versioned, older key set). + + The version prefix exists precisely so this case is legible: v1 and v2 are + digests of different key sets, so they are incomparable by construction + rather than by luck. + """ + s = _seal_with_fingerprint(repo, "v1:0123456789abcdef") + report = M.verify(s, repo) + assert [r.env_status for r in report.results] == [ENV_INCOMPARABLE] + assert "version 1" in report.results[0].env_reason + assert "incomparable, not drifted" in report.results[0].env_reason + assert M.verify(s, repo, require_env_match=True).all_verified is True + + +def test_a_note_that_bound_no_fingerprint_is_not_recorded_not_drifted(repo: Path): + """A v0.1-shaped note has no evidence block at all — a third distinct fact. + + Reported separately from `incomparable` because "the seal never recorded + one" and "it recorded one this binary cannot compare" are different, and + collapsing them would hide which notes could start being compared. + """ + s = _sealed_green(repo) + head = gp.head_commit(repo) + body = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", head], + cwd=str(repo), + capture_output=True, + text=True, + check=True, + ).stdout + stripped = body.replace('"evidence"', '"evidence_removed"') + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-F", "-", head], + cwd=str(repo), + input=stripped.encode("ascii"), + capture_output=True, + check=True, + ) + + report = M.verify(s, repo) + assert [r.env_status for r in report.results] == [ENV_NOT_RECORDED] + assert "bound no environment fingerprint" in report.results[0].env_reason + text = render.render_verdict(report) + assert f"1 {ENV_NOT_RECORDED}" in text + assert ENV_DRIFT not in text + assert M.verify(s, repo, require_env_match=True).all_verified is True + + +# --- test 6: determinism ----------------------------------------------------- + +_FINGERPRINT_IN_CHILD = """ + import json, sys + from didrun import capture + env = json.loads(sys.argv[1]) + print(capture.env_fingerprint(env, umask=int(sys.argv[2], 8))) +""" + + +def test_fingerprint_is_reproducible_across_processes(tmp_path: Path): + """Determinism is a trust-path invariant (no wall clock, no randomness). + + Mirrors the cross-process chain-hash check in tests/test_ledger.py: a + same-process comparison cannot catch hash randomization or key-order + instability, because both are fixed for a process's lifetime. + """ + import json + + out = [ + _run_child(_FINGERPRINT_IN_CHILD, json.dumps(_ENV, sort_keys=True), "022") + for _ in range(2) + ] + assert out[0] == out[1] + assert out[0] == capture.env_fingerprint(_ENV, umask=0o022) + + +def test_the_umask_read_is_cached_not_repeated(monkeypatch): + """Read once, reuse. The read is a read-modify-write on process-global + state, so doing it per event would open that window on every wrapped + command.""" + monkeypatch.setattr(capture, "_UMASK", None) + reads = [] + real_umask = os.umask + + def counting_umask(mask): + reads.append(mask) + return real_umask(mask) + + monkeypatch.setattr(capture.os, "umask", counting_umask) + first = capture.process_umask() + assert reads == [0, first] # set to 0, then restored + for _ in range(5): + assert capture.process_umask() == first + assert reads == [0, first] # and never touched again + + +# --- test 7: no leak --------------------------------------------------------- + +def test_the_fingerprint_leaks_no_input_value(): + """Values are digested, never stored. The fingerprint must never become the + place a secret-bearing environment variable lands verbatim in a note.""" + inputs = { + "LANG": "xq_ZZ.KOI8-R", + "SHELL": "/opt/quixotic/bin/wobbleshell", + "TZ": "Antarctica/South_Pole", + "VIRTUAL_ENV": "/opt/envs/zephyrous-marmoset-3141", + "PATH": "/opt/quixotic/bin:/usr/lib/kumquat/sbin", + } + value = capture.env_fingerprint(inputs, umask=0o022) + assert re.fullmatch(r"v2:[0-9a-f]{16}:path=[0-9a-f]{8}", value) + for raw in inputs.values(): + assert raw not in value + for i in range(len(raw) - 3): + assert raw[i : i + 4] not in value + + +def test_the_drift_reason_carries_digests_not_environment_values( + repo: Path, monkeypatch +): + """The reason names both gates. They are 16-hex digests, so naming them + leaks nothing — which is what makes the message safe to print.""" + monkeypatch.setenv("TZ", "UTC") + s = _sealed_green(repo) + monkeypatch.setenv("TZ", "Antarctica/South_Pole") + reason = M.verify(s, repo).results[0].env_reason + assert "Antarctica" not in reason + assert len(re.findall(r"\b[0-9a-f]{16}\b", reason)) == 2 + + +# --- the parsers ------------------------------------------------------------- + +def test_fingerprint_accessors_refuse_rather_than_guess(): + value = capture.env_fingerprint(_ENV, umask=0o022) + assert capture.fingerprint_version(value) == 2 + assert capture.fingerprint_gate(value) == value.split(":")[1] + assert capture.fingerprint_path_digest(value) == value.split("path=")[1] + + # A bare v0.1 digest: no version, no gate, no path digest — INCOMPARABLE at + # every accessor, so no consumer can accidentally compare one. + assert capture.fingerprint_version("0123456789abcdef") is None + assert capture.fingerprint_gate("0123456789abcdef") is None + assert capture.fingerprint_path_digest("0123456789abcdef") is None + + # A v1 value: comparable within v1, and carries no path digest. + assert capture.fingerprint_version("v1:0123456789abcdef") == 1 + assert capture.fingerprint_gate("v1:0123456789abcdef") == "0123456789abcdef" + assert capture.fingerprint_path_digest("v1:0123456789abcdef") is None + + # Malformed shapes are refusals, never empty-string matches. + assert capture.fingerprint_gate("v2:") is None + assert capture.fingerprint_path_digest("v2:0123456789abcdef:") is None + assert capture.fingerprint_path_digest("v2:0123456789abcdef:other=aa") is None diff --git a/tests/test_projection_contract.py b/tests/test_projection_contract.py new file mode 100644 index 0000000..f483939 --- /dev/null +++ b/tests/test_projection_contract.py @@ -0,0 +1,342 @@ +"""P3.1 — the exported redaction projection is a reconstructible span partition. + +The criterion here is EXACT RECONSTRUCTION, not "the output's non-marker +characters are a subsequence of the input". The subsequence version was proposed +first and it does not detect the bug it targets: deletion always preserves the +subsequence property, so a redactor that deletes arbitrary text satisfies it. +`test_subsequence_criterion_is_insufficient` pins that, so the weaker check +cannot be reintroduced as an improvement. + +Every credential-shaped string below is synthetic and structural — a shape the +detectors match, never a real token. +""" + +from __future__ import annotations + +import random +import subprocess +import sys +from pathlib import Path + +from didrun import manifest as M +from didrun import redact +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + + +# The marker format is written out here rather than imported: `_REDACTION_MARKER` +# is a frozen part of the exported format, so the test should fail if it moves. +def _marker(kind: str) -> str: + return "«redacted:" + kind + "»" + + +def _reconstruct(source: str, result) -> str: + """Independent reimplementation of the reconstruction contract. + + Deliberately NOT `redact.reconstruct`: a test that calls the same helper the + source uses can only check that the helper agrees with itself. Kept slices + come from the OUTPUT, replaced slices from the source, and markers are + located by arithmetic — never searched for, because an argv is allowed to + contain the literal marker text. + """ + rebuilt = [] + out_i = 0 + src_i = 0 + for span in result.applied: + keep = span.start - src_i + assert keep >= 0, f"applied spans overlap or go backwards: {result.applied}" + rebuilt.append(result.text[out_i:out_i + keep]) + out_i += keep + marker = _marker(span.kind) + assert result.text[out_i:out_i + len(marker)] == marker, ( + f"no marker at offset {out_i} for span {span}" + ) + out_i += len(marker) + rebuilt.append(source[span.start:span.end]) + src_i = span.end + rebuilt.append(result.text[out_i:]) + return "".join(rebuilt) + + +# Both red-team repros verbatim, plus the nested case from the overlap finding. +# Synthetic token bodies; nothing here is a credential. +_GHP = "ghp_" + "A" * 36 +_SK = "sk-AbCdEf0123456789XyZwVuTsQq" + +CASES = [ + "run --key MY_TOKEN=" + _GHP + " && echo done", + 'sh -c "OPENAI_API_KEY=' + _SK + ' pytest -q" && rm -rf /tmp/x', + "MY_SECRET=sk-AAAAAAAAAAAAAAAAAAAA tail words here", + "plain text with nothing to hide", + "AKIAIOSFODNN7EXAMPLE at the start", + "trailing secret AKIAIOSFODNN7EXAMPLE", + "two MY_TOKEN=" + _GHP + " secrets API_SECRET=" + _SK + " in one string", + # A hostile argv containing the literal marker text. Reconstruction must + # still be exact: markers are located by offset, not by search. + "echo " + _marker("openai-key") + " MY_TOKEN=" + _GHP + " done", +] + + +def test_exact_reconstruction(): + """The criterion (red-team HIGH-7): substituting each marker back with the + source span the projection records reproduces the input byte for byte.""" + for source in CASES: + result = redact.redact(source) + assert _reconstruct(source, result) == source, f"lost content in: {source!r}" + + +def test_applied_spans_are_disjoint_and_ascending(): + for source in CASES: + spans = redact.redact(source).applied + for prev, nxt in zip(spans, spans[1:]): + assert prev.end <= nxt.start, f"overlapping partition in {source!r}: {spans}" + for span in spans: + assert 0 <= span.start < span.end <= len(source) + + +def test_length_identity(): + for source in CASES: + result = redact.redact(source) + widths = sum(s.end - s.start for s in result.applied) + markers = sum(len(_marker(s.kind)) for s in result.applied) + assert len(result.text) == len(source) - widths + markers, source + + +_BENIGN_WORDS = [ + "run", "make", "test", "--verbose", "-q", "build", "src/app.py", + "&&", "echo", "done", "pytest", "lint", "config.toml", "the", "unit", +] +_SECRET_SHAPES = [ + "AKIAIOSFODNN7EXAMPLE", + _GHP, + _SK, + "MY_TOKEN=" + _GHP, + "API_SECRET=hunter2taffylongvalue", + "xoxb-1234567890-abcdefghijkl", + "sk-proj-AbCdEf0123456789XyZwVuTsQq", +] + + +def test_property_reconstruction_and_length_over_seeds(): + """Random benign text with k injected secrets, over 200 FIXED seeds. + + Seeded per case and never from the global `random` state: determinism is a + hard invariant, and a property test that cannot be replayed is not evidence. + """ + for seed in range(200): + rng = random.Random(seed) + words = [rng.choice(_BENIGN_WORDS) for _ in range(rng.randint(4, 20))] + for _ in range(rng.randint(1, 4)): + words.insert(rng.randint(0, len(words)), rng.choice(_SECRET_SHAPES)) + source = " ".join(words) + result = redact.redact(source) + assert _reconstruct(source, result) == source, f"seed {seed}: {source!r}" + widths = sum(s.end - s.start for s in result.applied) + markers = sum(len(_marker(s.kind)) for s in result.applied) + assert len(result.text) == len(source) - widths + markers, f"seed {seed}" + + +def _is_subsequence(needle: str, haystack: str) -> bool: + it = iter(haystack) + return all(ch in it for ch in needle) + + +def test_subsequence_criterion_is_insufficient(): + """Why the criterion is exact reconstruction and not a subsequence check. + + v0.1 replaced overlapping spans end-first, so the outer span's stale end + offset ran past the shortened string and ate whatever followed. Below is + that measured output. The proposed subsequence assertion PASSES on it, + because deletion always preserves the subsequence property — which is why + it was replaced. Do not reintroduce it as a simplification. + """ + source = CASES[0] + v01_buggy_output = "run --key " + _marker("env-secret-assignment") + non_marker = v01_buggy_output.replace(_marker("env-secret-assignment"), "") + + assert _is_subsequence(non_marker, source) # the weaker criterion passes + assert "&& echo done" not in v01_buggy_output # on output missing 12 bytes + + # The criterion that actually catches it. + fake = redact.RedactionResult( + text=v01_buggy_output, + findings=[], + applied=[redact.AppliedSpan(10, 59, "env-secret-assignment", + "env-secret-assignment")], + ) + assert _reconstruct(source, fake) != source + + # And the current redactor does not produce it. + assert redact.redact(source).text != v01_buggy_output + + +def test_merge_labels_the_widest_finding_not_the_first(): + """Merge policy over a chain that no single finding covers. + + A(0,10) then B(5,30) then C(6,15): the merged extent is (0,30), which + belongs to no finding, and the label must track the widest contributor + (B) rather than whichever one opened the run. Hand-built findings, because + a three-way straddle like this is not reachable from the current detector + set — the policy still has to be right when it becomes reachable. + """ + findings = [ + redact.Finding("high-entropy", (0, 10), ""), + redact.Finding("high-entropy", (5, 30), ""), + redact.Finding("openai-key", (6, 15), ""), + ] + (span,) = redact.merge_findings(findings) + assert (span.start, span.end) == (0, 30) + assert span.kind == "high-entropy" + + # Same extents, but now the widest is the structured one. + findings[1] = redact.Finding("github-token", (5, 30), "") + (span,) = redact.merge_findings(findings) + assert span.kind == "github-token" + + # Exact-width tie: the structured detector wins over the entropy sweep. + (span,) = redact.merge_findings([ + redact.Finding("high-entropy", (0, 20), ""), + redact.Finding("aws-access-key", (0, 20), ""), + ]) + assert span.kind == "aws-access-key" + + +def test_nested_spans_merge_to_one_outermost(): + source = "MY_SECRET=sk-AAAAAAAAAAAAAAAAAAAA tail words here" + result = redact.redact(source) + assert len(result.applied) == 1, result.applied + span = result.applied[0] + assert (span.start, span.end) == (0, 33) + # The wider structured detector labels the merged span, not the inner one. + assert span.kind == "env-secret-assignment" + assert span.detector_id == "env-secret-assignment" + assert not span.withheld + assert result.text == _marker("env-secret-assignment") + " tail words here" + + +def test_sk_proj_keys_are_recalled(): + """Finding 18: `\\bsk-[A-Za-z0-9]{20,}\\b` missed every `sk-proj-` key.""" + kinds = {f.kind for f in redact.scan("sk-proj-AbCdEf0123456789XyZwVuTsQq")} + assert "openai-key" in kinds + + # The leading \b still holds: an `sk-` inside a word is not a key. + benign = "risk-assessment-framework-rollout-plan" + assert not any(f.kind == "openai-key" for f in redact.scan(benign)) + + +def test_projection_is_stable_for_the_same_input(): + """No wall clock, no randomness: the same argv projects identically.""" + argv = ["run", "--key", "MY_TOKEN=" + _GHP, "&&", "echo", "done"] + assert redact.redact_argv(argv) == redact.redact_argv(argv) + + +def test_withheld_element_is_never_an_empty_string(monkeypatch): + """Redact-or-withhold, never mangle — and never silently. + + The post-condition cannot fail once findings are merged, so this forces it. + The honesty risk is a withheld element exported as an empty preview, which + reads as "nothing ran here" instead of "this was suppressed". + """ + def broken(source, result): + raise ValueError("forced post-condition failure") + + monkeypatch.setattr(redact, "reconstruct", broken) + out = redact.redact_argv(["safe", "MY_TOKEN=" + _GHP]) + assert out.argv[0] == "«withheld:unknown»" # no findings -> unknown + assert out.argv[1] == "«withheld:env-secret-assignment»" + assert "" not in out.argv + assert all(e["withheld"] is True for e in out.applied) + assert [e["argv_index"] for e in out.applied] == [0, 1] + + +# --- the README guarantee ---------------------------------------------------- + +_README = Path(__file__).resolve().parents[1] / "README.md" + +# Locked Decision 10: every guarantee in README.md / TRUST_MODEL.md must have a +# test that fails when it stops being true. The removed sentence claimed the +# HTML report was safe to attach to a PR because "secrets are redacted from +# every export" — the HTML path applies no redaction at all. Two assertions, +# not one: a bare negative is satisfied by replacing one false sentence with a +# different false sentence. +# +# The sentence widened once the claim FIELDS started being redacted too, and +# the narrowing that replaced it has to stay narrow: the HTML report's file +# list for a stale claim is recomputed against the working tree at verify time, +# so it is NOT the note's redacted copy. Both halves are asserted, so a future +# edit cannot quietly drop the caveat while keeping the reassuring half. +_REMOVED_CLAIM = "secrets are redacted from every export" +_REPLACEMENT = ( + "Redaction covers the sealed note: every exported claim string — the " + "`argv_preview`, the label, the pathspecs and the changed paths — is " + "redacted, and the projection is declared in the manifest." +) +_CAVEAT = ( + "a stale claim's file list is recomputed against your working tree at " + "verify time and is shown as it is on disk" +) + + +def test_readme_does_not_claim_every_export_is_redacted(): + text = " ".join(_README.read_text(encoding="utf-8").split()) + assert _REMOVED_CLAIM not in text + + +def test_readme_states_what_is_actually_redacted(): + text = " ".join(_README.read_text(encoding="utf-8").split()) + assert _REPLACEMENT in text + + +def test_readme_keeps_the_caveat_the_widened_claim_needs(): + """The widened sentence is only true because of what follows it.""" + text = " ".join(_README.read_text(encoding="utf-8").split()) + assert _CAVEAT in text + + +# --- the projection reaches the note ---------------------------------------- + +def _git(repo: Path, *a: str) -> None: + subprocess.run(["git", *a], cwd=str(repo), capture_output=True, text=True, check=True) + + +def test_sealed_note_declares_the_projection(repo: Path): + """Test 8: after a seal, `redaction.applied` names exactly the spans that + were replaced, and a consumer can tell which argv elements were touched.""" + session = Session(repo / ".didrun") + secret_arg = "MY_TOKEN=" + _GHP + run_wrapped([sys.executable, "-c", "print(1)", secret_arg], session, repo) + M.declare_claim( + session, + Claim(ctype="tests-pass", label="t", event_indices=(0,), declared_at_index=0), + ) + m = M.seal(session, repo, allow_secrets=True) + + entry = m.claims[0] + block = entry["redaction"] + assert block["projection_version"] == redact.PROJECTION_VERSION + assert block["detector_set_version"] == redact.DETECTOR_SET_VERSION + + recorded = list(session.events()[0].argv) + preview = entry["claim"]["argv_preview"] + assert len(preview) == len(recorded) + + touched = {e["argv_index"] for e in block["applied"]} + assert touched == {3} + for i, element in enumerate(recorded): + if i not in touched: + assert preview[i] == element # untouched elements travel verbatim + + (applied,) = block["applied"] + assert applied["detector_id"] == "env-secret-assignment" + assert applied["withheld"] is False + assert (applied["start"], applied["end"]) == (0, len(secret_arg)) + + # A consumer reconstructs the redacted element from the declared span. + rebuilt = ( + recorded[3][:applied["start"]] + + _marker(applied["detector_id"]) + + recorded[3][applied["end"]:] + ) + assert preview[3] == rebuilt + assert _GHP not in " ".join(preview) diff --git a/tests/test_redact_render.py b/tests/test_redact_render.py index 458d11d..034f4f6 100644 --- a/tests/test_redact_render.py +++ b/tests/test_redact_render.py @@ -6,7 +6,14 @@ from didrun import redact from didrun import render -from didrun.claims import GradeResult, Claim +from didrun.claims import ( + ENV_DRIFT, + ENV_INCOMPARABLE, + ENV_MATCH, + ENV_NOT_RECORDED, + GradeResult, + Claim, +) from didrun.gitplumbing import PathChange @@ -35,16 +42,20 @@ def test_benign_git_hash_not_flagged(): def test_redact_replaces_with_marker(): text = "token sk-ABCDEFGHIJKLMNOPQRSTUVWX here" - out, findings = redact.redact(text) - assert findings - assert "sk-ABCDEFGHIJKLMNOPQRSTUVWX" not in out - assert "redacted" in out + result = redact.redact(text) + assert result.findings + assert "sk-ABCDEFGHIJKLMNOPQRSTUVWX" not in result.text + assert "redacted" in result.text # --- render helpers ---------------------------------------------------------- def _report(results, all_verified=False, worst="stale", chain="intact", - chain_index=None, chain_why=""): + chain_index=None, chain_why="", require_env_match=False): + # Bound outside the class body on purpose: a class-body default may not read + # a name the class body itself also assigns (LOAD_NAME, not LOAD_CLASSDEREF). + gating = require_env_match + @dataclass class R: commit: str = "abcdef123456" @@ -61,11 +72,21 @@ class R: chain_status: str = chain chain_broken_index: object = chain_index chain_reason: str = chain_why + # Mirrors VerifyReport's environment fields, for the same reason. + require_env_match: bool = gating @property def worst_status(self): return worst + @property + def env_counts(self): + counts = {ENV_MATCH: 0, ENV_DRIFT: 0, ENV_INCOMPARABLE: 0, + ENV_NOT_RECORDED: 0} + for r in self.results: + counts[r.env_status] = counts.get(r.env_status, 0) + 1 + return counts + @property def all_verified(self): return all_verified diff --git a/tests/test_seal_publication.py b/tests/test_seal_publication.py index 9d60cf6..b3f72a0 100644 --- a/tests/test_seal_publication.py +++ b/tests/test_seal_publication.py @@ -257,8 +257,15 @@ def test_happy_path_cli_still_exits_zero(repo: Path): def test_manifest_json_bytes_are_pinned(): - """The manifest's on-the-wire bytes are compat surface across 65 sealed - notes. This seal change moves no note byte; a `to_json` tidy-up would. + """The manifest's on-the-wire bytes are compat surface across every sealed + note. A `to_json` tidy-up would move them silently; this pin makes any move + a deliberate diff. + + P3.2 adds exactly one key, `secrets`, whose default is `{}`. Every key a + stored note carries still comes back with an equal value, so the addition + is additive with a v1-reproducing default — which is why this is a one-line + diff here plus a one-line diff in the compat allowlist, and not a + MANIFEST_VERSION bump. """ m = M.Manifest( version=1, @@ -290,6 +297,6 @@ def test_manifest_json_bytes_are_pinned(): b'"supporting_event_index":0}],' b'"commit":"0000000000000000000000000000000000000000",' b'"coverage":{"by_coverage":{"complete":1},"total_events":1},' - b'"secrets_override":false,' + b'"secrets":{},"secrets_override":false,' b'"tree":"1111111111111111111111111111111111111111","version":1}' ) From 229ffd8c2cff25df8d19d1f32719bf059151a1b5 Mon Sep 17 00:00:00 2001 From: Drew Date: Wed, 29 Jul 2026 16:53:56 -0700 Subject: [PATCH 4/8] claims: supersede and conjoin, so a fix-verify loop can converge Two claim-layer additions and one remediation of the reader they exposed. Supersession. A repair re-declares the same gate, and a later claim carrying the same (ctype, label) now retires every earlier one in the same seal window. That is what lets a fix clear a failed or stale claim with no intervening commit; before this the only escapes were seal-early, which needs a commit, and re-seal, which truncated the record, so discarding the ledger was the rational third. Supersession subtracts from the verdict and never from the record: every claim in the window is still graded and published, the retired entry keeps its grade and reason verbatim, and both surfaces mark it and count it. Retrying a gate until it passes is therefore visible rather than prevented, which is the honest bound and strictly better than the invisible truncation it replaces. The mark is validated on read, not trusted, and an all-superseded window grades unknown rather than falling through all([]) to a vacuous green. Conjunction. A new claim type that grades no evidence of its own: it names other claims in the same window by label and grades as the worst of them, so it can never come out better than any one of its members. It binds no event, and verify resolves it in a second pass against the results in that same note -- never against the live claims.jsonl, which would let a claim declared after the seal change an old commit's verdict. This substitutes for nothing. Every conjunct keeps its own witnessed exit code and its own tree comparison, and the design review's refusal to accept a transition marker in place of a recorded execution still stands untouched. The vocabulary change bumps MANIFEST_VERSION to 3, which is the first bump that is not additive in either direction, and docs/COMPAT.md states the break at full strength: an older reader meeting a v3 note raises out of the command rather than refusing gracefully. Because the bump makes the version guard the thing that whole story rests on, the guard was measured -- and it did not hold. Over 22 malformed note shapes the reader left an uncaught traceback on eleven and exited 1, the same code --strict uses for a note that graded badly, so a corrupt note could not be told from an honest failure by exit status. Four shapes were accepted outright: a note whose version was the JSON value true verified green at exit 0, because bool is an int subclass and True > 3 is False, and 2.5, 0 and -1 did the same. So an unreadable note is now a graded refusal with a reason naming what is wrong, for every shape rather than the remembered ones. version is read with the same rule the supersession mark gets and must be an integer of at least 1; the required fields are checked; and a claim type this binary cannot parse is a refusal instead of a ClaimError escaping verify, which closes from this end the hole COMPAT.md documents from the other. The two refusals stay distinguishable because the tree-fallback scan depends on it: a malformed body is skipped and counted so one foreign note cannot hide every good one, while a version above this reader's maximum propagates, since scanning past it would report an older note's verdict as if it were current. That split used to happen by accident, the scan catching bare Exception. This is fail-closed reading and not a forgery barrier; whoever can rewrite a note can rewrite a grade directly. Nothing legitimately published is newly refused -- every note in the world carries an integer version of 1, 2 or 3 and the required fields, and the compat corpus replays all three. Also redacts reason for every claim rather than exporting it as generated text. A conjunction's reason names a conjunct, which is operator-authored, so the field stopped being generated-only; covering it for all claim types means a later version that interpolates something into it is covered without anyone remembering. And a narrowing re-seal is refused unless --reseal is passed: git notes add -f force-replaces, and a seal scoped to all_claims[watermark:] was silently publishing a narrow note over a wide one and destroying a record that exists nowhere else. Three known-not-closed items are recorded in docs/COMPAT.md rather than dropped: a duplicate conjunct label is counted twice in the reason, the no-events refusal for a conjunction borrows a message about binding events, and a conjunction's redacted reason is not guaranteed to contain its redacted conjunct label as a substring since the two are scanned as separate fields. None can produce a grade the evidence does not back. Suite 435 passed 4 skipped, up from 405 with no regressions; harness.recall freeze INTACT, defect-class recall 100.0%, VERDICT PASS; the changed files parse at feature_version 3.11 and the refusal paths were exercised under a real 3.11.14; runtime dependencies remain empty. --- README.md | 5 +- docs/COMPAT.md | 184 +++++++- src/didrun/claims.py | 202 ++++++++- src/didrun/cli.py | 111 ++++- src/didrun/manifest.py | 550 ++++++++++++++++++++-- src/didrun/redact.py | 7 +- src/didrun/render.py | 113 ++++- tests/compat/synthetic.py | 114 ++++- tests/compat/test_corpus_replay.py | 15 +- tests/test_claim_supersession.py | 625 +++++++++++++++++++++++++ tests/test_conjunction_claim.py | 628 ++++++++++++++++++++++++++ tests/test_detector_tiers.py | 7 +- tests/test_projection_contract.py | 11 +- tests/test_redact_render.py | 12 + tests/test_seal_publication.py | 39 +- tests/test_unreadable_note_refuses.py | 305 +++++++++++++ 16 files changed, 2836 insertions(+), 92 deletions(-) create mode 100644 tests/test_claim_supersession.py create mode 100644 tests/test_conjunction_claim.py create mode 100644 tests/test_unreadable_note_refuses.py diff --git a/README.md b/README.md index 5e19445..945afa6 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,8 @@ didrun verify --html evidence.html The report is a single file with zero external assets — it opens offline and prints cleanly. Redaction covers the sealed note: every exported claim string — -the `argv_preview`, the label, the pathspecs and the changed paths — is +the `argv_preview`, the label, the pathspecs, the changed paths, the conjuncts +and the reason — is redacted, and the projection is declared in the manifest. The HTML report carries no argv at all and renders the note's redacted label, but a stale claim's file list is recomputed against your working tree at verify time and is @@ -146,7 +147,7 @@ correct or that the command meaningfully tested anything. ## The five commands - `didrun run -- ` — record a wrapped execution (complete capture). -- `didrun claim [--path ]` — declare a structured claim. +- `didrun claim [--path ]` — declare a structured claim. `didrun claim conjunction --label REL --of a,b,c` declares one over other claims in the same seal window: it is graded as the worst of them and can never be better than any one of them. - `didrun seal` — compile a commit-bound evidence manifest (redacts secrets; refuses when a structured secret is found in what it is about to publish, warns about the rest). - `didrun verify [--strict] [--require-env-match] [--html ]` — check claims against recorded evidence. - `didrun show [--session]` — show the verdict, or the recorded session history. diff --git a/docs/COMPAT.md b/docs/COMPAT.md index d4769ff..5614228 100644 --- a/docs/COMPAT.md +++ b/docs/COMPAT.md @@ -77,6 +77,55 @@ The evidence block lives in the manifest, never in an `Event`. Putting it into t preimage would invalidate every `entry_hash` ever written — the break the frozen preimage above exists to prevent. +### Version 3 — the claim-type vocabulary gained `conjunction` + +**A v3 note may contain a claim whose `ctype` is `conjunction`, and reading it in a binary +whose claim vocabulary does not include that type is not a graded refusal but an uncaught +traceback: `verify` calls `Claim.from_dict` outside the `try` that grades a claim, so +`ClaimError: unknown claim type: 'conjunction'` propagates out of the command.** That is +the vocabulary break, stated at full strength — "raises" understates it, and the +`MANIFEST_VERSION` bump to 3 exists so a v0.1 or early-v0.2 reader refuses the note on its +version *before* it reaches the claim it cannot parse. + +There is no way to extend a closed vocabulary without that break. The alternative — +skipping claim types a reader does not know — is worse: the note would silently +under-report, and print a verdict over the claims it happened to understand. + +A `conjunction` claim names other claims in the same seal window **by label** and is graded +as the **worst** of them under the same severity order the whole-report verdict uses, so it +can never be better than any one of its members. It binds no event (`event_indices` is +required to be empty) and carries no `evidence` block; `verify` grades it in a second pass, +against the results of the other claims **in that same note** — never against the live +`claims.jsonl`, which would let a claim declared after the seal change an old commit's +verdict. A conjunct label that names no live claim in the window, or more than one, grades +the conjunction `unknown` and the reason names it. + +Two additive keys carry it, both read with `.get()`: + +```json +"claim": {"ctype": "conjunction", "label": "REL", "event_indices": [], + "conjuncts": ["unit", "lint"]}, +"conjunct_grades": [{"label": "unit", "grade": "tree-exact"}, + {"label": "lint", "grade": "stale"}] +``` + +`claim.conjuncts` is emitted for every claim (as `[]` for the ones that name none, like +`pathspecs`); `conjunct_grades` appears only on a conjunction, and a member's `grade` is +`null` when the name resolved to nothing — an unresolved name has no grade, and giving it +one would be an invented fact. + +**What it does NOT do.** It does not reconstitute an execution witness. A design review +declined a proposal to replace recorded executions with a transition marker, on the ground +that a marker witnesses *that a transition occurred*, not *that the command ran and exited +0* — that refusal stands and nothing here attempts it. A conjunction substitutes for +nothing: every conjunct keeps its own witnessed exit code and its own tree comparison, and +the conjunction asserts nothing its members do not already back. + +Stored evidence is unaffected. Archived notes contain no conjunction and stay `version: 1`; +`conjuncts` is additive with a `()` default, so every archived note reads exactly as it did. +The break is forward-only, and it is the same shape as v2's: **upgrade the verifier before +the sealer.** + When didrun falls back to resolving a manifest by tree id it scans the notes ref. A note it cannot parse at all — a foreign note under the same ref, a corrupt body — is skipped and counted, and the count is reported, so one bad note cannot hide every good one. A @@ -145,8 +194,13 @@ A v0.2 `redaction` block carries a second, additive list beside `applied`: `applied` is unchanged and stays argv-addressed by `argv_index`; `fields` is addressed by field name, because a label has no argv index. Field names in v0.2 are `claim.label`, -`claim.pathspecs[i]` and `delta[i].path`. A reader that does not know the key sees -exactly the argv projection it always did. +`claim.pathspecs[i]`, `delta[i].path`, `claim.conjuncts[i]` and `reason`. A reader that +does not know the key sees exactly the argv projection it always did. + +`claim.conjuncts[i]` is scanned as a **label**, because that is what a conjunct name is — +the label of another claim in the same window. Both sides of the match are therefore +redacted by the same deterministic projection, which is what keeps a conjunction resolvable +at verify time even when a conjunct's label carried a secret. This exists because blocking and redacting were not the same set. A structured secret in a claim label refused the seal — and the refusal says *"re-run with `--allow-secrets` to @@ -189,6 +243,48 @@ confined to the local output blobs, and every `notice`-tier finding, warn on std do not stop the seal. So some previously-refused seals now pass and some previously passing ones now refuse; both directions are measured in MEASUREMENTS.md. +## The claim window is recorded, and a narrowing re-seal is refused + +A v0.2 manifest carries two additive top-level integers, `claims_from` and `claims_to` — +the half-open range of the session's declared claims this note covers. They are read with +`.get()` and defaults of `0` and `len(claims)`, so no stored note changes meaning and +`MANIFEST_VERSION` does not bump. The compat harness's per-version allowlist carries both +for v1 and v2: the second deliberate one-line diff that criterion has forced. + +A claim entry may also carry an additive `superseded_by` — the index, within the same +note's claim list, of a later claim declaring the same `(ctype, label)`. `verify` excludes +a superseded entry from the verdict and from `worst_status`, and reports the count. The +entry keeps its grade, its reason and its delta verbatim: supersession subtracts from the +verdict, never from the record. A mark that is not a forward index into the same note's +claims carrying the same `(ctype, label)` is dropped, which *counts* the claim. That is +fail-closed reading, not a forgery barrier — whoever can rewrite a note can rewrite a +grade directly. + +**What this changes for evidence already in the world.** + +- **Reading is unaffected.** Both fields are additive with reading defaults, so every + archived note verifies exactly as it did, down to the reason string. An absent window is + *unknown*, not `[0, N)`: the refusal below reads key presence off the raw body rather + than a parsed default, precisely so a legacy note is never treated as containable. +- **Re-sealing an already-sealed commit now requires `--reseal`** unless the new window + contains the recorded one. This is a deliberate behaviour break, and it is the data-loss + path being closed: `git notes add -f` force-replaces and a seal is scoped to + `all_claims[watermark:]`, so re-sealing the same commit published a note holding only + the post-watermark claims and silently destroyed everything the earlier note held. A + superset re-seal (the watermark-rotation case) still needs no flag. With `--reseal` the + narrowing happens and is visible in `claims_from` / `claims_to` afterwards. +- **An absent `seals.jsonl` warns.** The watermark lives in a local, gitignored, unhashed + file, and its *absence* is not visible from the note: delete it and every later seal + silently re-covers the whole claim history. Measured where it matters most — only 62 of + 167 archived ledgers carried a `seals.jsonl` at all. The seal proceeds (rotation is a + legitimate workflow, and a session's first seal is indistinguishable from a rotated one + from inside `seal`) but it says so on stderr. + +**What it does not do.** Claiming a gate repeatedly until one attempt goes green is not +prevented. Every attempt is retained in the note and the retired count is printed on both +surfaces, so it is visible — which is the honest bound, and strictly better than the +invisible truncation it replaces. + ## The environment fingerprint is versioned in band `env_fingerprint()` returns `v2:<16 hex gate>:path=<8 hex>`. The prefix exists because the @@ -370,12 +466,84 @@ stops, not a plan. is a `notice`-tier detector either way. Nothing in the rule resists an attacker who knows it — the rule is published here, and satisfying it deliberately is trivial. -3. **`reason` and `coverage` are exported unredacted.** Both are didrun's own generated - text — grade names, counts, path *counts* — and carry no operator or repository - string today. That is a property of the current generators, not an enforced - invariant: a future `reason` that interpolates a path or a label would leave the - redacted set without anything failing. The whole-blob manifest scan still *blocks* on - such a field, so the failure mode would be a blocked seal, not a silent leak. +3. **`coverage` is exported unredacted.** It is didrun's own generated text — grade names + and counts — and carries no operator or repository string today. That is a property of + the current generator, not an enforced invariant. The whole-blob manifest scan still + *blocks* on such a field, so the failure mode would be a blocked seal, not a silent leak. + + **`reason` was in this bullet until v3, and v3 is the change that took it out.** The + note in the previous edition said a future `reason` interpolating a path or a label would + leave the redacted set with nothing failing — and a conjunction's reason names a conjunct, + which is operator-authored. So `reason` is now redacted for every claim by the same pass + that reports it to the export gate, under the field name `reason`, rather than only for + the claim type that made it necessary: a field a later version interpolates something + into is then covered without anyone remembering. For every other claim type the reason + is generated text with nothing to find, so no stored or newly published note changes. + +## An unreadable note is a graded refusal + +`verify` refuses a note body it cannot parse, with exit 2 and a reason naming what is +wrong, and it does that for every malformed shape rather than for the ones someone +remembered. The refusal type is `ManifestFormatError`, a subclass of `ManifestError`. + +This was measured broken. Over 22 malformed note shapes, the pre-fix reader left an +uncaught `KeyError` / `TypeError` / `JSONDecodeError` / `ClaimError` traceback on 11 of +them and exited **1** — which is `--strict`'s "this note graded badly" code, so a corrupt +note was indistinguishable from an honest failure by exit status. Worse, it **accepted** +four: `{"version": true}` verified green at exit 0, because `bool` is an `int` subclass and +`True > 3` is False. `2.5`, `0` and `-1` did the same. + +The accepted set is why this matters more than tidiness. The "Version 3" section above +rests the whole forward-compatibility story on the version guard — the `MANIFEST_VERSION` +bump exists so an older reader refuses a v3 note *on its version* before it reaches the +claim vocabulary it cannot parse. A guard a malformed field walks past is not a guard. +`version` is now read with the same rule `superseded_by` gets (`_is_index`, which rejects +`bool`) and must be an integer >= 1. + +**The two refusals stay distinguishable, and the distinction is load-bearing.** A +malformed body is a `ManifestFormatError`; an integer version above `MANIFEST_VERSION` +stays a plain `ManifestError` with the `upgrade didrun` message. On the tree-fallback scan a +format error is **skipped and counted** (a foreign note under the same ref must not hide +every good one) while a too-new version **propagates** (scanning past it would report some +older note's verdict as if it were current). Before, that split was made by accident — the +scan caught bare `Exception`, so a malformed body was skipped only because `json` happened +to raise something that was not a `ManifestError`. + +`verify` also no longer raises out of the command on a claim type it cannot read. The +"Version 3" section notes that `Claim.from_dict` is called outside the `try` that grades a +claim; that hole is now closed from this end too, as a whole-manifest refusal naming the +entry. A per-claim `unknown` was rejected for the reason that section already gives: it +would print a verdict over the claims this binary happened to understand. + +**This is fail-closed reading, not a forgery barrier**, and it is the same caveat +`superseded_by` carries. Whoever can rewrite a note can rewrite a grade directly +(docs/TRUST_MODEL.md). Nothing above resists an attacker; it stops a corrupt or foreign +note from being read as a verdict. + +Stored evidence is unaffected: every note ever published carries an integer version of 1, 2 +or 3 and the required fields, so nothing legitimately published is newly refused. The +compat corpus replays all three versions. + +## Known, not closed + +Small things measured and left alone, recorded rather than dropped. None of them can +produce a green verdict that the evidence does not back. + +- **A duplicate conjunct label is counted twice.** `--of a,a` grades correctly (the worst + of the two resolutions of `a`, which is `a`'s grade) but the reason reads `worst of 2 + conjunct(s)` for one distinct member, and `conjunct_grades` carries the pair twice. + Deduplicating would silently rewrite what the operator declared, which is worse than a + cosmetically odd count; refusing a duplicate is the better fix and is not built. +- **`didrun claim conjunction` in a session with no recorded events** refuses with "no + recorded events to bind to (run something first)". A conjunction binds no event, so the + wording is not its refusal — but a session with no events has no claims either, so a + conjunction there could only ever grade `unknown`, and the early refusal is correct even + though the sentence is borrowed. +- **A conjunction's `reason` names a conjunct, and the reason is redacted as one string.** + Both the reason and the conjunct label go through the same deterministic projection, so a + secret in a label is redacted in both places; but the redacted reason is not guaranteed to + contain the redacted label as a substring, because the two are scanned as separate fields. + Resolution never reads the reason, so this affects display only. ## Two things v0.2 does not close diff --git a/src/didrun/claims.py b/src/didrun/claims.py index c7ed628..3d455f1 100644 --- a/src/didrun/claims.py +++ b/src/didrun/claims.py @@ -15,6 +15,24 @@ instrumentation in disguise with no deterministic v1 implementation; it is killed by design (coverage belongs in an adapter, not the core). Registering it raises with the flip condition. + +``conjunction`` is the one claim type that grades no evidence of its own. It +names other claims in the same seal window by label and is graded as the WORST +of them, so it can never be better than any one of its members. The property it +represents — "all N of these gates passed against this tree" — had no +representation in the tool and was being enforced by hand. + +**The distinction this type is on the right side of, stated because the next +reader will conflate the two.** A design review of an 80-command proof ledger +declined a proposal to replace some of those recorded executions with a +transition marker, and the reason is quoted here rather than softened: a marker +witnesses *that a transition occurred*, not *that the command ran and exited +0*, so substituting one for the other genuinely weakens the claim. That refusal +stands, permanently, and nothing here attempts it — a conjunction substitutes +for nothing. Every conjunct keeps its own witnessed exit code and its own tree +comparison, graded by the same ladder as any other claim, and the conjunction +asserts nothing its members do not already back. It conjoins execution witnesses +that exist; it never reconstitutes one that does not. """ from __future__ import annotations @@ -26,9 +44,16 @@ from .ledger import Event, Session from . import gitplumbing -# The v1 claim vocabulary. `tests-pass` and `lint-clean` are labels over one +# The claim vocabulary. `tests-pass` and `lint-clean` are labels over one # generic engine: "a declared command exited 0". No claim type infers meaning. -CLAIM_TYPES = ("command-succeeded", "tests-pass", "lint-clean") +# +# `conjunction` is the one entry that binds no event of its own: it names other +# claims in the same seal window and is graded as the worst of them. Extending +# this tuple is a MANIFEST_VERSION bump (docs/COMPAT.md) — an older reader hits +# `unknown claim type` on a note that carries a type it has never seen. +CONJUNCTION = "conjunction" + +CLAIM_TYPES = ("command-succeeded", "tests-pass", "lint-clean", CONJUNCTION) KILLED_CLAIM_TYPES = { "diff-exercised": ( @@ -69,6 +94,45 @@ ENV_INCOMPARABLE = "incomparable" ENV_NOT_RECORDED = "not-recorded" +# Worst first. This is the ONE severity order in the package: the whole-report +# verdict (manifest.VerifyReport.worst_status) reads it, and a conjunction reads +# it to grade as the worst of its conjuncts. Two copies could disagree about +# which grade is worse, and a conjunction that disagreed with the verdict line +# above it would be the exact overclaim this type has to be incapable of. +# +# `chain-broken` is deliberately absent: it is a statement about the ledger every +# grade was read out of, not a grade any claim can carry, and it dominates the +# verdict before this order is consulted. +GRADE_ORDER_WORST_FIRST = ( + GRADE_FAILED, + GRADE_UNKNOWN, + GRADE_WITNESS_UNAVAILABLE, + GRADE_STALE, + GRADE_SCOPE_EXACT, + GRADE_TREE_EXACT, +) + + +def worst_grade(grades) -> Optional[str]: + """The worst grade in ``grades`` under ``GRADE_ORDER_WORST_FIRST``. + + Ties go to the first occurrence, so the caller's order decides which of two + equally-bad members is named — never the iteration order of a set. + + A grade this version does not know ranks WORSE than every registered one. A + grade nobody here can interpret is not evidence that anything is fine, and + ranking it best is how an unregistered token would carry a conjunction to + green. Returns None for an empty input; callers must not treat that as a + grade. + """ + worst: Optional[str] = None + worst_rank: Optional[int] = None + for g in grades: + rank = GRADE_ORDER_WORST_FIRST.index(g) if g in GRADE_ORDER_WORST_FIRST else -1 + if worst_rank is None or rank < worst_rank: + worst, worst_rank = g, rank + return worst + class ClaimError(Exception): """A claim declaration violated an invariant.""" @@ -81,6 +145,13 @@ class Claim: ``pathspecs`` are supplied by the claimant for a scope-exact grade; they are never auto-derived. ``declared_at_index`` is the session index at which the claim was declared — used for the retroactive-binding rule. + + ``conjuncts`` are the LABELS of other claims in the same seal window that a + ``conjunction`` conjoins. They are labels rather than event indices on + purpose: a conjunction must not be a claim bound to several events. That + shape already exists, grades on the first bound success it finds, and is the + illegitimate composite the design review ruled out — so ``event_indices`` is + required to be EMPTY here and its meaning is untouched. """ ctype: str @@ -88,12 +159,30 @@ class Claim: event_indices: tuple[int, ...] pathspecs: tuple[str, ...] = () declared_at_index: int = -1 + conjuncts: tuple[str, ...] = () def __post_init__(self) -> None: if self.ctype in KILLED_CLAIM_TYPES: raise ClaimError(KILLED_CLAIM_TYPES[self.ctype]) if self.ctype not in CLAIM_TYPES: raise ClaimError(f"unknown claim type: {self.ctype!r}") + if self.ctype == CONJUNCTION: + if not self.conjuncts: + raise ClaimError( + "a conjunction claim must name the claims it conjoins " + "(at least one conjunct label)" + ) + if self.event_indices: + raise ClaimError( + "a conjunction claim binds no event: it conjoins claims that " + "carry their own witnessed events, and widening its bound " + "events would make it a multi-index claim instead" + ) + elif self.conjuncts: + raise ClaimError( + f"claim type {self.ctype!r} names no conjuncts; only a " + f"conjunction claim conjoins other claims" + ) def to_dict(self) -> dict: return { @@ -102,6 +191,11 @@ def to_dict(self) -> dict: "event_indices": list(self.event_indices), "pathspecs": list(self.pathspecs), "declared_at_index": self.declared_at_index, + # Emitted always, like `pathspecs`, and read with a default below: + # an older reader that does not know the key sees the claim it + # always saw, and every claim this version writes declares the key + # whether or not it carries one. + "conjuncts": list(self.conjuncts), } @classmethod @@ -112,6 +206,7 @@ def from_dict(cls, d: dict) -> "Claim": event_indices=tuple(d.get("event_indices", ())), pathspecs=tuple(d.get("pathspecs", ())), declared_at_index=d.get("declared_at_index", -1), + conjuncts=tuple(d.get("conjuncts", ())), ) @@ -128,6 +223,21 @@ class GradeResult: are deliberately NOT in ``to_dict``: the seal has nothing to compare against itself, so writing them would put a permanently-default key in every published note. + + ``superseded_by`` is verify-side for a different reason: supersession is a + fact about the seal WINDOW, not about this claim's evidence, so the seal + writes the mark onto the exported entry itself (manifest._redact_result) and + verify reads it back onto the result. Keeping it out of ``to_dict`` is what + makes the mark appear only on the entries that carry one, instead of putting + a permanently-null key on every claim in every note. + + ``conjunct_grades`` holds one ``(label, grade)`` pair per conjunct a + conjunction declared, in declaration order, so the note carries what each + member earned instead of only the worst of them. ``grade`` is None for a + conjunct that resolved to no single live claim: an unresolved name has no + grade, and giving it one would be the invented fact this whole type exists + not to produce. Serialized only when non-empty, on ``superseded_by``'s + precedent — every other claim would otherwise carry an empty list forever. """ claim: Claim @@ -140,9 +250,21 @@ class GradeResult: sealed_grade: Optional[str] = None env_status: str = ENV_NOT_RECORDED env_reason: str = "" + superseded_by: Optional[int] = None + conjunct_grades: tuple = () + + @property + def is_superseded(self) -> bool: + """Whether a later claim in the same seal window retired this one. + + A superseded result is RECORD, not verdict: its grade and reason are + kept verbatim and stay visible, and it is excluded from the verdict — + never relabelled, re-graded or dropped. + """ + return self.superseded_by is not None def to_dict(self) -> dict: - return { + d = { "claim": self.claim.to_dict(), "grade": self.grade, "reason": self.reason, @@ -152,6 +274,11 @@ def to_dict(self) -> dict: "evidence_bound": self.evidence_bound, "sealed_grade": self.sealed_grade, } + if self.conjunct_grades: + d["conjunct_grades"] = [ + {"label": label, "grade": g} for label, g in self.conjunct_grades + ] + return d def _supporting_event(claim: Claim, events: list[Event]) -> Optional[tuple[int, Event]]: @@ -184,18 +311,87 @@ def _witnessed_failure(claim: Claim, events: list[Event]) -> Optional[tuple[int, return None +def grade_conjunction(claim: Claim, siblings: Optional[list] = None) -> GradeResult: + """Grade a conjunction as the WORST of the claims it names. Never better. + + ``siblings`` are the other graded results of the same seal window. Two kinds + are filtered out here rather than at the call site, so both callers (seal and + verify) get the same resolution from the same rule: + + - **superseded entries.** A repair re-declares the same gate, so the retired + attempt shares its label with the claim that stands in its place. Skipping + it is what lets a conjunction name a gate that failed and was fixed inside + the window; counting it would make every such name ambiguous. + - **other conjunctions.** A conjunction naming a conjunction resolves to no + match and grades ``unknown``. That is deliberate: it keeps resolution one + level deep, so there is no recursion to bound and no cycle to detect, and + the grade of a name this function will not follow is not asserted. + + Resolution is by EXACT label match — never a prefix, never a fuzzy one. A + name matching no live claim, or more than one, makes the whole conjunction + ``unknown`` and the reason names it: a member it cannot identify is a member + it cannot vouch for, and grading around the gap is how a conjunction would + come out better than its parts. + """ + pool = [ + s + for s in (siblings or ()) + if not s.is_superseded and s.claim.ctype != CONJUNCTION + ] + pairs: list = [] + unresolved: list[str] = [] + for name in claim.conjuncts: + matches = [s for s in pool if s.claim.label == name] + if len(matches) == 1: + pairs.append((name, matches[0].grade)) + continue + pairs.append((name, None)) + unresolved.append( + f"{name!r} names no claim in this seal window" + if not matches + else f"{name!r} names {len(matches)} claims in this seal window" + ) + if unresolved: + return GradeResult( + claim, + GRADE_UNKNOWN, + reason="unresolved conjunct: " + "; ".join(unresolved), + conjunct_grades=tuple(pairs), + ) + worst = worst_grade(g for _name, g in pairs) + # Ties go to the first declared conjunct, which is what makes this reason + # reproducible for the same window rather than a function of dict ordering. + named = next(name for name, g in pairs if g == worst) + return GradeResult( + claim, + worst, + reason=f"worst of {len(pairs)} conjunct(s): {named!r} graded {worst}", + conjunct_grades=tuple(pairs), + ) + + def grade( claim: Claim, sealed_tree: str, events: list[Event], repo: Path, ledger_objects: Optional[Path] = None, + siblings: Optional[list] = None, ) -> GradeResult: """Grade one claim against ``sealed_tree``. First match wins. The ordering is the honesty ladder: only concede a weaker grade when the stronger one cannot be honestly asserted. + + ``siblings`` is consulted for a ``conjunction`` and for nothing else: it + grades no evidence of its own, so it is dispatched BEFORE the ladder rather + than added as a rung. Every rung below is untouched, and a caller that + forgets ``siblings`` gets a conjunction graded ``unknown`` — the fail-closed + direction, never a green one. """ + if claim.ctype == CONJUNCTION: + return grade_conjunction(claim, siblings) + # A witnessed failure among the bound events DOMINATES a witnessed success. # Consulting _supporting_event first graded a claim bound to # [failed, success] on the success and made the caught lie invisible. diff --git a/src/didrun/cli.py b/src/didrun/cli.py index e419de5..4aa4124 100644 --- a/src/didrun/cli.py +++ b/src/didrun/cli.py @@ -23,7 +23,7 @@ run_wrapped, signal_name, ) -from .claims import Claim +from .claims import CLAIM_TYPES, CONJUNCTION, Claim from . import manifest as _manifest from . import redact from . import render @@ -71,6 +71,19 @@ def cmd_run(args) -> int: return event.exit_code if event.exit_code is not None else 0 +def _conjunct_labels(raw: Optional[str]) -> tuple: + """Parse ``--of``: the comma-separated labels of the claims to conjoin. + + Order is the operator's, and it is kept: it decides which of two + equally-bad conjuncts the grade's reason names. Surrounding whitespace is + stripped and an empty name is dropped rather than matched against a label + nothing carries; an empty result reaches ``Claim``, which refuses it. + """ + if not raw: + return () + return tuple(part.strip() for part in raw.split(",") if part.strip()) + + def cmd_claim(args) -> int: repo = Path(args.repo or os.getcwd()) session = _session(repo) @@ -78,16 +91,46 @@ def cmd_claim(args) -> int: if not events: print("didrun claim: no recorded events to bind to (run something first)", file=sys.stderr) return 2 - # Default: bind to the most recent successful event. - if args.event is not None: - indices = (args.event,) + conjuncts: tuple = () + indices: tuple = () + if args.type == CONJUNCTION: + # A conjunction binds no event — it conjoins claims that carry their own + # witnessed events — so the "no successful event to bind" refusal below + # is not its refusal. `--event` and `--path` are refused rather than + # silently ignored: an operator who passed one meant something by it. + if args.event is not None: + print( + "didrun claim: a conjunction binds no event; name the claims it " + "conjoins with --of instead of --event", + file=sys.stderr, + ) + return 2 + if args.path: + print( + "didrun claim: a conjunction declares no pathspec; its conjuncts " + "carry their own", + file=sys.stderr, + ) + return 2 + conjuncts = _conjunct_labels(args.of) else: - indices = tuple( - i for i, ev in enumerate(events) if ev.exit_code == 0 - )[-1:] # last success - if not indices: - print("didrun claim: no successful event to bind (last command did not exit 0)", file=sys.stderr) - return 2 + if args.of: + print( + f"didrun claim: --of names the claims a conjunction conjoins; " + f"{args.type} conjoins none", + file=sys.stderr, + ) + return 2 + # Default: bind to the most recent successful event. + if args.event is not None: + indices = (args.event,) + else: + indices = tuple( + i for i, ev in enumerate(events) if ev.exit_code == 0 + )[-1:] # last success + if not indices: + print("didrun claim: no successful event to bind (last command did not exit 0)", file=sys.stderr) + return 2 try: claim = Claim( ctype=args.type, @@ -95,12 +138,20 @@ def cmd_claim(args) -> int: event_indices=indices, pathspecs=tuple(args.path or ()), declared_at_index=len(events) - 1, + conjuncts=conjuncts, ) except Exception as exc: print(f"didrun claim: {exc}", file=sys.stderr) return 2 _manifest.declare_claim(session, claim) - print(f"declared claim {claim.ctype} ({claim.label}) bound to event(s) {list(indices)}") + if claim.conjuncts: + print( + f"declared claim {claim.ctype} ({claim.label}) over " + f"{len(claim.conjuncts)} conjunct(s): {', '.join(claim.conjuncts)} " + f"— graded as the worst of them" + ) + else: + print(f"declared claim {claim.ctype} ({claim.label}) bound to event(s) {list(indices)}") return 0 @@ -114,6 +165,7 @@ def cmd_seal(args) -> int: commitish=args.commit or "HEAD", allow_secrets=args.allow_secrets, bundle_path=Path(args.bundle) if args.bundle else None, + reseal=args.reseal, ) except _manifest.redact.SecretsBlocked as exc: print(f"didrun seal: {exc}", file=sys.stderr) @@ -121,7 +173,13 @@ def cmd_seal(args) -> int: except _manifest.ManifestError as exc: print(f"didrun seal: {exc}", file=sys.stderr) return 2 - verified = sum(1 for c in m.claims if c["grade"] in ("tree-exact", "scope-exact")) + # Counted over the live entries, like the verdict surface: a window whose + # earlier attempt was retired would otherwise print "1/2 claims + # recorded-exact" for a unit `verify` calls ALL RECORDED-EXACT, and the two + # numbers describing the same note have to describe the same set. + live = _manifest.live_claims(m.claims) + superseded = len(m.claims) - len(live) + verified = sum(1 for c in live if c["grade"] in ("tree-exact", "scope-exact")) # The tier split is reported even when nothing blocked: "0 findings" and # "12 findings, none of them blocking" are different facts about the same # green seal, and only one of them is worth a second look. @@ -130,8 +188,9 @@ def cmd_seal(args) -> int: noticed = by_tier.get("notice", 0) print( f"sealed manifest for {m.commit[:12]} (tree {m.tree[:12]}) " - f"{verified}/{len(m.claims)} claims recorded-exact " - f"{blocking + noticed} findings ({blocking} block / {noticed} notice)" + f"{verified}/{len(live)} claims recorded-exact " + + (f"{superseded} superseded " if superseded else "") + + f"{blocking + noticed} findings ({blocking} block / {noticed} notice)" + (" [--allow-secrets]" if m.secrets_override else "") ) return 0 @@ -298,16 +357,38 @@ def build_parser() -> argparse.ArgumentParser: pr.set_defaults(func=cmd_run) pc = sub.add_parser("claim", help="declare a structured claim over recorded events") - pc.add_argument("type", choices=["command-succeeded", "tests-pass", "lint-clean"]) + # Read from the vocabulary rather than repeated here. The two lists were + # duplicates, and a duplicate of a closed vocabulary drifts silently: the CLI + # would keep refusing a type the library had accepted for a release. + pc.add_argument("type", choices=list(CLAIM_TYPES)) pc.add_argument("--label", help="human label (default: the type)") pc.add_argument("--event", type=int, help="bind to a specific event index") pc.add_argument("--path", action="append", help="declare a pathspec for scope-exact grading") + pc.add_argument( + "--of", + metavar="LABEL[,LABEL…]", + help=( + "for a conjunction: the comma-separated labels of the claims in this " + "seal window it conjoins. The conjunction is graded as the WORST of " + "them and can never be better than any one of them; a label that " + "names no single live claim in the window grades it unknown" + ), + ) pc.set_defaults(func=cmd_claim) ps = sub.add_parser("seal", help="compile and attach a commit-bound manifest") ps.add_argument("--commit", help="commit to bind (default: HEAD)") ps.add_argument("--allow-secrets", action="store_true", help="export despite secret findings (redacted; logged)") ps.add_argument("--bundle", help="also write an exportable bundle to this path") + ps.add_argument( + "--reseal", + action="store_true", + help=( + "permit replacing this commit's note with a NARROWER record: the " + "claims the replaced note held and this seal does not are then in " + "no note at all. Without it, a narrowing re-seal is refused" + ), + ) ps.set_defaults(func=cmd_seal) pv = sub.add_parser("verify", help="verify a commit's claims against recorded evidence") diff --git a/src/didrun/manifest.py b/src/didrun/manifest.py index ca411da..50346e5 100644 --- a/src/didrun/manifest.py +++ b/src/didrun/manifest.py @@ -34,12 +34,15 @@ from .ledger import Session, canonical_json, _append_lock, _open_private_append from .claims import ( + CONJUNCTION, ENV_DRIFT, ENV_INCOMPARABLE, ENV_MATCH, ENV_NOT_RECORDED, + GRADE_ORDER_WORST_FIRST, GRADE_WITNESS_UNAVAILABLE, Claim, + ClaimError, GradeResult, grade, ) @@ -50,7 +53,15 @@ # bump is for the change of meaning, not for the new key: an older reader parses # a v2 note fine (every key is additive) and regrades it index-only, exactly as # it does today. See docs/COMPAT.md. -MANIFEST_VERSION = 2 +# +# 3: the claim-type vocabulary gained `conjunction`. This one is NOT additive in +# either direction — `Claim.from_dict` raises `unknown claim type` on a type it +# does not know, and in an older binary it raises from OUTSIDE verify's try, so a +# v0.1/v0.2 reader meeting a v3 note exits with an uncaught traceback rather than +# a graded refusal. There is no way to extend a closed vocabulary without that, +# and silently skipping claim types a reader does not know would be worse: the +# note would under-report and still print a verdict. See docs/COMPAT.md. +MANIFEST_VERSION = 3 NOTES_REF = "refs/notes/didrun" @@ -58,6 +69,26 @@ class ManifestError(Exception): pass +class ManifestFormatError(ManifestError): + """A note body this binary cannot parse at all. + + Split from ``ManifestError`` because the two get different treatment on the + tree-fallback scan: a body that is not a readable manifest is SKIPPED and + counted (a foreign note under the same ref must not hide every good one), + while a version this binary does not understand PROPAGATES. Both are a + graded refusal on the direct-commit path — it subclasses ``ManifestError``, + which is what the CLI already catches, so an unreadable note exits 2 rather + than raising out of the command. + + That distinction used to be made by exception type ACCIDENTALLY: the scan + caught bare ``Exception``, so a malformed body was skipped because + ``json`` happened to raise something that was not a ``ManifestError``. On + the direct-commit path the same shapes had nothing to catch them and left as + an uncaught ``TypeError``/``KeyError`` traceback, with exit 1 — which is + ``--strict``'s "this graded badly" code, not "I could not read this". + """ + + @dataclass class Manifest: version: int @@ -71,6 +102,24 @@ class Manifest: # "almost nothing was scanned". This says which. Additive, defaults to {}, # read with .get(): no MANIFEST_VERSION bump (docs/COMPAT.md). secrets: dict = field(default_factory=dict) + # The half-open claim window this note covers: the seal watermark and the + # claim count at seal time. Without it the note's SCOPE was invisible from + # the note — a re-seal narrowed to `all_claims[watermark:]` and force-wrote + # over the wider record, and nothing in the artifact said so. Additive, + # read with .get() defaults of 0 / len(claims): no MANIFEST_VERSION bump + # (docs/COMPAT.md). + claims_from: int = 0 + claims_to: Optional[int] = None + + def __post_init__(self) -> None: + # The reading default for `claims_to`, applied once so every consumer + # sees an int. `None` is the "key was absent" marker, and it must NOT + # survive into `to_json` — a note with a null window would be a third + # state nobody reads. What "absent" MEANS (the window is unknown, not + # [0, N)) is answered by `_recorded_window`, which reads key presence + # off the raw body instead of a parsed default. + if self.claims_to is None: + self.claims_to = len(self.claims) def to_json(self) -> bytes: return canonical_json( @@ -82,6 +131,8 @@ def to_json(self) -> bytes: "coverage": self.coverage, "secrets_override": self.secrets_override, "secrets": self.secrets, + "claims_from": self.claims_from, + "claims_to": self.claims_to, } ) @@ -94,14 +145,59 @@ def from_json(cls, data: bytes) -> "Manifest": would produce a confident verdict about a format this binary has never seen. Older versions are accepted — new fields are additive and are read with a v1-reproducing default (see docs/COMPAT.md). + + Every shape that is not a readable manifest raises + ``ManifestFormatError`` rather than whatever ``json`` and ``dict`` + happened to raise on the way past. The version comparison is the reason + this matters: ``version`` arrives from a file, so it is not necessarily + an int, and ``"3" > 3`` raises while ``True > 3`` is quietly False. The + first left a traceback where a refusal belonged; the second ACCEPTED the + note and verified it green, because ``bool`` is an ``int`` subclass and + ``True`` compares as 1. A guard the whole forward-compat story rests on + (docs/COMPAT.md, "Version 3") cannot be one a malformed field walks past. """ - d = json.loads(data) + try: + d = json.loads(data) + except ValueError as exc: + raise ManifestFormatError(f"note body is not valid JSON: {exc}") from exc + if not isinstance(d, dict): + raise ManifestFormatError( + f"note body is a JSON {type(d).__name__}, not a manifest object" + ) + if "version" not in d: + raise ManifestFormatError("note body declares no manifest version") version = d["version"] + # `_is_index` and not `isinstance(..., int)`: bool is an int subclass, so + # a `version` of `true` would otherwise compare as 1 and be accepted as a + # v1 note. Measured before the fix: `didrun verify --strict` exited 0 on + # it. The same reading rule the supersession mark gets, for the same + # reason — a field that arrived from a file is not a field this process + # computed. + if not _is_index(version) or version < 1: + raise ManifestFormatError( + f"manifest version is {version!r}, which is not a version number " + f"(expected an integer >= 1)" + ) if version > MANIFEST_VERSION: raise ManifestError( f"manifest version {version} is newer than this didrun understands " f"(max {MANIFEST_VERSION}); upgrade didrun" ) + # The keys with no reading default. A note missing one is not a note an + # older-or-newer default can stand in for, and indexing straight into + # `d` turned each one into a `KeyError` traceback out of the command. + missing = [k for k in ("commit", "tree", "claims", "coverage") if k not in d] + if missing: + raise ManifestFormatError( + f"manifest is missing required field(s): {', '.join(missing)}" + ) + # `claims` is indexed and iterated by every consumer, and `__post_init__` + # takes its len. A non-list here surfaced as `TypeError: string indices + # must be integers` from deep inside verify. + if not isinstance(d["claims"], list): + raise ManifestFormatError( + f"manifest `claims` is a {type(d['claims']).__name__}, not a list" + ) return cls( version=version, commit=d["commit"], @@ -110,6 +206,8 @@ def from_json(cls, data: bytes) -> "Manifest": coverage=d["coverage"], secrets_override=d.get("secrets_override", False), secrets=d.get("secrets", {}), + claims_from=d.get("claims_from", 0), + claims_to=d.get("claims_to"), ) @@ -145,6 +243,70 @@ def _gc_durable_copy(session: Session, results: list[GradeResult]) -> None: raise ManifestError(f"claim-referenced blob missing at seal: {digest}") +def _supersession(claims: list[Claim]) -> dict: + """Which claims in this window a later one retired, as {index: live index}. + + A repair re-declares the same gate: same ``ctype``, same ``label``. That + pair is the identity, so a later claim carrying it supersedes every earlier + one — which is what lets a fix retire a `failed` or `stale` claim INSIDE the + window, with no intervening commit. Before this, the only two escapes were + seal-early (needs a commit) and re-seal (which truncated the record), and + discarding the ledger was the rational third. + + **Claim-preserving by construction.** Nothing here relabels, deletes or + re-grades anything: every claim in the window is graded and every graded + result is published. The earlier grade and reason stay in the note verbatim, + marked with the index of the entry that retired it. Supersession subtracts + from the VERDICT, never from the RECORD. + + The value is the index of the LIVE entry for that key — the last one in the + window — not the immediate successor. A reader asking "which claim stands in + place of this one" gets that answer directly instead of walking a chain, and + three attempts at the same gate all point at the one that stands. + + Window-local by construction: the caller passes only this seal's claims, so + a claim in an earlier window (already sealed into an earlier commit's note) + can never be retired from here. History is not rewritten. + """ + live: dict = {} + for index, claim in enumerate(claims): + live[(claim.ctype, claim.label)] = index + return { + index: live[(claim.ctype, claim.label)] + for index, claim in enumerate(claims) + if live[(claim.ctype, claim.label)] != index + } + + +def _warn_missing_watermark(session: Session, claim_count: int) -> None: + """Say, loudly, when the watermark file is absent and claims exist. + + The watermark is a local, gitignored, unhashed file, and its ABSENCE is not + visible from the note: delete it and every later seal silently re-covers the + whole claim history. Measured in the archives this matters most for — only + 62 of 167 archived ledgers carried a seals.jsonl at all. + + This does not block. A first seal legitimately has no watermark file, and so + does a rotated ledger, and nothing available here distinguishes the two — + saying which is impossible, so the warning says BOTH and leaves the operator + to know which one they are in. Blocking would break the first seal of every + session; silence is what produced a note whose scope was a function of a + deleted file. + """ + if _seals_path(session).exists() or claim_count == 0: + return + print( + f"didrun seal: WARNING: no seal watermark file, so the watermark is 0 " + f"and this seal covers the WHOLE claim history " + f"({claim_count} declared claim(s)). That is correct for a session's " + f"first seal. If this session has sealed before, the watermark file was " + f"removed or rotated and this seal re-covers claims an earlier note " + f"already recorded — nothing in the note or the ledger distinguishes the " + f"two from here.", + file=sys.stderr, + ) + + def seal( session: Session, repo: Path, @@ -152,6 +314,7 @@ def seal( allow_secrets: bool = False, write_notes: bool = True, bundle_path: Optional[Path] = None, + reseal: bool = False, ) -> Manifest: """Compile, redact, and attach a manifest for ``commitish``. @@ -162,6 +325,11 @@ def seal( are not published, so refusing over them stopped work without protecting anything. On override, the exported artifact is still redacted and the override is recorded in the manifest. + + Raises ``ManifestError`` when the commit already carries a note whose claim + window this seal would NARROW, unless ``reseal`` is True. That overwrite was + silent and lossy: `git notes add -f` force-replaces, and a seal scoped to + `all_claims[watermark:]` replaced a wide record with a narrow one. """ repo = Path(repo) commit = gitplumbing.head_commit(repo) if commitish == "HEAD" else _rev(repo, commitish) @@ -186,12 +354,34 @@ def seal( # The claims file itself remains append-only; only the seal's view is scoped. all_claims = _load_claims(session) watermark = _last_seal_watermark(session) + _warn_missing_watermark(session, len(all_claims)) new_claims = all_claims[watermark:] - # Grade the claims for THIS seal against the sealed tree. - results: list[GradeResult] = [] - for claim in new_claims: - results.append(grade(claim, tree, events, repo, ledger_objects)) + # Grade the claims for THIS seal against the sealed tree. EVERY claim in the + # window is graded, including the ones a later claim supersedes: the mark + # below excludes a retired claim from the verdict, and grading it anyway is + # what keeps the record honest about what was declared and what it earned. + # + # Two passes, because a conjunction is graded against its SIBLINGS and pass + # one is what produces them. `verify` is restructured the same way, over the + # note's own results — the two must agree or a note that sealed green would + # regrade `unknown`. + results: list = [None] * len(new_claims) + conjunctions: list[int] = [] + for i, claim in enumerate(new_claims): + if claim.ctype == CONJUNCTION: + conjunctions.append(i) + continue + results[i] = grade(claim, tree, events, repo, ledger_objects) + superseded = _supersession(new_claims) + # The retired attempts are dropped from the pool here rather than marked on + # the result: `superseded_by` on a GradeResult is verify-side, and the seal + # holds the authoritative mapping already. + siblings = [r for i, r in enumerate(results) if r is not None and i not in superseded] + for i in conjunctions: + results[i] = grade( + new_claims[i], tree, events, repo, ledger_objects, siblings=siblings + ) _gc_durable_copy(session, results) @@ -205,13 +395,18 @@ def seal( # `secrets_override` and `secrets` — and they are the scan's own counters, # versions and booleans, never operator- or command-authored text. Every # field that carries content from outside is in these bytes, and a field - # added later is in them too, without anyone remembering to add it. + # added later is in them too, without anyone remembering to add it. That is + # why the claim window is here even though it is two integers this function + # computed: the rule is "the published bytes are the scanned bytes", and a + # field exempted because it looked harmless is how that rule stops holding. # # Compiling the export domain also REDACTS it, and the findings that come # back are what the gate blocks on. A field cannot be blocked-but-published # (the label bug) or published-but-unscanned, because one pass does both. redacted = [ - _redact_result(r, session, entries, claim_index=i) + _redact_result( + r, session, entries, claim_index=i, superseded_by=superseded.get(i) + ) for i, r in enumerate(results) ] claims_payload = [rc.payload for rc in redacted] @@ -225,6 +420,8 @@ def seal( "tree": tree, "claims": claims_payload, "coverage": coverage, + "claims_from": watermark, + "claims_to": len(all_claims), } ) scan = _scan_for_secrets( @@ -246,6 +443,8 @@ def seal( coverage=coverage, secrets_override=overridden, secrets=scan.to_dict(overridden), + claims_from=watermark, + claims_to=len(all_claims), ) # Publication and the watermark are one atomic pair. A published note with @@ -255,6 +454,7 @@ def seal( # and if recording fails the note goes back to what it was. prior_note = _read_note(repo, commit) if write_notes else None if write_notes: + _refuse_narrowing_reseal(commit, prior_note, manifest, reseal) _attach_note(repo, commit, manifest) # Everything after publication runs inside the rollback. A caller-supplied @@ -313,6 +513,27 @@ class VerifyReport: def chain_faulted(self) -> bool: return self.chain_status in _CHAIN_FAULTS + @property + def live_results(self) -> list: + """The results the verdict is computed over: everything not superseded. + + A superseded result is record, not verdict — a claim a later claim in the + same seal window retired. It stays in ``results``, keeps its grade and + reason verbatim, and renders; it just does not vote. + """ + return [r for r in self.results if not r.is_superseded] + + @property + def superseded_count(self) -> int: + """How many results are retained as record and excluded from the verdict. + + Printed on both human surfaces. Every retry at the same gate is kept and + counted, so "claim until it goes green" is VISIBLE rather than prevented + — which is the honest bound, and strictly better than the invisible + truncation it replaces. + """ + return len(self.results) - len(self.live_results) + @property def worst_status(self) -> str: """Worst grade present — drives the verdict and the CI exit code.""" @@ -321,17 +542,19 @@ def worst_status(self) -> str: # them is a statement anyone should act on. if self.chain_faulted: return CHAIN_BROKEN - order = [ - "failed", - "unknown", - GRADE_WITNESS_UNAVAILABLE, - "stale", - "scope-exact", - "tree-exact", - ] - present = {r.grade for r in self.results} + # Read from claims.py rather than repeated here: a conjunction grades as + # the worst of its conjuncts under this same order, and two copies of it + # could disagree about which grade is worse. + order = GRADE_ORDER_WORST_FIRST + present = {r.grade for r in self.live_results} if not present: - return "empty" + # "no results at all" and "results, every one of them superseded" + # are different facts and must not share a word. The first is an + # unsealed commit. The second is a window that declared claims and + # retired all of them, which has nothing left to verify — so it + # grades `unknown`, never `empty` (which renders as NO CLAIMS) and + # never anything the strict set accepts. + return "empty" if not self.results else "unknown" for status in order: if status in present: return status @@ -347,11 +570,19 @@ def all_verified(self) -> bool: # --require-env-match. `incomparable` never does, under either mode: it # is the state every archived v1 note lands in, and refusing on it would # turn "this note predates the current fingerprint" into a failure. + # + # Superseded results are excluded — they are record, not verdict — and + # the NON-EMPTINESS check is over the surviving set for that reason. + # `all([])` is True, so filtering without re-checking would make a window + # whose every claim was retired verify vacuously green: a green verdict + # over zero live claims, which is the exact shape of overclaiming this + # tool exists to prevent. if self.chain_faulted: return False - if not self.results: + live = self.live_results + if not live: return False - for r in self.results: + for r in live: if r.grade not in ("tree-exact", "scope-exact"): return False if self.require_env_match and r.env_status == ENV_DRIFT: @@ -365,6 +596,10 @@ def env_counts(self) -> dict: Always all four keys, including zeros: "0 drifted" and "not checked" are different facts, and a counter that disappears when it is zero cannot say the first one. + + Over the live set, like every other header counter: these gloss the + verdict, and a denominator that silently included retired claims would + not add up against the one beside it. """ counts = { ENV_MATCH: 0, @@ -372,7 +607,7 @@ def env_counts(self) -> dict: ENV_INCOMPARABLE: 0, ENV_NOT_RECORDED: 0, } - for r in self.results: + for r in self.live_results: counts[r.env_status] = counts.get(r.env_status, 0) + 1 return counts @@ -387,11 +622,20 @@ def evidence_bound_count(self) -> int: The complement is not a failure: a v1 note carries no binding to check, so its claims are regraded by index exactly as they always were. The counter exists so a reader can tell the two apart at a glance. + + Over the live set, for the same reason as ``env_counts``. """ - return sum(1 for r in self.results if r.evidence_bound) + return sum(1 for r in self.live_results if r.evidence_bound) @property def total(self) -> int: + """Entries in the RECORD, superseded ones included. + + The human surfaces count the live set instead — the verdict's + denominator has to be the set the verdict was computed over — and print + ``superseded_count`` beside it. This stays the record's size because + that is what a caller counting what the note holds is asking for. + """ return len(self.results) @@ -458,21 +702,48 @@ def verify( # Once per verify, not once per claim: it is a property of this process, and # reading the umask inside it is a read-modify-write (capture.process_umask). current_fingerprint = capture.env_fingerprint() - results: list[GradeResult] = [] - for c in manifest.claims: - claim = Claim.from_dict(c["claim"]) + + def _graded(position: int, stored: dict, claim: Claim, siblings=None) -> GradeResult: result = _verify_claim( - c, claim, manifest, entries, events, repo, ledger_objects + stored, claim, manifest, entries, events, repo, ledger_objects, siblings ) + # Assigned around the ladder for the same reason as the two below: which + # claim retired this one is a fact about the seal window, not something + # the evidence earns. + result.superseded_by = _superseded_by(stored, position, manifest.claims) # Assigned around the ladder, like witness-unavailable: the environment # comparison is a fact about this run, not a grade the evidence earns. # It is set on EVERY path, including witness-unavailable — the sealed # fingerprint is in the note whether or not the ledger still holds the # entry it names. result.env_status, result.env_reason = _env_comparison( - c.get("evidence"), current_fingerprint + stored.get("evidence"), current_fingerprint ) - results.append(result) + return result + + # Two passes, the same shape `seal` grades in. A conjunction is graded + # against its siblings' results FROM THIS NOTE, so pass one grades every + # ordinary claim and pass two grades the conjunctions over what pass one + # produced. Regrading a conjunction in one pass with no siblings resolves + # every conjunct to "no match" and verifies `unknown` — so a note that + # sealed green would fail --strict, which is this restructure's whole point. + # Resolving them against the live claims.jsonl instead would be worse still: + # it breaks the source-of-truth rule above and lets a later, unsealed claim + # change an old commit's verdict. + results: list = [None] * len(manifest.claims) + deferred: list = [] + for position, c in enumerate(manifest.claims): + claim = _stored_claim(c, position) + if claim.ctype == CONJUNCTION: + deferred.append((position, c, claim)) + continue + results[position] = _graded(position, c, claim) + # Superseded pass-one results are already marked, and grade_conjunction drops + # them from the pool: a retired attempt shares its label with the claim that + # replaced it, so counting it would make that name ambiguous. + siblings = [r for r in results if r is not None] + for position, c, claim in deferred: + results[position] = _graded(position, c, claim, siblings) return VerifyReport( commit=manifest.commit, tree=manifest.tree, @@ -617,6 +888,7 @@ def _verify_claim( events: list, repo: Path, ledger_objects: Optional[Path], + siblings: Optional[list] = None, ) -> GradeResult: """Grade one sealed claim, checking the note's binding to its evidence first. @@ -631,10 +903,14 @@ def _verify_claim( ``witness-unavailable`` rather than recomputing a fresh verdict from whatever the ledger holds now. That silent recomputation is the defect this whole unit exists to close. + + A conjunction takes the first path: it binds no event, so the seal wrote it + no evidence block and there is no binding to check. ``siblings`` is what it + is graded against, and it is forwarded unread on every other path. """ evidence = stored.get("evidence") if not isinstance(evidence, dict): - return _regrade(claim, manifest, events, repo, ledger_objects) + return _regrade(claim, manifest, events, repo, ledger_objects, siblings) sealed_grade = stored.get("grade") idx = evidence.get("event_index") @@ -651,7 +927,7 @@ def _verify_claim( sealed_grade=sealed_grade, ) - result = _regrade(claim, manifest, events, repo, ledger_objects) + result = _regrade(claim, manifest, events, repo, ledger_objects, siblings) result.evidence_bound = True result.sealed_grade = sealed_grade return result @@ -662,6 +938,89 @@ def _is_index(value) -> bool: return isinstance(value, int) and not isinstance(value, bool) +def _stored_claim(entry, position: int) -> Claim: + """Parse one stored claim entry, refusing rather than raising out of verify. + + ``Claim.__post_init__`` is the claim vocabulary's gate, and it raises + ``ClaimError`` — which is not a ``ManifestError``, so it escaped ``verify`` + and the CLI's handler as an uncaught traceback. docs/COMPAT.md's "Version 3" + section names this exact shape as the vocabulary break an OLD reader hits on + a v3 note, and the ``MANIFEST_VERSION`` bump is what steers such a reader + into a version refusal first. This closes the same hole from the other end: + THIS reader, meeting a claim type it cannot parse in a note whose version it + accepts, now also refuses instead of crashing. + + A whole-manifest refusal, not a per-claim ``unknown``: grading around the gap + would print a verdict over the claims this binary happened to understand, + which docs/COMPAT.md rejects in as many words as the worse of the two + options. The position is named because a note has many claims and "one of + them is unreadable" is not an actionable sentence. + """ + if not isinstance(entry, dict): + raise ManifestFormatError( + f"claim entry {position} is a {type(entry).__name__}, not an object" + ) + stored = entry.get("claim") + if not isinstance(stored, dict): + raise ManifestFormatError( + f"claim entry {position} carries no claim object" + ) + try: + return Claim.from_dict(stored) + except (ClaimError, TypeError, ValueError) as exc: + # ClaimError is the vocabulary gate. TypeError/ValueError cover the field + # shapes it never gets to look at — `tuple(d.get("event_indices"))` on an + # int raises before __post_init__ runs at all. + raise ManifestFormatError(f"claim entry {position} is unreadable: {exc}") from exc + + +def _claim_key(entry) -> Optional[tuple]: + """The (ctype, label) identity of a stored claim entry, or None.""" + if not isinstance(entry, dict): + return None + claim = entry.get("claim") + if not isinstance(claim, dict): + return None + return claim.get("ctype"), claim.get("label") + + +def live_claims(claims: list) -> list: + """A manifest's claim entries that no later claim in the window retired. + + The one definition of "live", shared by the verdict (``verify``) and by the + seal's own summary line, so the two cannot report different denominators for + the same note. + """ + return [c for i, c in enumerate(claims) if _superseded_by(c, i, claims) is None] + + +def _superseded_by(stored: dict, position: int, claims: list) -> Optional[int]: + """The entry that retired this one, or None — validated, not trusted. + + A supersession mark SUBTRACTS a claim from the verdict, so it is the one + published field that can turn a red note green. It is therefore checked + against the definition that produced it rather than taken at face value: the + target must be a real forward index into this note's own claim list, and it + must carry the same ``(ctype, label)`` the seal supersedes by. + + A mark that fails any of those is dropped, which counts the claim toward the + verdict — the direction that refuses rather than the one that excuses. This + is not a forgery barrier and does not pretend to be one: whoever can rewrite + a note can rewrite a grade directly (docs/TRUST_MODEL.md). It is the same + fail-closed reading didrun applies to every other field it did not compute + in this process. + """ + mark = stored.get("superseded_by") + if not _is_index(mark): + return None + if not (position < mark < len(claims)): + return None + key = _claim_key(stored) + if key is None or key != _claim_key(claims[mark]): + return None + return mark + + def _witness_problem(entries: list, idx, recorded_hash) -> Optional[str]: """Why the live ledger cannot supply the sealed evidence, or None. @@ -697,9 +1056,10 @@ def _regrade( events: list, repo: Path, ledger_objects: Optional[Path], + siblings: Optional[list] = None, ) -> GradeResult: try: - return grade(claim, manifest.tree, events, repo, ledger_objects) + return grade(claim, manifest.tree, events, repo, ledger_objects, siblings) except Exception as exc: # object gc'd or unreadable → unknown, not crash return GradeResult(claim, "unknown", reason=f"regrade failed: {exc}") @@ -798,6 +1158,78 @@ def _read_note(repo: Path, commit: str) -> Optional[bytes]: return proc.stdout if proc.returncode == 0 else None +def _recorded_window(body: Optional[bytes]) -> Optional[tuple]: + """The claim window an existing note records, or None when it records none. + + None is NOT ``(0, 0)`` and it is not the reading default either. A note + published before v0.2 carries no window keys at all, so what it covered is + genuinely unknown — and ``Manifest.from_json``'s defaults (0 / len(claims)) + would turn that unknown into a confident ``[0, N)`` the note never claimed, + which is exactly the kind of invented fact that would let the guard below + wave a lossy overwrite through. So this reads key PRESENCE off the raw body. + + A body this function cannot parse also yields None: a foreign or corrupt + note under the same ref is the case where an overwrite destroys the most and + explains the least. + """ + if not body: + return None + try: + d = json.loads(body) + except ValueError: + return None + if not isinstance(d, dict): + return None + lo, hi = d.get("claims_from"), d.get("claims_to") + if not _is_index(lo) or not _is_index(hi): + return None + return lo, hi + + +def _refuse_narrowing_reseal( + commit: str, prior_note: Optional[bytes], manifest: Manifest, reseal: bool +) -> None: + """Refuse to replace a commit's note with a narrower record. + + ``git notes add -f`` force-replaces, and a seal is scoped to + ``all_claims[watermark:]``. Together those two made the documented repair + loop lossy: re-sealing the same commit published a note holding only the + post-watermark claims and silently destroyed the record of everything the + earlier note held. Nothing in the artifact said so, which is what made it + dangerous rather than merely surprising. + + Three ways past this, and they are all explicit: the commit carries no note; + the new window CONTAINS the recorded one (a superset re-seal loses nothing); + or the operator passed ``reseal``. An unknown recorded window is not treated + as containable — see ``_recorded_window``. + """ + if prior_note is None or reseal: + return + new_window = (manifest.claims_from, manifest.claims_to) + old_window = _recorded_window(prior_note) + if old_window is not None and ( + new_window[0] <= old_window[0] and new_window[1] >= old_window[1] + ): + return + recorded = ( + "the note records no claim window (it was published before v0.2, or it is " + "not a didrun manifest), so what replacing it would destroy cannot be " + "determined from here" + if old_window is None + else ( + f"the note records claims [{old_window[0]},{old_window[1]}), " + f"which that does not contain" + ) + ) + raise ManifestError( + f"refusing to replace the note on {commit[:12]}: this seal covers claims " + f"[{new_window[0]},{new_window[1]}) and {recorded} — replacing the note " + f"would drop the earlier record, which exists nowhere else. Commit first " + f"so this unit gets its own note, or re-run with --reseal to replace it " + f"anyway (the narrower window is then visible in claims_from/claims_to)." + ) + + def _attach_note(repo: Path, commit: str, manifest: Manifest) -> None: """Publish the manifest as ``commit``'s note. Fails closed. @@ -912,6 +1344,16 @@ def _resolve_manifest( if show.returncode == 0 and show.stdout.strip(): try: m = Manifest.from_json(show.stdout.encode("ascii")) + except ManifestFormatError: + # Not a readable manifest: a foreign note under the same + # ref, or a corrupt body. Skipped and counted, so one bad + # note cannot hide every good one. Ordered BEFORE the + # clause below because it is a subclass of it — and it + # states by type what used to happen by accident, when a + # bare `except Exception` skipped these only because + # `json` raised something that was not a ManifestError. + skipped += 1 + continue except ManifestError: # A version this binary cannot read. Fail closed: the # alternative is scanning past it and reporting the @@ -1093,7 +1535,11 @@ class RedactedClaim: def _redact_result( - r: GradeResult, session: Session, entries: list, claim_index: int = 0 + r: GradeResult, + session: Session, + entries: list, + claim_index: int = 0, + superseded_by: Optional[int] = None, ) -> RedactedClaim: d = r.to_dict() # Redact argv in the exported claim view, and DECLARE the projection that @@ -1135,6 +1581,22 @@ def _field(name: str, kind: str, value: str) -> str: _field(f"claim.pathspecs[{i}]", "pathspec", spec) for i, spec in enumerate(d["claim"]["pathspecs"]) ] + # A conjunct name is the LABEL of another claim, so it is scanned as one. + # Every conjunct is redacted by the same deterministic projection that + # redacted the label it refers to, which is what keeps a conjunction + # resolvable at verify time: both sides of the match carry the same text. + d["claim"]["conjuncts"] = [ + _field(f"claim.conjuncts[{i}]", "label", name) + for i, name in enumerate(d["claim"]["conjuncts"]) + ] + # `reason` is generated text for every other claim type, and it was exported + # unredacted for that reason (docs/COMPAT.md). A conjunction's reason NAMES a + # conjunct, which is operator-authored, so it stops being generated-only text + # here — and "blocked by the gate, published verbatim on the override" is the + # exact defect the claim-field pass exists to have closed. Redacted for every + # claim, not only for a conjunction: a reason a later version interpolates + # something into is then covered without anyone remembering to add it. + d["reason"] = _field("reason", "reason", d["reason"]) d["delta"] = [ { "status": change["status"], @@ -1154,9 +1616,33 @@ def _field(name: str, kind: str, value: str) -> str: # projection it always did. "fields": fields, } + pairs = d.pop("conjunct_grades", None) + if pairs is not None: + # The pairs' labels ARE `claim.conjuncts`, one per declared conjunct in + # declaration order (claims.grade_conjunction builds them that way), so + # they take the redacted text from above instead of being scanned — and + # reported to the export gate — a second time. The length check is not + # decoration: a silent zip would drop a graded conjunct out of the + # published record, so it fails closed instead. + names = d["claim"]["conjuncts"] + if len(pairs) != len(names): + raise ManifestError( + f"refusing to publish a conjunction claim carrying " + f"{len(pairs)} graded conjunct(s) for {len(names)} declared " + f"conjunct(s): the note would not say what each member earned" + ) + d["conjunct_grades"] = [ + {"label": name, "grade": pair["grade"]} + for name, pair in zip(names, pairs) + ] block = _evidence_block(entries, r.supporting_event_index) if block is not None: d["evidence"] = block + # Written only on the entries that carry one. A permanently-null key on + # every claim in every note would say nothing, and the absence of the key is + # already the complete statement "this claim was not retired". + if superseded_by is not None: + d["superseded_by"] = superseded_by return RedactedClaim(payload=d, findings=findings, bytes_scanned=scanned) diff --git a/src/didrun/redact.py b/src/didrun/redact.py index a1fc2a2..12948f4 100644 --- a/src/didrun/redact.py +++ b/src/didrun/redact.py @@ -70,8 +70,8 @@ TIERS = (TIER_BLOCK, TIER_NOTICE) # Where a finding is. `argv`/`stdout`/`stderr` address a recorded event; -# `label`, `pathspec` and `delta-path` address individual exported claim fields; -# `manifest` addresses the serialized bytes about to be published. +# `label`, `pathspec`, `delta-path` and `reason` address individual exported +# claim fields; `manifest` addresses the serialized bytes about to be published. SOURCE_KINDS = ( "argv", "stdout", @@ -79,12 +79,13 @@ "label", "pathspec", "delta-path", + "reason", "manifest", ) # Source kinds that address ONE exported field rather than a stream of bytes. # A line number inside them would always be 1, which is noise, not locality. -_LINELESS_KINDS = ("argv", "label", "pathspec", "delta-path") +_LINELESS_KINDS = ("argv", "label", "pathspec", "delta-path", "reason") @dataclass(frozen=True) diff --git a/src/didrun/render.py b/src/didrun/render.py index 1a45e19..e1511a1 100644 --- a/src/didrun/render.py +++ b/src/didrun/render.py @@ -162,10 +162,63 @@ def render_chain_banner(report) -> str: return _c(f"x {text}", "31") if text else "" +def superseded_text(report) -> str: + """One line for the claims a later claim in the same window retired, or "". + + Appended only when it is nonzero, on the `not-recorded` precedent: zero + superseded claims is the shape of nearly every window, and a counter on + every verify would be boilerplate about a case that did not occur. + + What it says is the honest bound of the mechanism. Retrying a gate until it + passes is not prevented — it is RETAINED and counted here, which is strictly + better than the invisible truncation it replaces. + """ + count = report.superseded_count + if not count: + return "" + return ( + f"{count} claim{'' if count == 1 else 's'} superseded within this seal " + f"window (retained in the record, excluded from the verdict)" + ) + + +def _conjunct_token(grade) -> str: + """The token for one conjunct of a conjunction — never a grade for a name + that resolved to nothing. + + ``None`` means the label matched no single live claim in the seal window. + That is not a grade, and rendering it through ``_GRADE_DISPLAY`` would print + it as UNKNOWN — a real grade, and a claim the note never made. The + conjunction itself already grades unknown in that case; this row says which + member is the reason. + """ + if grade is None: + return "UNRESOLVED" + return _GRADE_DISPLAY.get(grade, ("UNKNOWN", "33", "?", ""))[0] + + +def _row_detail(r, reason: str) -> str: + """The detail cell text, prefixed when the row is record rather than verdict. + + Without this a retired FAILED row still reads FAILED under an ALL + RECORDED-EXACT headline, which looks like the tool contradicting itself. The + grade and reason are NOT rewritten — that is the whole point of keeping the + entry — so the marker goes in front of them. + """ + if not r.is_superseded: + return reason + return f"superseded by claim #{r.superseded_by} (record, not verdict) — {reason}" + + def render_verdict(report, width: int = 80) -> str: """Render a VerifyReport as the CLI instrument panel.""" lines: list[str] = [] - results = sorted(report.results, key=lambda r: _SORT_RANK.get(r.grade, 9)) + # Retired rows sort BELOW every live one, whatever they graded: they are the + # record of how the unit got here, and the live claims are what a reviewer + # is being asked about. + results = sorted( + report.results, key=lambda r: (r.is_superseded, _SORT_RANK.get(r.grade, 9)) + ) banner = render_chain_banner(report) worst = report.worst_status if report.results else "empty" @@ -181,8 +234,12 @@ def render_verdict(report, width: int = 80) -> str: return "\n".join(lines) token, color, marker, _gloss = _GRADE_DISPLAY.get(worst, ("UNKNOWN", "33", "?", "")) - verified = sum(1 for r in report.results if r.grade in ("tree-exact", "scope-exact")) - total = len(report.results) + # Counted over the LIVE set, which is the set the verdict was computed over. + # Counting retired claims here would print "1/2 recorded-exact" under an ALL + # RECORDED-EXACT headline; the retained count gets its own line instead. + live = report.live_results + verified = sum(1 for r in live if r.grade in ("tree-exact", "scope-exact")) + total = len(live) # Verdict header — the one line a reviewer reads first. The marker prefix # carries severity even with color stripped. @@ -198,6 +255,9 @@ def render_verdict(report, width: int = 80) -> str: # How many of those verdicts were checked against the recorded entry the # seal named, rather than regraded against whatever ledger is on disk now. lines.append(f" {report.evidence_bound_count}/{total} claims evidence-bound") + superseded = superseded_text(report) + if superseded: + lines.append(f" {superseded}") lines.append(f" {env_summary_text(report)}") if report.secrets_override: lines.append(_c(" ! sealed with --allow-secrets (redacted export)", "33")) @@ -211,7 +271,7 @@ def render_verdict(report, width: int = 80) -> str: for r in results: token, color, marker, gloss = _GRADE_DISPLAY.get(r.grade, ("UNKNOWN", "33", "?", "")) label = _sanitize(r.claim.label)[:26] - detail = _sanitize(r.reason) + detail = _row_detail(r, _sanitize(r.reason)) if r.exit_code is not None: detail = f"exit {r.exit_code} {detail}" lines.append(f" {marker} {_c(f'{token:<12}', color)} {label:<26} {detail}") @@ -223,6 +283,14 @@ def render_verdict(report, width: int = 80) -> str: lines.append( f" {'':<12} {'':<26} {ENV_DRIFT}: {_sanitize(r.env_reason)}" ) + # A conjunction's members, each with what it earned. Uncapped on + # purpose: the row's grade IS one of these, and hiding the tail would + # hide the conjunct the verdict came from. + for label, cgrade in r.conjunct_grades: + lines.append( + f" {'':<12} {'':<26} of {_sanitize(label)}: " + f"{_conjunct_token(cgrade)}" + ) # Drill-down: show the delta for stale/scope, capped. for change in r.delta[:5]: lines.append(f" {'':<12} {'':<26} {change.status} {_sanitize(change.path)}") @@ -243,9 +311,12 @@ def render_html(report, title: str = "didrun evidence report") -> str: CSP; 100% of dynamic content HTML-escaped; failing/stale first; responsive; print-clean. This is the shareable evidence surface. """ - results = sorted(report.results, key=lambda r: _SORT_RANK.get(r.grade, 9)) - verified = sum(1 for r in report.results if r.grade in ("tree-exact", "scope-exact")) - total = len(report.results) + results = sorted( + report.results, key=lambda r: (r.is_superseded, _SORT_RANK.get(r.grade, 9)) + ) + live = report.live_results + verified = sum(1 for r in live if r.grade in ("tree-exact", "scope-exact")) + total = len(live) worst = report.worst_status if report.results else "empty" all_ok = report.all_verified @@ -256,9 +327,24 @@ def esc(s: str) -> str: for r in results: token, _color, _marker, _gloss = _GRADE_DISPLAY.get(r.grade, ("UNKNOWN", "", "?", "")) grade_class = r.grade.replace("-", "_") - detail = esc(r.reason) + if r.is_superseded: + # A second class, not a replaced one: the row keeps the grade it + # earned (that is the record) and the tone says it does not vote. + grade_class += " superseded" + detail = _row_detail(r, esc(r.reason)) exit_txt = f"exit {r.exit_code}" if r.exit_code is not None else "—" argv = " ".join(r.claim.get("argv_preview", [])) if isinstance(r.claim, dict) else "" + # The conjunct list rides in the detail column, above the delta: a + # conjunction's grade is one of these, so it is the first thing a reader + # of that row needs. Same list markup as the delta — a conjunction + # carries an existing grade, so no new badge or token is introduced. + conjunct_html = "" + if r.conjunct_grades: + items = "".join( + f"
  • {esc(_conjunct_token(g))} {esc(label)}
    • " + for label, g in r.conjunct_grades + ) + conjunct_html = f"
      {items}
    " delta_html = "" if r.delta: items = "".join( @@ -280,7 +366,8 @@ def esc(s: str) -> str: f"{esc(token)}" f"{esc(claim_label)}" f"{detail}" - f"{esc(exit_txt)}{env_html}{delta_html}" + f"{esc(exit_txt)}{env_html}" + f"{conjunct_html}{delta_html}" f"" ) @@ -291,6 +378,8 @@ def esc(s: str) -> str: chain_html = ( f'
    {esc(chain_text)}
    ' if chain_text else "" ) + superseded = superseded_text(report) + superseded_html = f"{esc(superseded)} ·" if superseded else "" if all_ok: verdict_class, verdict_text = "ok", "ALL RECORDED-EXACT" @@ -377,6 +466,11 @@ def esc(s: str) -> str: tr.stale td, tr.unknown td, tr.witness_unavailable td {{ background:color-mix(in srgb, var(--warn) 10%, transparent); }} tr.failed td {{ background:color-mix(in srgb, var(--bad) 12%, transparent); }} + /* A retired row is record, not verdict: toned down and stripped of its accent + bar so it cannot be read as a live problem. Dimmed, never hidden — the + whole point of supersession is that the earlier attempt stays visible. */ + tr.superseded td {{ opacity:.62; }} + tr.superseded td.status {{ box-shadow:none; }} tr.failed td.status {{ box-shadow:inset 3px 0 0 var(--bad); }} tr.stale td.status {{ box-shadow:inset 3px 0 0 var(--warn); }} tr.unknown td.status {{ box-shadow:inset 3px 0 0 var(--warn); }} @@ -413,6 +507,7 @@ def esc(s: str) -> str: {chain_html}
    {esc(verified)}/{esc(total)} claims recorded-exact · {esc(report.evidence_bound_count)}/{esc(total)} claims evidence-bound · + {superseded_html} {esc(env_summary_text(report))} · commit {esc(report.commit[:12])} · tree {esc(report.tree[:12])} · resolved-by {esc(report.resolved_by)} · coverage {coverage_html}
    diff --git a/tests/compat/synthetic.py b/tests/compat/synthetic.py index de09293..45f5e1b 100644 --- a/tests/compat/synthetic.py +++ b/tests/compat/synthetic.py @@ -208,6 +208,48 @@ def _claim_entry( return d +def _conjunction_entry( + label: str, + grade: str, + conjunct_grades: tuple, + declared_at: int, +) -> dict: + """One graded conjunction, in the shape seal() writes into a v3 note. + + ``grade`` is declared, not derived: every value in this fixture is a literal + so the legs measure the instrument against a shape whose every field is + stated. It binds no event and declares no pathspec, so its `argv_preview` is + empty — the same thing seal() writes for a claim with no supporting event. + """ + result = GradeResult( + claim=Claim( + ctype="conjunction", + label=label, + event_indices=(), + declared_at_index=declared_at, + conjuncts=tuple(name for name, _g in conjunct_grades), + ), + grade=grade, + reason="synthetic fixture", + conjunct_grades=conjunct_grades, + ) + d = result.to_dict() + d["claim"]["argv_preview"] = [] + return d + + +def _strip_post_v2_claim_keys(body: dict) -> None: + """Remove the claim keys no note before v3 carried. + + ``Claim.to_dict`` emits ``conjuncts`` for every claim, so a body built from + the current dataclass carries it whatever version number the fixture stamps + on the manifest. A note that says version 1 and holds a v3 claim shape would + make the older legs read as passing on a shape they never see in an archive. + """ + for entry in body.get("claims", []): + entry.get("claim", {}).pop("conjuncts", None) + + def _coverage() -> dict: return {"events": 3, "complete": 3, "observed_text_only": 0, "display_only": 0} @@ -266,10 +308,13 @@ def build_corpus(root: Path) -> SyntheticCorpus: broken[1]["entry_hash"] = ("0" if stored[0] != "0" else "1") + stored[1:] _write_log(history / "unit-07-broken-hash" / ".didrun" / "session.log", broken) - # Two notes. The second deliberately omits `secrets_override` — the one key - # from_json already reads with a v1-reproducing default — so the round-trip - # leg's additive-key allowlist is exercised by the good path, not only by a - # meta-test. + # Four notes, one per manifest version this binary must read plus one that + # exercises the additive-key allowlist. The second deliberately omits + # `secrets_override` — the one key from_json already reads with a + # v1-reproducing default — so the allowlist is exercised by the good path and + # not only by a meta-test. The version ladder is covered on purpose: v3 is + # what this build writes, and a leg that only ever reads the version it wrote + # would say nothing about the 65 v1 notes already published. commit_a = _commit(root, "work-a.txt", "one\n") manifest_a = Manifest( version=MANIFEST_VERSION, @@ -287,10 +332,13 @@ def build_corpus(root: Path) -> SyntheticCorpus: ) _attach_note(root, commit_a, manifest_a.to_json()) + # A v2 note: the version before the claim vocabulary gained `conjunction`. + # Pinned to the literal 2 rather than MANIFEST_VERSION so this leg keeps + # reading a note one version behind whatever this build writes. commit_b = _commit(root, "work-b.txt", "two\n") body_b = json.loads( Manifest( - version=MANIFEST_VERSION, + version=2, commit=commit_b, tree=_SYNTHETIC_TREE, claims=[_claim_entry("tests-pass", "suite", "failed", 0, 0, exit_code=1)], @@ -299,8 +347,51 @@ def build_corpus(root: Path) -> SyntheticCorpus: ).to_json() ) del body_b["secrets_override"] + _strip_post_v2_claim_keys(body_b) _attach_note(root, commit_b, canonical_json(body_b)) + # A v3 note carrying a conjunction: the claim shape the vocabulary extension + # added. It binds no event and names its members by label, and the note holds + # what each member earned beside the grade the conjunction took from them. + commit_c = _commit(root, "work-c.txt", "three\n") + manifest_c = Manifest( + version=MANIFEST_VERSION, + commit=commit_c, + tree=_SYNTHETIC_TREE, + claims=[ + _claim_entry("tests-pass", "unit", "tree-exact", 0, 0), + _claim_entry("lint-clean", "lint", "tree-exact", 1, 1), + _conjunction_entry( + "release", + "tree-exact", + (("unit", "tree-exact"), ("lint", "tree-exact")), + 2, + ), + ], + coverage=_coverage(), + secrets_override=False, + ) + _attach_note(root, commit_c, manifest_c.to_json()) + + # A v1 note, in the shape the archives actually hold one: none of the keys + # any later version added. Every one of them is on v1's additive allowlist, + # so this is the leg's statement that a note published before v0.2 still + # round-trips under a reader three versions later. + commit_d = _commit(root, "work-d.txt", "four\n") + body_d = json.loads( + Manifest( + version=1, + commit=commit_d, + tree=_SYNTHETIC_TREE, + claims=[_claim_entry("tests-pass", "suite", "tree-exact", 0, 0)], + coverage=_coverage(), + ).to_json() + ) + for added_after_v1 in ("secrets_override", "secrets", "claims_from", "claims_to"): + del body_d[added_after_v1] + _strip_post_v2_claim_keys(body_d) + _attach_note(root, commit_d, canonical_json(body_d)) + expected = Expectations( log_statuses=( "intact", @@ -313,16 +404,19 @@ def build_corpus(root: Path) -> SyntheticCorpus: ), log_entries=(3, 1, 2, 1, 1, 0, 3), objects={"dirs": 2, "flat-64hex": 3, "git-fanout": 3}, - notes=2, + notes=4, claim_shape={ - "notes": 2, - "claims": 4, + "notes": 4, + "claims": 8, + # The conjunction binds no event, so it is not a multi-index claim + # and never becomes one: that shape is what the design review ruled + # out, and this counter is the leg that would catch it arriving. "multi-index": 0, "max-pathspecs": 1, - "pathspecs": {0: 3, 1: 1}, + "pathspecs": {0: 7, 1: 1}, "gap-ge-1": 2, "gap-ge-5": 1, - "grades": {"failed": 1, "scope-exact": 1, "stale": 1, "tree-exact": 1}, + "grades": {"failed": 1, "scope-exact": 1, "stale": 1, "tree-exact": 5}, }, ) return SyntheticCorpus(root=root, expected=expected) diff --git a/tests/compat/test_corpus_replay.py b/tests/compat/test_corpus_replay.py index f590624..9aa54f1 100644 --- a/tests/compat/test_corpus_replay.py +++ b/tests/compat/test_corpus_replay.py @@ -304,9 +304,20 @@ def test_leg1_chain_recompute(replay_source): # version legitimately lacks it, and both allowlists carry it. This entry is # the deliberate one-line diff the criterion exists to force — without it the # round-trip leg would go red across every stored note at once. +# `claims_from`/`claims_to` are P4.1's: the half-open claim window the seal +# covered, which made a truncating re-seal visible in the artifact. Both are read +# with a default (0 / len(claims)), so a stored note of either version +# legitimately lacks them and both allowlists carry them. Second deliberate +# one-line diff this criterion has forced, and the reason it exists. +# Version 3 is P4.2's: the claim-type vocabulary gained `conjunction`. That bump +# added no TOP-LEVEL manifest key — the conjunct list and the per-conjunct grades +# live inside a claim entry, and a claim entry round-trips verbatim — so v3's +# allowlist is v2's. It is spelled out rather than aliased so that a v3-only +# additive key stays a deliberate one-line diff here, exactly as for v1 and v2. MANIFEST_ADDITIVE_KEYS = { - 1: frozenset({"secrets_override", "secrets"}), - 2: frozenset({"secrets_override", "secrets"}), + 1: frozenset({"secrets_override", "secrets", "claims_from", "claims_to"}), + 2: frozenset({"secrets_override", "secrets", "claims_from", "claims_to"}), + 3: frozenset({"secrets_override", "secrets", "claims_from", "claims_to"}), } diff --git a/tests/test_claim_supersession.py b/tests/test_claim_supersession.py new file mode 100644 index 0000000..b443f1d --- /dev/null +++ b/tests/test_claim_supersession.py @@ -0,0 +1,625 @@ +"""P4.1 — supersede a claim inside the window instead of truncating the record. + +The pair these pin. Within one seal window a single `failed` or `stale` claim +made `all_verified` false with no in-window way to retire it, and the only two +escapes were seal-early (needs a commit) and re-seal — which force-replaced the +commit's note with `all_claims[watermark:]` and silently destroyed everything +the earlier note held. Discarding the ledger and starting over was the rational +third option, which is what the observed atomicity wall was. + +So: a later claim with the same `(ctype, label)` retires an earlier one inside +the window, both stay in the note, and a narrowing re-seal is refused unless the +operator asks for it. The retirement subtracts from the VERDICT, never from the +RECORD — every test here that asserts a green verdict also asserts the retired +claim is still present, with its original grade and reason. + +What this does NOT do: prevent claim-until-green. Every attempt is retained and +counted, which makes it visible. That is the honest bound and it is strictly +better than the invisible truncation it replaces. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import Claim, grade +from didrun.ledger import Session +from didrun import gitplumbing as gp + + +def _git(repo: Path, *a: str) -> None: + subprocess.run(["git", *a], cwd=str(repo), capture_output=True, text=True, check=True) + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _run(session: Session, repo: Path, code: str): + return run_wrapped([sys.executable, "-c", code], session, repo) + + +def _claim(session: Session, ctype: str, label: str, event_index: int) -> Claim: + """Declare a claim exactly as the CLI would: one event, declared at the tail.""" + claim = Claim( + ctype=ctype, + label=label, + event_indices=(event_index,), + declared_at_index=len(session.events()) - 1, + ) + M.declare_claim(session, claim) + return claim + + +def _note_json(repo: Path, commit: str = "HEAD") -> dict: + """The note body on ``commit``, parsed — read with raw git, not didrun.""" + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + assert proc.returncode == 0, "the commit carries no note" + body = proc.stdout + assert body.endswith(b"\n") + return json.loads(body[:-1]) + + +def _attach_raw_note(repo: Path, commit: str, body: bytes) -> None: + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-F", "-", commit], + cwd=str(repo), + input=body, + capture_output=True, + check=True, + ) + + +def _gate_pair(session: Session, repo: Path, tmp_path: Path) -> None: + """One gate run twice in one window: fails, is fixed, passes. No commit. + + The fix lives OUTSIDE the repo on purpose. A fix that edited a tracked file + would move the working tree away from HEAD's tree, and the second claim would + grade `stale` for that reason rather than for anything to do with + supersession — the test would then be measuring the tree digest. This keeps + the tree fixed so the only variable is the two attempts at the same gate. + """ + flag = tmp_path / "gate-fixed" + code = f"import os, sys; sys.exit(0 if os.path.exists({str(flag)!r}) else 1)" + _run(session, repo, code) # event 0 — the gate fails + _claim(session, "tests-pass", "t", 0) + flag.write_text("fixed\n", encoding="ascii") + _run(session, repo, code) # event 1 — the same gate passes + _claim(session, "tests-pass", "t", 1) + + +# --- test 1: the commit-free checkpoint -------------------------------------- + + +def test_a_later_claim_retires_an_earlier_one_with_no_commit(repo: Path, tmp_path: Path): + """The point of the phase: the gate opens without an intervening commit, and + the failed attempt is still in the note.""" + s = _session(repo) + _gate_pair(s, repo, tmp_path) + head = gp.head_commit(repo) + + m = M.seal(s, repo) + + assert len(m.claims) == 2, "the record must keep both attempts" + assert m.claims[0]["grade"] == "failed" + assert m.claims[0]["superseded_by"] == 1 + assert m.claims[1]["grade"] == "tree-exact" + assert "superseded_by" not in m.claims[1], "the live claim must carry no mark" + assert (m.claims_from, m.claims_to) == (0, 2) + + report = M.verify(s, repo) + assert report.all_verified is True, "the gate must open with no commit" + assert report.worst_status == "tree-exact", "the retired FAILED must not vote" + assert report.superseded_count == 1 + assert len(report.results) == 2 + assert [r.grade for r in report.live_results] == ["tree-exact"] + assert gp.head_commit(repo) == head, "no commit may be needed to converge" + + # The CI gate, not just the report object. + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_the_verdict_line_says_how_many_were_retired(repo: Path, tmp_path: Path, monkeypatch): + """A green headline over a retained FAILED row has to explain itself.""" + monkeypatch.setenv("NO_COLOR", "1") + s = _session(repo) + _gate_pair(s, repo, tmp_path) + M.seal(s, repo) + + out = render.render_verdict(M.verify(s, repo)) + assert "ALL RECORDED-EXACT" in out + assert "1/1 claims recorded-exact" in out, "the denominator must be the live set" + assert "1 claim superseded within this seal window" in out + # The retired row is still shown, and it says which claim replaced it. + assert "FAILED" in out + assert "superseded by claim #1 (record, not verdict)" in out + # And it is below the live row: the record explains, the verdict is asked about. + lines = out.splitlines() + live = next(i for i, ln in enumerate(lines) if "TREE-EXACT" in ln) + retired = next(i for i, ln in enumerate(lines) if "FAILED" in ln) + assert live < retired + + +def test_three_attempts_all_point_at_the_one_that_stands(repo: Path, tmp_path: Path): + """`superseded_by` names the LIVE entry, not the immediate successor.""" + s = _session(repo) + _gate_pair(s, repo, tmp_path) + _run(s, repo, "import sys; sys.exit(0)") + _claim(s, "tests-pass", "t", 2) + + m = M.seal(s, repo) + + assert len(m.claims) == 3 + assert m.claims[0]["superseded_by"] == 2 + assert m.claims[1]["superseded_by"] == 2 + assert "superseded_by" not in m.claims[2] + assert M.verify(s, repo).superseded_count == 2 + assert M.verify(s, repo).all_verified is True + + +# --- test 2: identity is (ctype, label), and nothing wider ------------------- + + +def test_a_different_label_or_ctype_does_not_supersede(repo: Path): + """Two gates are two gates. Supersession is same-type AND same-label only.""" + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "unit", 0) + _claim(s, "tests-pass", "integration", 0) + _claim(s, "lint-clean", "unit", 0) + + m = M.seal(s, repo) + + assert len(m.claims) == 3 + assert all("superseded_by" not in c for c in m.claims) + report = M.verify(s, repo) + assert report.superseded_count == 0 + assert len(report.live_results) == 3 + assert report.all_verified is True + + +# --- test 3: the record grows, never shrinks --------------------------------- + + +def test_the_retired_entry_keeps_its_grade_and_reason_verbatim(repo: Path, tmp_path: Path): + """Nothing is relabelled, re-graded or dropped — checked against an + independent grading of the same claim, not against a string in this file.""" + s = _session(repo) + _gate_pair(s, repo, tmp_path) + + tree = gp.commit_tree(repo, "HEAD") + declared = M._load_claims(s) + independent = grade(declared[0], tree, s.events(), repo, s.blobs.root) + + m = M.seal(s, repo) + retired = m.claims[0] + + assert retired["grade"] == independent.grade == "failed" + assert retired["reason"] == independent.reason + assert retired["exit_code"] == independent.exit_code == 1 + assert retired["claim"]["label"] == "t", "the label is not rewritten either" + assert retired["supporting_event_index"] == 0 + + # And verify hands the same values back, marked rather than altered. + result = M.verify(s, repo).results[0] + assert result.grade == "failed" + assert result.reason == independent.reason + assert result.is_superseded and result.superseded_by == 1 + + +def test_a_retired_stale_claim_keeps_its_delta(repo: Path): + """`stale` must carry its delta wherever it appears, retired included.""" + s = _session(repo) + _run(s, repo, "print(1)") # event 0, at the old tree + _claim(s, "tests-pass", "t", 0) + (repo / "calc.py").write_text("def add(a, b):\n return a + b # edited\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit") + _run(s, repo, "print(1)") # event 1, at the new tree + _claim(s, "tests-pass", "t", 1) + + m = M.seal(s, repo) + + assert m.claims[0]["grade"] == "stale" + assert m.claims[0]["superseded_by"] == 1 + assert [c["path"] for c in m.claims[0]["delta"]] == ["calc.py"] + assert m.claims[1]["grade"] == "tree-exact" + assert M.verify(s, repo).all_verified is True + + +# --- test 4: the window is in the artifact ----------------------------------- + + +def test_the_note_records_the_claim_window(repo: Path): + """claims_from/claims_to match the watermark arithmetic on every seal.""" + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "one", 0) + first_commit = gp.head_commit(repo) + M.seal(s, repo) + + body = _note_json(repo, first_commit) + assert body["claims_from"] == 0 + assert body["claims_to"] == 1 + assert len(body["claims"]) == 1 + + (repo / "calc.py").write_text("def add(a, b):\n return a + b # two\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "unit two") + _run(s, repo, "print(2)") + _claim(s, "tests-pass", "two", 1) + second = M.seal(s, repo) + + assert (second.claims_from, second.claims_to) == (1, 2) + body = _note_json(repo, gp.head_commit(repo)) + assert (body["claims_from"], body["claims_to"]) == (1, 2) + assert len(body["claims"]) == 1, "the window is the note's scope, verbatim" + + # The earlier commit's note is untouched: history is never rewritten. + assert _note_json(repo, first_commit)["claims_from"] == 0 + assert _note_json(repo, first_commit)["claims_to"] == 1 + + +def test_a_v1_note_reads_back_with_the_window_defaults(): + """Additive with a reading default, so every stored note still parses.""" + body = json.dumps( + { + "version": 1, + "commit": "0" * 40, + "tree": "1" * 40, + "claims": [{"grade": "tree-exact"}, {"grade": "tree-exact"}], + "coverage": {"total_events": 2, "by_coverage": {"complete": 2}}, + "secrets_override": False, + } + ).encode("ascii") + + m = M.Manifest.from_json(body) + + assert m.claims_from == 0 + assert m.claims_to == 2, "the default is len(claims), never None" + assert json.loads(m.to_json())["claims_to"] == 2 + + +# --- test 5: a narrowing re-seal is refused ---------------------------------- + + +def _ten_gates(repo: Path) -> Session: + s = _session(repo) + _run(s, repo, "print(1)") + for i in range(10): + _claim(s, "tests-pass", f"gate-{i}", 0) + return s + + +def test_a_narrowing_reseal_is_refused_and_names_both_windows(repo: Path): + s = _ten_gates(repo) + M.seal(s, repo) + assert _note_json(repo)["claims_to"] == 10 + + _claim(s, "tests-pass", "gate-10", 0) + _claim(s, "tests-pass", "gate-11", 0) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + message = str(exc.value) + assert "[0,10)" in message, "the message must name the recorded window" + assert "[10,12)" in message, "and the window that would replace it" + assert "--reseal" in message + # Refused means refused: the note and the watermark are where they were. + assert _note_json(repo)["claims_to"] == 10 + assert M._last_seal_watermark(s) == 10 + + m = M.seal(s, repo, reseal=True) + assert (m.claims_from, m.claims_to) == (10, 12) + assert _note_json(repo)["claims_from"] == 10, "the narrowing is now visible" + assert len(_note_json(repo)["claims"]) == 2 + + +def test_a_superset_reseal_needs_no_flag(repo: Path): + """Re-covering the recorded window plus more loses nothing, so it is allowed. + + The realistic way to land here is watermark rotation — the gitignored + seals.jsonl is gone, so the seal covers the whole claim history again. + """ + s = _ten_gates(repo) + M.seal(s, repo) + _claim(s, "tests-pass", "gate-10", 0) + (s.root / "seals.jsonl").unlink() + + m = M.seal(s, repo) + + assert (m.claims_from, m.claims_to) == (0, 11) + assert len(_note_json(repo)["claims"]) == 11 + + +def test_the_seal_line_counts_the_same_set_the_verdict_does(repo: Path, tmp_path: Path, capsys): + """`seal` and `verify` describe the same note, so "1/2 recorded-exact" beside + an ALL RECORDED-EXACT verdict is one of them being wrong.""" + s = _session(repo) + _gate_pair(s, repo, tmp_path) + capsys.readouterr() + + assert cli.main(["--repo", str(repo), "seal"]) == 0 + + out = capsys.readouterr().out + assert "1/1 claims recorded-exact" in out + assert "1 superseded" in out + assert "1/2" not in out + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_the_cli_wires_reseal_and_refuses_without_it(repo: Path, capsys): + s = _ten_gates(repo) + assert cli.main(["--repo", str(repo), "seal"]) == 0 + capsys.readouterr() + _claim(s, "tests-pass", "gate-10", 0) + + assert cli.main(["--repo", str(repo), "seal"]) == 2 + captured = capsys.readouterr() + assert "refusing to replace the note" in captured.err + assert "--reseal" in captured.err + assert "Traceback" not in captured.err + + assert cli.main(["--repo", str(repo), "seal", "--reseal"]) == 0 + assert (_note_json(repo)["claims_from"], _note_json(repo)["claims_to"]) == (10, 11) + + +# --- test 6: a legacy note carries no window, so it is not overwritten blind -- + + +def test_a_legacy_note_requires_reseal(repo: Path): + """An absent window is UNKNOWN, not [0,N): the reading default must never + let the guard conclude a legacy note is containable.""" + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "t", 0) + head = gp.head_commit(repo) + legacy = json.dumps( + { + "version": 1, + "commit": head, + "tree": gp.commit_tree(repo, head), + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + "secrets_override": False, + } + ).encode("ascii") + _attach_raw_note(repo, head, legacy) + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + message = str(exc.value) + assert "records no claim window" in message + assert "--reseal" in message + assert "[0,1)" in message, "the seal's own window is still named" + + m = M.seal(s, repo, reseal=True) + assert (m.claims_from, m.claims_to) == (0, 1) + + +def test_a_foreign_note_is_not_overwritten_blind(repo: Path): + """A body that is not a manifest at all reads as no window, for the same + reason: it is the case where an overwrite destroys the most.""" + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "t", 0) + _attach_raw_note(repo, gp.head_commit(repo), b"someone else's note body") + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo) + assert "records no claim window" in str(exc.value) + assert M.seal(s, repo, reseal=True).claims_to == 1 + + +# --- test 7: a missing watermark file is never silent ------------------------ + + +def test_a_missing_watermark_file_warns_and_proceeds(repo: Path, capsys): + s = _ten_gates(repo) + M.seal(s, repo) + capsys.readouterr() + seals = s.root / "seals.jsonl" + assert seals.exists() + seals.unlink() + + m = M.seal(s, repo) + + err = capsys.readouterr().err + assert "no seal watermark file" in err + assert "WHOLE claim history" in err + assert "10 declared claim(s)" in err + assert m.claims_from == 0, "the watermark really did reset" + assert seals.exists(), "and the seal proceeded" + + +def test_the_warning_is_silent_once_the_watermark_exists(repo: Path, capsys): + """It fires on absence, not on every seal — otherwise it is ignorable noise. + + The first seal of a session also has no watermark file and also warns: the + two are genuinely indistinguishable from inside `seal`, and the warning text + says so rather than picking one. + """ + s = _session(repo) + _run(s, repo, "print(1)") + _claim(s, "tests-pass", "one", 0) + M.seal(s, repo) + assert "no seal watermark file" in capsys.readouterr().err + + _git(repo, "commit", "-q", "--allow-empty", "-m", "unit two") + _claim(s, "tests-pass", "two", 0) + M.seal(s, repo) + assert "no seal watermark file" not in capsys.readouterr().err + + +def test_no_claims_no_watermark_warning(repo: Path, capsys): + """Nothing to re-cover, nothing to warn about.""" + s = _session(repo) + _run(s, repo, "print(1)") + M.seal(s, repo) + assert "no seal watermark file" not in capsys.readouterr().err + + +# --- test 8: supersession is window-local, so the documented loop is unchanged + + +def test_the_same_label_in_a_later_window_is_not_superseded(repo: Path): + """What test_fix_loop_converges encodes, asserted directly: the watermark + advanced between the two seals, so the second window holds one claim and + nothing in the first is reachable from it.""" + s = _session(repo) + _run(s, repo, "import sys; sys.exit(1)") + _claim(s, "tests-pass", "t", 0) + M.seal(s, repo) + first_commit = gp.head_commit(repo) + + (repo / "calc.py").write_text("def add(a, b):\n return a + b # fixed\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "fix") + _run(s, repo, "import sys; sys.exit(0)") + _claim(s, "tests-pass", "t", 1) + second = M.seal(s, repo) + + assert (second.claims_from, second.claims_to) == (1, 2) + assert len(second.claims) == 1 + assert all("superseded_by" not in c for c in second.claims) + report = M.verify(s, repo) + assert report.all_verified is True + assert len(report.results) == 1 + assert report.superseded_count == 0 + + # History is never rewritten: the first commit's note still shows FAILED and + # nothing retired it from the later window. + old = M.verify(s, repo, commitish=first_commit) + assert old.worst_status == "failed" + assert old.superseded_count == 0 + + +# --- the vacuous-green guard ------------------------------------------------ + + +def _result(label: str, grade_name: str, superseded_by=None): + from didrun.claims import GradeResult + + return GradeResult( + Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0), + grade_name, + reason=f"{grade_name} reason", + supporting_event_index=0, + superseded_by=superseded_by, + ) + + +def test_a_window_of_only_superseded_results_is_never_green(): + """`all([])` is True, so filtering the superseded out of `all_verified` + without re-checking non-emptiness would make a window whose every claim was + retired verify vacuously green — a green verdict over zero live claims.""" + report = M.VerifyReport( + commit="a" * 40, + tree="b" * 40, + resolved_by="commit", + results=[ + _result("t", "tree-exact", superseded_by=1), + _result("t", "tree-exact", superseded_by=1), + ], + coverage={}, + chain_status="intact", + ) + + assert report.live_results == [] + assert report.superseded_count == 2 + assert report.all_verified is False + # And it must not read as an unsealed commit either: there IS a record here. + assert report.worst_status == "unknown" + assert report.total == 2 + out = render.render_verdict(report) + assert "ALL RECORDED-EXACT" not in out + assert "NO CLAIMS" not in out + assert "2 claims superseded" in out + + +def test_an_empty_report_still_reads_as_empty(): + """The complement: no results at all is `empty`, which is a different fact.""" + report = M.VerifyReport( + commit="a" * 40, tree="b" * 40, resolved_by="none", results=[], coverage={} + ) + assert report.worst_status == "empty" + assert report.all_verified is False + assert report.superseded_count == 0 + + +@pytest.mark.parametrize( + "mark", + [ + 0, # itself + -1, # backwards + 2, # out of range + True, # a bool is not an index + "1", # not an int + None, # explicitly null + ], +) +def test_an_unusable_supersession_mark_counts_the_claim(repo: Path, mark): + """A mark that is not a forward index into this note's own claims is dropped, + and dropping it COUNTS the claim — the direction that refuses. + + This is not a forgery barrier and does not pretend to be one: whoever can + rewrite a note can rewrite a grade directly. It is the same fail-closed + reading didrun applies to every field it did not compute in this process. + """ + s = _session(repo) + _run(s, repo, "import sys; sys.exit(1)") + _claim(s, "tests-pass", "t", 0) + _run(s, repo, "import sys; sys.exit(0)") + _claim(s, "tests-pass", "t", 1) + m = M.seal(s, repo) + assert M.verify(s, repo).all_verified is True # the honest note is green + + body = _note_json(repo) + body["claims"][0]["superseded_by"] = mark + _attach_raw_note( + repo, + m.commit, + json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii"), + ) + + report = M.verify(s, repo) + assert report.superseded_count == 0, "an unusable mark must not retire a claim" + assert report.all_verified is False + assert report.worst_status == "failed" + + +def test_a_mark_pointing_at_a_different_gate_is_dropped(repo: Path): + """Supersession is defined by (ctype, label), so a mark across two different + gates is malformed — it would retire a claim nothing replaced.""" + s = _session(repo) + _run(s, repo, "import sys; sys.exit(1)") + _claim(s, "tests-pass", "unit", 0) + _run(s, repo, "import sys; sys.exit(0)") + _claim(s, "tests-pass", "integration", 1) + m = M.seal(s, repo) + + body = _note_json(repo) + assert all("superseded_by" not in c for c in body["claims"]) + body["claims"][0]["superseded_by"] = 1 + _attach_raw_note( + repo, + m.commit, + json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii"), + ) + + report = M.verify(s, repo) + assert report.superseded_count == 0 + assert report.all_verified is False diff --git a/tests/test_conjunction_claim.py b/tests/test_conjunction_claim.py new file mode 100644 index 0000000..4e14e4c --- /dev/null +++ b/tests/test_conjunction_claim.py @@ -0,0 +1,628 @@ +"""P4.2 — a conjunction claim, graded as the worst of its conjuncts. + +The property an 80-command proof ledger was hand-enforcing — "all N of these +gates passed against this tree" — had no representation in the tool. This gives +it one, and the whole design rests on a single negative: a conjunction must +never produce a grade its members do not already back. + +Test 1 is that negative, over every combination, and it is the gate. If any +combination comes out better than the worst of its pair, the design is weakening +a claim and the design review's refusal to substitute a marker for a witnessed +execution still stands. + +Test 2 is the one that fails if the verify-side resolution pass is missing: a +sealed conjunction regraded with no siblings resolves every conjunct to "no +match" and verifies `unknown`, so a note that sealed green fails --strict. It +proves the resolution reads the NOTE's own results by deleting claims.jsonl and +verifying again. +""" + +from __future__ import annotations + +import itertools +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import ( + CONJUNCTION, + GRADE_ORDER_WORST_FIRST, + Claim, + ClaimError, + GradeResult, + grade, +) +from didrun.ledger import Session +from didrun import gitplumbing as gp + +# tests/compat's leg 2 owns the note round-trip criterion. Test 6 imports it +# rather than restating it: a second copy of a three-part criterion is a second +# chance to write a weaker version of it, and the pack names that leg as the +# compat gate for exactly this reason. +from compat.test_corpus_replay import note_violations + +# The six grades a conjunct can carry, written out rather than imported, so the +# matrix below is a statement about the vocabulary and not a restatement of +# whatever the vocabulary happens to be. The equality against the severity order +# is what turns a new grade into a deliberate edit here. +GRADES = ( + "tree-exact", + "scope-exact", + "stale", + "unknown", + "failed", + "witness-unavailable", +) + + +def test_the_matrix_covers_every_grade_the_order_knows(): + """A grade added to the ladder must be added to test 1's matrix.""" + assert set(GRADES) == set(GRADE_ORDER_WORST_FIRST) + + +# --- helpers ----------------------------------------------------------------- + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _run(session: Session, repo: Path, code: str): + return run_wrapped([sys.executable, "-c", code], session, repo) + + +def _claim(session: Session, ctype: str, label: str, event_index: int) -> Claim: + """Declare an ordinary claim exactly as the CLI would.""" + claim = Claim( + ctype=ctype, + label=label, + event_indices=(event_index,), + declared_at_index=len(session.events()) - 1, + ) + M.declare_claim(session, claim) + return claim + + +def _conjunction(session: Session, label: str, *conjuncts: str) -> Claim: + claim = Claim( + ctype=CONJUNCTION, + label=label, + event_indices=(), + declared_at_index=len(session.events()) - 1, + conjuncts=tuple(conjuncts), + ) + M.declare_claim(session, claim) + return claim + + +def _result(label: str, grade_value: str) -> GradeResult: + """One graded sibling, at an arbitrary grade. Not produced by the ladder — + the point is to reach grade combinations no single fixture repo could.""" + return GradeResult( + claim=Claim( + ctype="tests-pass", + label=label, + event_indices=(0,), + declared_at_index=0, + ), + grade=grade_value, + reason="fixture sibling", + ) + + +def _report(*results: GradeResult) -> M.VerifyReport: + """A report over ``results`` alone, so `worst_status` is the only authority + this file consults for which of two grades is worse.""" + return M.VerifyReport( + commit="0" * 40, tree="1" * 40, resolved_by="commit", + results=list(results), coverage={}, + ) + + +def _graded_conjunction(*siblings: GradeResult, names=None) -> GradeResult: + conjuncts = names if names is not None else tuple(s.claim.label for s in siblings) + claim = Claim( + ctype=CONJUNCTION, + label="REL", + event_indices=(), + declared_at_index=9, + conjuncts=tuple(conjuncts), + ) + return grade(claim, "1" * 40, [], Path("."), None, siblings=list(siblings)) + + +def _note_body(repo: Path, commit: str = "HEAD") -> bytes: + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + ) + assert proc.returncode == 0, "the commit carries no note" + return proc.stdout + + +def _note_json(repo: Path, commit: str = "HEAD") -> dict: + body = _note_body(repo, commit) + assert body.endswith(b"\n") + return json.loads(body[:-1]) + + +def _entry(body: dict, label: str) -> dict: + (entry,) = [c for c in body["claims"] if c["claim"]["label"] == label] + return entry + + +# --- test 1: never better than the worst (THE criterion) --------------------- + + +@pytest.mark.parametrize("left,right", list(itertools.product(GRADES, GRADES))) +def test_a_conjunction_is_never_better_than_its_worst_conjunct(left: str, right: str): + """The gate: over EVERY pair of grades, in both orders. + + Three assertions, because "equals the worst" alone is satisfiable by an + implementation that invents a grade nobody claimed: + + 1. it equals what `worst_status` says about the two conjuncts together — + the same order the verdict line above it is computed with; + 2. it is one of the two grades its conjuncts actually carry, never a + third one; + 3. adding either conjunct back alongside the conjunction does not make the + verdict worse, which is "the conjunction is already at least as bad as + each of them" stated without reusing the implementation's own helper. + """ + a, b = _result("a", left), _result("b", right) + result = _graded_conjunction(a, b) + + expected = _report(a, b).worst_status + assert result.grade == expected, ( + f"conjunction over ({left}, {right}) graded {result.grade}, " + f"but the worst of them is {expected}" + ) + assert result.grade in (left, right), "a conjunction invented a grade" + conj_only = _report(result).worst_status + assert _report(result, a).worst_status == conj_only + assert _report(result, b).worst_status == conj_only + # The reason names the conjunct the grade came from, and its grade. + named = "a" if expected == left else "b" + assert repr(named) in result.reason + assert expected in result.reason + # The per-conjunct pairs travel with the result, in declaration order. + assert result.conjunct_grades == (("a", left), ("b", right)) + + +def test_a_tie_names_the_first_declared_conjunct(): + """Determinism: which of two equally-bad members is named is the operator's + order, never an iteration artefact.""" + first = _graded_conjunction(_result("a", "stale"), _result("b", "stale")) + reversed_ = _graded_conjunction(_result("b", "stale"), _result("a", "stale")) + assert "'a'" in first.reason + assert "'b'" in reversed_.reason + + +def test_a_conjunction_over_one_conjunct_is_that_conjunct_s_grade(): + for g in GRADES: + result = _graded_conjunction(_result("only", g)) + assert result.grade == g + assert result.conjunct_grades == (("only", g),) + + +# --- test 2: an all-green conjunction verifies, from the note alone ---------- + + +def _three_green_gates(s: Session, repo: Path) -> None: + """Three self-stable commands, one claim each, plus a conjunction of them.""" + for i, label in enumerate(("unit", "lint", "integration")): + _run(s, repo, "pass") + _claim(s, "tests-pass", label, i) + _conjunction(s, "REL", "unit", "lint", "integration") + + +def test_an_all_green_conjunction_seals_and_verifies_tree_exact(repo: Path): + s = _session(repo) + _three_green_gates(s, repo) + + m = M.seal(s, repo) + body = _note_json(repo) + assert _entry(body, "REL")["grade"] == "tree-exact" + assert [c["grade"] for c in m.claims] == ["tree-exact"] * 4 + + report = M.verify(s, repo) + by_label = {r.claim.label: r for r in report.results} + assert by_label["REL"].grade == "tree-exact" + assert by_label["REL"].conjunct_grades == ( + ("unit", "tree-exact"), + ("lint", "tree-exact"), + ("integration", "tree-exact"), + ) + assert report.all_verified is True + assert report.worst_status == "tree-exact" + # A conjunction binds no event, so it is not evidence-bound and must not be + # counted as though it were. + assert report.evidence_bound_count == 3 + assert report.total == 4 + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_the_conjunction_resolves_against_the_note_not_the_live_claims_file(repo: Path): + """Delete claims.jsonl; the grade must not move. + + This is the test that fails when verify grades conjunctions in one pass with + no siblings — and the one that fails again if a later change reaches for the + live claims file to get them. `verify` reads the manifest and the repo, never + the session's declared claims: a conjunction resolved against claims.jsonl + would let a claim declared after the seal change an old commit's verdict. + """ + s = _session(repo) + _three_green_gates(s, repo) + M.seal(s, repo) + + before = {r.claim.label: r.grade for r in M.verify(s, repo).results} + claims_file = repo / ".didrun" / "claims.jsonl" + assert claims_file.exists() + claims_file.unlink() + + after_report = M.verify(s, repo) + after = {r.claim.label: r.grade for r in after_report.results} + assert after == before + assert after["REL"] == "tree-exact" + assert after_report.all_verified is True + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + +def test_a_conjunction_graded_with_no_siblings_is_unknown_never_green(repo: Path): + """What the second pass buys, stated as an assertion. + + Grading a conjunction without its siblings — one pass over the note, which + is what verify did before this unit — resolves every conjunct to no match. + The result is `unknown`, so the failure mode of forgetting the pass is a + refusal, never a green verdict over members nobody looked at. + """ + s = _session(repo) + _three_green_gates(s, repo) + tree = gp.commit_tree(repo, "HEAD") + conjunction = M._load_claims(s)[-1] + assert conjunction.ctype == CONJUNCTION + + unbound = grade(conjunction, tree, s.events(), repo, s.blobs.root) + assert unbound.grade == "unknown" + assert "unit" in unbound.reason + + +# --- test 3: an unresolvable conjunct is unknown, never tree-exact ---------- + + +def test_a_missing_conjunct_grades_unknown_and_names_it(repo: Path): + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "unit", "nosuch") + + m = M.seal(s, repo) + entry = _entry(json.loads(m.to_json()), "REL") + assert entry["grade"] == "unknown" + assert "'nosuch'" in entry["reason"] + assert "names no claim" in entry["reason"] + assert entry["conjunct_grades"] == [ + {"label": "unit", "grade": "tree-exact"}, + {"label": "nosuch", "grade": None}, + ] + + report = M.verify(s, repo) + rel = next(r for r in report.results if r.claim.label == "REL") + assert rel.grade == "unknown" + assert "'nosuch'" in rel.reason + assert report.all_verified is False + assert report.worst_status == "unknown" + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 1 + + +def test_an_ambiguous_conjunct_grades_unknown_and_says_how_many(repo: Path): + """One label, two live claims: a member it cannot identify is a member it + cannot vouch for. Two different ctypes carrying one label are both live — + supersession is keyed on the (ctype, label) pair, so neither retires the + other.""" + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "gate", 0) + _claim(s, "lint-clean", "gate", 0) + _conjunction(s, "REL", "gate") + + m = M.seal(s, repo) + entry = _entry(json.loads(m.to_json()), "REL") + assert entry["grade"] == "unknown" + assert "'gate' names 2 claims" in entry["reason"] + assert M.verify(s, repo).all_verified is False + + +def test_a_conjunction_naming_a_conjunction_is_unknown(repo: Path): + """Resolution is one level deep: no recursion to bound, no cycle to detect, + and no grade asserted for a name this code will not follow.""" + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "INNER", "unit") + _conjunction(s, "OUTER", "INNER") + + m = M.seal(s, repo) + body = json.loads(m.to_json()) + assert _entry(body, "INNER")["grade"] == "tree-exact" + outer = _entry(body, "OUTER") + assert outer["grade"] == "unknown" + assert "'INNER' names no claim" in outer["reason"] + + +# --- test 4: superseded conjuncts are skipped ------------------------------- + + +def test_a_conjunction_resolves_to_the_surviving_claim(repo: Path, tmp_path: Path): + """P4.1's retired attempt shares its label with the claim that replaced it. + + The fix lives OUTSIDE the repo so the tree does not move: the only variable + is the two attempts at one gate. + """ + s = _session(repo) + flag = tmp_path / "gate-fixed" + code = f"import os, sys; sys.exit(0 if os.path.exists({str(flag)!r}) else 1)" + _run(s, repo, code) # event 0 — the gate fails + _claim(s, "tests-pass", "gate", 0) + flag.write_text("fixed\n", encoding="ascii") + _run(s, repo, code) # event 1 — the same gate passes + _claim(s, "tests-pass", "gate", 1) + _conjunction(s, "REL", "gate") + head = gp.head_commit(repo) + + m = M.seal(s, repo) + body = json.loads(m.to_json()) + assert m.claims[0]["grade"] == "failed" + assert m.claims[0]["superseded_by"] == 1, "P4.1's mark must be on the note" + assert m.claims[1]["grade"] == "tree-exact" + rel = _entry(body, "REL") + assert rel["grade"] == "tree-exact", "the retired FAILED must not be resolved to" + assert rel["conjunct_grades"] == [{"label": "gate", "grade": "tree-exact"}] + + report = M.verify(s, repo) + live = {r.claim.label: r.grade for r in report.live_results} + assert live == {"gate": "tree-exact", "REL": "tree-exact"} + assert report.superseded_count == 1 + assert report.all_verified is True + assert gp.head_commit(repo) == head, "no commit may be needed to converge" + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 + + # The record keeps the failed attempt, verbatim, and the row says why the + # green headline stands over it. + text = render.render_verdict(report) + assert "FAILED" in text + assert "superseded by claim #1" in text + + +def test_a_superseded_conjunction_does_not_vote(repo: Path): + """A conjunction is retired by the same (ctype, label) rule as any claim.""" + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "nosuch") # attempt 1: unresolvable + _conjunction(s, "REL", "unit") # attempt 2: the corrected declaration + + m = M.seal(s, repo) + assert m.claims[1]["grade"] == "unknown" + assert m.claims[1]["superseded_by"] == 2 + assert m.claims[2]["grade"] == "tree-exact" + report = M.verify(s, repo) + assert report.all_verified is True + assert report.superseded_count == 1 + + +# --- test 5: vocabulary invariants ------------------------------------------ + + +def test_a_conjunction_with_no_conjuncts_is_refused(): + with pytest.raises(ClaimError) as exc: + Claim(ctype=CONJUNCTION, label="REL", event_indices=()) + assert "conjunction" in str(exc.value) + + +def test_a_non_conjunction_carrying_conjuncts_is_refused(): + with pytest.raises(ClaimError): + Claim( + ctype="tests-pass", + label="t", + event_indices=(0,), + conjuncts=("unit",), + ) + + +def test_a_conjunction_binding_an_event_is_refused(): + """The illegitimate composite: a claim spanning several bound events grades + on the first success it finds and hides a witnessed failure. A conjunction + must not be built out of that shape, so its bound-event set is required to + be empty.""" + with pytest.raises(ClaimError) as exc: + Claim( + ctype=CONJUNCTION, + label="REL", + event_indices=(0,), + conjuncts=("unit",), + ) + assert "binds no event" in str(exc.value) + + +def test_the_cli_refuses_the_flags_a_conjunction_has_no_use_for(repo: Path, capsys): + """Refused, not silently ignored: an operator who passed --event meant it.""" + s = _session(repo) + _run(s, repo, "pass") + + assert cli.main(["--repo", str(repo), "claim", "conjunction", "--of", "unit", "--event", "0"]) == 2 + assert cli.main(["--repo", str(repo), "claim", "conjunction", "--of", "unit", "--path", "src/"]) == 2 + assert cli.main(["--repo", str(repo), "claim", "conjunction"]) == 2 + assert cli.main(["--repo", str(repo), "claim", "tests-pass", "--of", "unit"]) == 2 + assert M._load_claims(s) == [], "no refusal may have declared a claim" + + assert cli.main( + ["--repo", str(repo), "claim", "conjunction", "--label", "REL", "--of", " unit , lint "] + ) == 0 + (declared,) = M._load_claims(s) + assert declared.conjuncts == ("unit", "lint"), "--of is order-preserving and trimmed" + assert declared.event_indices == () + + +def test_the_cli_choices_are_the_vocabulary(): + """The parser's choices used to duplicate CLAIM_TYPES, which is how a CLI + keeps refusing a type the library has accepted for a release.""" + parser = cli.build_parser() + with pytest.raises(SystemExit): + parser.parse_args(["claim", "no-such-type"]) + assert parser.parse_args(["claim", "conjunction"]).type == CONJUNCTION + + +# --- test 6: serialization -------------------------------------------------- + + +def test_the_note_carries_the_conjuncts_and_round_trips(repo: Path): + """The published note says what each member earned, and the body satisfies + the pack's three-part round-trip criterion — newline discipline, key-superset + semantic equality, serialization idempotence — not byte equality.""" + s = _session(repo) + _run(s, repo, "pass") + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _claim(s, "lint-clean", "lint", 1) + _conjunction(s, "REL", "unit", "lint") + M.seal(s, repo) + + raw = _note_body(repo) + assert note_violations(raw, 0) == [] + + body = json.loads(raw[:-1]) + rel = _entry(body, "REL") + assert rel["claim"]["ctype"] == CONJUNCTION + assert rel["claim"]["conjuncts"] == ["unit", "lint"] + assert rel["claim"]["event_indices"] == [] + assert rel["conjunct_grades"] == [ + {"label": "unit", "grade": "tree-exact"}, + {"label": "lint", "grade": "tree-exact"}, + ] + # A conjunction binds no event, so the seal writes it no evidence block: + # "nothing to bind" and "bound to nothing" are different facts. + assert "evidence" not in rel + # Every OTHER claim keeps the keys it always had, and none of them gains a + # conjunct list it did not declare. + for entry in body["claims"]: + if entry["claim"]["label"] != "REL": + assert entry["claim"]["conjuncts"] == [] + assert "conjunct_grades" not in entry + + # The Claim survives the round trip through its own serialization. + reparsed = Claim.from_dict(rel["claim"]) + assert reparsed.ctype == CONJUNCTION + assert reparsed.conjuncts == ("unit", "lint") + + +def test_a_claim_dict_without_conjuncts_reads_as_carrying_none(): + """A v1/v2 claim entry lacks the key entirely; it reads with a default.""" + claim = Claim.from_dict( + {"ctype": "tests-pass", "label": "t", "event_indices": [0], "pathspecs": []} + ) + assert claim.conjuncts == () + assert claim.to_dict()["conjuncts"] == [] + + +def test_a_conjunct_label_carrying_a_secret_is_redacted_everywhere_it_is_published( + repo: Path, +): + """The conjunct list and the reason are exported operator-authored text. + + A token in a claim label blocked the seal and was then published verbatim on + the override — the defect the claim-field redaction pass closed. A + conjunction reintroduces the same shape twice: the conjunct NAME, and the + reason that names it. + """ + token = "ghp_" + "b3TA" * 9 + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", f"unit {token}", 0) + _conjunction(s, "REL", f"unit {token}") + + with pytest.raises(M.redact.SecretsBlocked): + M.seal(s, repo) + m = M.seal(s, repo, allow_secrets=True) + + published = _note_body(repo).decode("ascii") + assert token not in published, "a conjunct name was published verbatim" + rel = _entry(json.loads(m.to_json()), "REL") + mask = M.redact._mask("github-token") + assert rel["claim"]["conjuncts"] == [f"unit {mask}"] + assert rel["conjunct_grades"] == [{"label": f"unit {mask}", "grade": "tree-exact"}] + assert mask in rel["reason"] + fields = {f["field"] for f in rel["redaction"]["fields"]} + assert {"claim.conjuncts[0]", "reason"} <= fields + # The redaction is the same projection on both sides of the match, so the + # conjunction still resolves when verify regrades it from the note. + report = M.verify(s, repo) + assert next(r for r in report.results if r.claim.label == "REL").grade == "tree-exact" + + +# --- test 7: the version bump ---------------------------------------------- + + +def test_a_note_carrying_a_conjunction_is_version_3(repo: Path): + assert M.MANIFEST_VERSION == 3 + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "unit") + m = M.seal(s, repo) + + body = _note_body(repo) + assert json.loads(body[:-1])["version"] == 3 + assert m.version == 3 + # A v0.2 binary whose MANIFEST_VERSION is 3 accepts it. + assert M.Manifest.from_json(body[:-1]).version == 3 + + +def test_the_refuse_unknown_version_gate_rejects_version_4(repo: Path): + """P0.3's refuse-never-coerce rule, one version above this one.""" + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "unit") + M.seal(s, repo) + + body = json.loads(_note_body(repo)[:-1]) + body["version"] = 4 + too_new = json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii") + with pytest.raises(M.ManifestError) as exc: + M.Manifest.from_json(too_new) + assert "version 4" in str(exc.value) + assert "max 3" in str(exc.value) + + +# --- rendering -------------------------------------------------------------- + + +def test_the_surfaces_show_the_conjuncts_and_never_grade_an_unresolved_one( + repo: Path, monkeypatch +): + monkeypatch.setenv("NO_COLOR", "1") + s = _session(repo) + _run(s, repo, "pass") + _claim(s, "tests-pass", "unit", 0) + _conjunction(s, "REL", "unit", "nosuch") + M.seal(s, repo) + report = M.verify(s, repo) + + text = render.render_verdict(report) + assert "of unit: TREE-EXACT" in text + assert "of nosuch: UNRESOLVED" in text + + html = render.render_html(report) + assert "UNRESOLVED" in html + assert "nosuch" in html + # No new grade token, and an unresolved conjunct is not dressed as a grade. + assert "conjuncts" in html diff --git a/tests/test_detector_tiers.py b/tests/test_detector_tiers.py index e845358..e49585b 100644 --- a/tests/test_detector_tiers.py +++ b/tests/test_detector_tiers.py @@ -580,7 +580,12 @@ def test_a_blocked_seal_advances_nothing(repo: Path): Claim(ctype="tests-pass", label="second unit", event_indices=(0,), declared_at_index=0), ) - m = M.seal(s, repo) + # `reseal=True` because this seals the SAME commit a second time with a + # narrower window ([1,2) over a note that records [0,1)), which P4.1 refuses + # by default — that narrowing is the data-loss path it closes. What this test + # is about is a blocked seal advancing nothing, so the flag keeps its subject + # intact rather than the assertion below being softened. + m = M.seal(s, repo, reseal=True) assert [c["claim"]["label"] for c in m.claims] == ["second unit"] diff --git a/tests/test_projection_contract.py b/tests/test_projection_contract.py index f483939..f04660f 100644 --- a/tests/test_projection_contract.py +++ b/tests/test_projection_contract.py @@ -266,11 +266,18 @@ def broken(source, result): # list for a stale claim is recomputed against the working tree at verify time, # so it is NOT the note's redacted copy. Both halves are asserted, so a future # edit cannot quietly drop the caveat while keeping the reassuring half. +# +# The enumeration widened again at v3: a conjunction's conjunct names and every +# claim's `reason` joined the redacted set, because a conjunction's reason NAMES a +# conjunct and a conjunct name is operator-authored. The sentence has to keep +# enumerating what is actually covered — a list that goes stale in the +# understating direction is the same defect in a quieter key, and the assertion +# below is what forces the edit. _REMOVED_CLAIM = "secrets are redacted from every export" _REPLACEMENT = ( "Redaction covers the sealed note: every exported claim string — the " - "`argv_preview`, the label, the pathspecs and the changed paths — is " - "redacted, and the projection is declared in the manifest." + "`argv_preview`, the label, the pathspecs, the changed paths, the conjuncts " + "and the reason — is redacted, and the projection is declared in the manifest." ) _CAVEAT = ( "a stale claim's file list is recomputed against your working tree at " diff --git a/tests/test_redact_render.py b/tests/test_redact_render.py index 034f4f6..67ac14f 100644 --- a/tests/test_redact_render.py +++ b/tests/test_redact_render.py @@ -91,6 +91,18 @@ def env_counts(self): def all_verified(self): return all_verified + # Mirrors VerifyReport: a superseded result is record, not verdict, so + # both surfaces count the live set and report the retained count + # separately. Real GradeResults carry the flag, so the fake derives + # these the same way the real report does rather than declaring them. + @property + def live_results(self): + return [r for r in self.results if not r.is_superseded] + + @property + def superseded_count(self): + return len(self.results) - len(self.live_results) + # Mirrors VerifyReport: both human surfaces report how many verdicts # were checked against the recorded entry the seal named. @property diff --git a/tests/test_seal_publication.py b/tests/test_seal_publication.py index b3f72a0..16ac454 100644 --- a/tests/test_seal_publication.py +++ b/tests/test_seal_publication.py @@ -29,7 +29,19 @@ # Additive fields land here as a DELIBERATE one-line diff, never silently. # v2 adds no TOP-LEVEL key: the evidence binding lives inside each claim entry, # so the manifest's own key set is unchanged from v1. -_ADDITIVE_KEYS_BY_MANIFEST_VERSION = {1: frozenset(), 2: frozenset()} +# `claims_from`/`claims_to` are P4.1's: the claim window the seal covered. Both +# are additive with a reading default (0 / len(claims)), so a stored note of +# EITHER version legitimately lacks them, and both rows carry them. +# v3 is P4.2's `conjunction` claim type, which adds no top-level key either: the +# conjunct list and the per-conjunct grades live inside a claim entry. The row +# exists so the version this build writes has a DECLARED allowlist instead of a +# KeyError, and so a v3-only top-level field would still have to land here on +# purpose. +_ADDITIVE_KEYS_BY_MANIFEST_VERSION = { + 1: frozenset({"claims_from", "claims_to"}), + 2: frozenset({"claims_from", "claims_to"}), + 3: frozenset({"claims_from", "claims_to"}), +} def _session(repo: Path) -> Session: @@ -109,10 +121,18 @@ def _round_trip(body: bytes) -> M.Manifest: for key, value in source.items(): assert key in emitted, f"round-trip dropped manifest key {key!r}" assert emitted[key] == value, f"round-trip changed manifest key {key!r}" + # The allowlist is the BOUND on what may be missing, not an exact set: every + # body this helper sees was written by the seal under test, so it already + # carries every key `to_json` emits and the extra set is empty. Exact + # equality here asserted that a note this binary just published must LACK a + # key it declares, which stopped being true the moment an additive field + # landed. The leg that actually exercises a key-lacking note is + # tests/compat's leg 2 (over stored v1 bodies), plus the byte pin below. extra = frozenset(emitted) - frozenset(source) - assert extra == _ADDITIVE_KEYS_BY_MANIFEST_VERSION[source["version"]], ( - f"unexpected extra manifest keys {sorted(extra)} — a field was added " - "without updating the per-version allowlist" + allowed = _ADDITIVE_KEYS_BY_MANIFEST_VERSION[source["version"]] + assert extra <= allowed, ( + f"unexpected extra manifest keys {sorted(extra - allowed)} — a field was " + "added without updating the per-version allowlist" ) # (c) serialization is idempotent; this is where canonical_json is pinned. @@ -179,8 +199,12 @@ def boom(*args, **kwargs): monkeypatch.setattr(M, "_record_seal", boom) + # `reseal=True` because the prior note here is deliberately NOT a didrun + # manifest, so it records no claim window and P4.1 refuses to replace it + # blind. This test's subject is the rollback restoring the overwritten body + # byte-for-byte, which needs the seal to reach `_record_seal` at all. with pytest.raises(M.ManifestError) as exc: - M.seal(s, repo) + M.seal(s, repo, reseal=True) message = str(exc.value) assert "no watermark was recorded" in message @@ -266,6 +290,10 @@ def test_manifest_json_bytes_are_pinned(): is additive with a v1-reproducing default — which is why this is a one-line diff here plus a one-line diff in the compat allowlist, and not a MANIFEST_VERSION bump. + + P4.1 adds `claims_from`/`claims_to`, the claim window the seal covered, on + the same terms: reading defaults of 0 and `len(claims)`, so no stored note + changes meaning and this stays a deliberate diff rather than a version bump. """ m = M.Manifest( version=1, @@ -295,6 +323,7 @@ def test_manifest_json_bytes_are_pinned(): b'"declared_at_index":0,"event_indices":[0],"label":"t","pathspecs":[]},' b'"delta":[],"exit_code":0,"grade":"tree-exact","reason":"r",' b'"supporting_event_index":0}],' + b'"claims_from":0,"claims_to":1,' b'"commit":"0000000000000000000000000000000000000000",' b'"coverage":{"by_coverage":{"complete":1},"total_events":1},' b'"secrets":{},"secrets_override":false,' diff --git a/tests/test_unreadable_note_refuses.py b/tests/test_unreadable_note_refuses.py new file mode 100644 index 0000000..ed9b80c --- /dev/null +++ b/tests/test_unreadable_note_refuses.py @@ -0,0 +1,305 @@ +"""A note this binary cannot read is a graded refusal — never a crash, never a pass. + +The invariant is stated in `cli.cmd_verify` itself: "Evidence this binary cannot +read is a graded refusal, not a crash. 2, not --strict's 1: 'could not read the +manifest' is a different fact from 'the manifest graded badly'." It was not +holding. Measured on the pre-fix tree, over 22 malformed note shapes: + + - 11 shapes left an uncaught traceback (`KeyError`, `TypeError`, + `JSONDecodeError`, `ClaimError`) and exited **1** — colliding with the exit + code `--strict` uses for a note that graded badly, so a corrupt note was + indistinguishable from an honest failure by exit code alone; + - 4 shapes were **ACCEPTED**. `{"version": true}` verified green at exit 0, + because `bool` is an `int` subclass and `True > 3` is False. So did `2.5`, + `0` and `-1`. + +The accepted set is the one that matters. docs/COMPAT.md's "Version 3" section +rests the whole forward-compatibility story on the version guard — the +MANIFEST_VERSION bump exists so an older reader refuses a v3 note *on its +version* before it reaches the claim vocabulary it cannot parse. A guard a +malformed field walks past is not a guard. + +Test 1 is the gate: over every malformed shape, `didrun verify --strict` must +exit 2 with no traceback. Not "must not crash" — must not PASS either, and the +two failures need one test that cannot be satisfied by fixing only one of them. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + + +# --- helpers ----------------------------------------------------------------- + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _seal_one(repo: Path) -> M.Manifest: + """A repo carrying one honest sealed claim, so every mutation below starts + from a note that verifies green.""" + session = _session(repo) + run_wrapped([sys.executable, "-c", "print('ok')"], session, repo) + M.declare_claim( + session, + Claim( + ctype="tests-pass", + label="unit", + event_indices=(0,), + declared_at_index=0, + ), + ) + return M.seal(session, repo) + + +def _note_text(repo: Path, commit: str = "HEAD") -> str: + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", commit], + cwd=str(repo), + capture_output=True, + text=True, + ) + assert proc.returncode == 0, "the commit carries no note" + return proc.stdout + + +def _put_note(repo: Path, raw: str, commit: str = "HEAD") -> None: + subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "add", "-f", "-m", raw, commit], + cwd=str(repo), + capture_output=True, + text=True, + check=True, + ) + + +def _mutated(raw: str, fn) -> str: + d = json.loads(raw) + fn(d) + return json.dumps(d, sort_keys=True, separators=(",", ":")) + "\n" + + +def _set_ctype(d, value) -> None: + d["claims"][0]["claim"]["ctype"] = value + + +# Every shape measured as broken on the pre-fix tree, plus the four that were +# accepted. Keyed by name so a failure says which shape regressed. +# +# The `version` cases are split into two groups on purpose. A NON-INTEGER +# version is a malformed field (ManifestFormatError). An integer version above +# MANIFEST_VERSION is the documented forward-compat refusal and keeps its own +# message and its own exception — the distinction is load-bearing on the +# tree-fallback scan, where one is skipped and the other propagates. +MALFORMED = { + # -- the fail-OPEN set: these verified green before the fix --------------- + "version true": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", True)), + "version 2.5": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", 2.5)), + "version 0": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", 0)), + "version -1": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", -1)), + # -- the crash set -------------------------------------------------------- + "version string": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", "3")), + "version null": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", None)), + "version list": lambda raw: _mutated(raw, lambda d: d.__setitem__("version", [3])), + "version absent": lambda raw: _mutated(raw, lambda d: d.pop("version")), + "not json": lambda raw: "this is not json\n", + "json list": lambda raw: "[1,2,3]\n", + "json string": lambda raw: '"hello"\n', + "no commit": lambda raw: _mutated(raw, lambda d: d.pop("commit")), + "no tree": lambda raw: _mutated(raw, lambda d: d.pop("tree")), + "no claims": lambda raw: _mutated(raw, lambda d: d.pop("claims")), + "no coverage": lambda raw: _mutated(raw, lambda d: d.pop("coverage")), + "claims not a list": lambda raw: _mutated( + raw, lambda d: d.__setitem__("claims", "x") + ), + "claim entry not a dict": lambda raw: _mutated( + raw, lambda d: d["claims"].__setitem__(0, 7) + ), + "claim object missing": lambda raw: _mutated( + raw, lambda d: d["claims"][0].pop("claim") + ), + "ctype unknown": lambda raw: _mutated(raw, lambda d: _set_ctype(d, "nonsense")), + "ctype killed": lambda raw: _mutated(raw, lambda d: _set_ctype(d, "diff-exercised")), + "ctype null": lambda raw: _mutated(raw, lambda d: _set_ctype(d, None)), + "event_indices not iterable": lambda raw: _mutated( + raw, lambda d: d["claims"][0]["claim"].__setitem__("event_indices", 7) + ), +} + + +# --- test 1: the gate -------------------------------------------------------- + + +@pytest.mark.parametrize("shape", sorted(MALFORMED)) +def test_an_unreadable_note_refuses_with_exit_2_and_no_traceback( + repo: Path, capsys, shape: str +): + """THE criterion, over every malformed shape, through the real CLI. + + Three assertions, because the two pre-fix failure modes are opposites and a + one-sided test would pass on half a fix: + + 1. exit 2 — not 0 (accepted), and not 1 (which is `--strict`'s "graded + badly" and would make a corrupt note look like an honest failure); + 2. the refusal is on stderr, prefixed like every other refusal; + 3. nothing raised out of the command. + """ + _seal_one(repo) + _put_note(repo, MALFORMED[shape](_note_text(repo))) + capsys.readouterr() + + # Any exception escaping cli.main IS the traceback this test is about, so it + # is not caught here — pytest reporting it as an error is the correct + # outcome for a regression. + rc = cli.main(["--repo", str(repo), "verify", "--strict"]) + captured = capsys.readouterr() + + assert rc == 2, ( + f"{shape!r}: expected the graded refusal (2), got {rc}. " + f"0 means the malformed note was ACCEPTED; 1 means it was reported as a " + f"grading failure rather than an unreadable one." + ) + assert "didrun verify:" in captured.err + assert "Traceback" not in captured.err + + +def test_the_malformed_set_covers_both_pre_fix_failure_modes(): + """A guard on test 1's fixture: it must keep testing both directions. + + If the fail-open cases were ever dropped from MALFORMED, test 1 would still + pass over a build that only stopped crashing — and the accepted-green shapes + are the ones that could put a false verdict in front of a reviewer. + """ + assert {"version true", "version 2.5", "version 0", "version -1"} <= set(MALFORMED) + assert {"not json", "ctype unknown", "no commit"} <= set(MALFORMED) + + +# --- test 2: the fail-open shape, stated on its own --------------------------- + + +def test_a_boolean_version_is_not_version_one(repo: Path): + """The measured false green, isolated. + + `bool` is an `int` subclass: `True > 3` is False, so the pre-fix guard let + `{"version": true}` through and graded the note under v1 semantics. It + verified at exit 0. This is the same reading rule `_superseded_by` already + applied via `_is_index`, and the reason is identical — a field that arrived + from a file is not a field this process computed. + """ + _seal_one(repo) + body = json.loads(_note_text(repo)) + body["version"] = True + raw = json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii") + + with pytest.raises(M.ManifestFormatError) as exc: + M.Manifest.from_json(raw) + assert "True" in str(exc.value) + # And it must NOT be reported as the forward-compat refusal, which would + # tell an operator to upgrade over a note that is simply corrupt. + assert "upgrade didrun" not in str(exc.value) + + +# --- test 3: the two refusals stay distinguishable --------------------------- + + +def test_a_format_refusal_is_a_manifest_error_but_a_version_refusal_is_not_a_format_one( + repo: Path, +): + """The type split the tree-fallback scan depends on. + + `ManifestFormatError` subclasses `ManifestError` so the CLI's existing + handler catches it and exits 2. But a version this binary cannot read must + NOT be a format error: on the scan path a format error is skipped and + counted, and a too-new version propagates. Collapsing the two would make a + v4 note skippable, and the scan would report some older note's verdict as if + it were current. + """ + assert issubclass(M.ManifestFormatError, M.ManifestError) + + _seal_one(repo) + body = json.loads(_note_text(repo)) + body["version"] = M.MANIFEST_VERSION + 1 + too_new = json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii") + + with pytest.raises(M.ManifestError) as exc: + M.Manifest.from_json(too_new) + assert not isinstance(exc.value, M.ManifestFormatError) + assert "upgrade didrun" in str(exc.value) + + +# --- test 4: the scan path still skips rather than dying --------------------- + + +def test_the_tree_fallback_still_skips_a_corrupt_note_and_counts_it( + repo: Path, git +): + """Preserved behaviour, pinned because the fix could plausibly break it. + + A foreign or corrupt note under the same ref must not hide every good one — + it is skipped and COUNTED. Before the fix that happened by accident, because + the scan caught bare `Exception` and `json` raised something that was not a + `ManifestError`. Now `ManifestFormatError` IS a `ManifestError`, so the skip + clause has to come first by type. If it did not, this verify would raise + instead of resolving the good note. + """ + _seal_one(repo) + good_tree = M.gitplumbing.commit_tree(repo, M.gitplumbing.head_commit(repo)) + + # A second commit carrying an unreadable note under the same ref. + (repo / "other.py").write_text("x = 1\n") + git(repo, "add", "other.py") + git(repo, "commit", "-qm", "other") + _put_note(repo, "not a manifest at all\n", "HEAD") + + # Ask for a tree no note carries, so the scan is forced to walk PAST the + # corrupt note instead of stopping at the good one. `git notes list` order is + # by object id, so a test that relied on the corrupt note being reached first + # would pass or fail on a hash — this reaches it every time. + m, resolved_by, skipped = M._resolve_manifest(repo, None, "f" * 40) + assert m is None + assert resolved_by == "none" + assert skipped == 1, "the corrupt note was not skipped and counted" + + # And the good note is still resolvable with the corrupt one in the ref. + m, resolved_by, _ = M._resolve_manifest(repo, None, good_tree) + assert m is not None, "the corrupt note hid the good one" + assert resolved_by == "tree-fallback" + + +# --- test 5: no false refusal of an honest note ------------------------------ + + +@pytest.mark.parametrize("version", [1, 2, M.MANIFEST_VERSION]) +def test_every_version_this_binary_writes_or_must_read_still_parses( + repo: Path, version: int +): + """The fix must refuse nothing that was legitimately published. + + Every note in the world carries an integer version of 1, 2 or 3, and the + compat corpus replays all three. A guard that tightened onto them would be a + worse defect than the one it closed. + """ + _seal_one(repo) + body = json.loads(_note_text(repo)) + body["version"] = version + raw = json.dumps(body, sort_keys=True, separators=(",", ":")).encode("ascii") + assert M.Manifest.from_json(raw).version == version + + +def test_an_honest_note_still_verifies_green(repo: Path, capsys): + """The end-to-end control for test 1: the same path, unmutated, exits 0.""" + _seal_one(repo) + capsys.readouterr() + assert cli.main(["--repo", str(repo), "verify", "--strict"]) == 0 From 49a8b746c83e896661a06281b9b8dd44848bad73 Mon Sep 17 00:00:00 2001 From: Drew Date: Wed, 29 Jul 2026 19:41:58 -0700 Subject: [PATCH 5/8] claim: bind the command that just ran, and cite out-of-band authority MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes to what a claim can honestly say, and one remediation pass over the surface the third one added. `didrun claim` with no `--event` used to bind to event 0 whatever event 0 was. In a session that ran a passing command and then a failing one, that produced a fully green sealed receipt over a suite which had just exited non-zero. It now binds the command that actually just ran, refuses when that command failed, and names the index, the exit code and `--event N` in the refusal, so the operator who really did mean an earlier event has a way to say so. A claim that lands `scope-exact` now says which pathspec would have made it tree-exact, naming the directories and files that fell outside what was declared. The hint is absent when the scope is already exact, so the reason line for an exact claim is unchanged. `didrun authorize` records that a file with a given SHA-256 was cited as the authority for one exceptional transition — `allow-secrets` or `reseal`, with no scope for an ordinary seal, because an authorisation on the happy path becomes a rubber stamp. It expires after a stated number of seals, and `--require-authority` refuses a bare override that no live citation covers. It grades `cited` and it buys exactly one property: retroactive fabrication becomes detectable, because the digest sits in a note sealed earlier. It is not a signature, it is not approval, and an agent can author its own authorization artifact in one line — the limitation sentence is printed beside every citation from a source constant so a note cannot edit it. The remediation. Publishing a new block of operator text re-opened a defect this repo had already closed once for claim labels: `principal` is `--principal` or `$USER`, it was published verbatim, and the pass that redacts authority fields never touched it. A token there refused the seal via the whole-manifest backstop under a message promising a redacted export, and then `--allow-secrets` wrote it into the git note unchanged with `redaction.fields` empty. Worse in the quiet case: one high-entropy value passed as both label and principal came out scrubbed in the first field and raw in the second, in the same JSON object, with nothing overridden. `principal` now goes through the same `redact_field` pass as the label and the cited path; the enumeration of which fields carry outside text is checked for completeness by a test instead of trusted, and a field neither table classifies is dropped from the note rather than published unscanned. The whole-manifest backstop that had been carrying this alone is now exercised too — removing the authority block from the scanned bytes used to break no test. Two of the six fields in the terminal's authority block were interpolated without `_sanitize`, because every use site spelled the call out by hand. `artifact_bytes` and `expires_after_seals` are numbers in a note didrun wrote and arbitrary strings in a note someone else edited, and a newline in either one fabricated whole verdict rows above the real table — rows the renderer never computed, carrying the words this vocabulary forbids, plus raw escape bytes into the operator's terminal. A note crosses machines over `git fetch`, so every field of it is outside data; the block now reads all of them through one sanitizing accessor, the way the HTML path already funnelled everything through one `esc`. The `seal` and `authorize` echoes had the same hole from the ledger side and are sanitized too — redaction is not sanitization, and a scrubbed label still carries whatever control characters surrounded the secret. `principal_basis` joins `grade` and `limitation` as a field pinned to its source constant at both bind and render. It qualifies the principal, so a note supplying "verified-ssh-signature" printed a claim of verified identity two lines above the sentence saying didrun cannot tell who wrote it. Left open and written down in docs/COMPAT.md rather than fixed quietly: a note's authority `scope` is still rendered as written and not re-validated against the scope vocabulary. It grants nothing — `--require-authority` gates against the local ledger, never against a fetched note — but a hand-edited note can misname the transition it claims to cover, and choosing what to render for an unrecognised scope deserves the same deliberate treatment the grade vocabulary got. --- docs/COMPAT.md | 37 ++ docs/TRUST_MODEL.md | 83 ++- src/didrun/claims.py | 45 +- src/didrun/cli.py | 173 +++++- src/didrun/manifest.py | 447 ++++++++++++++ src/didrun/render.py | 123 ++++ tests/compat/synthetic.py | 8 +- tests/compat/test_corpus_replay.py | 12 +- tests/test_claim_binding.py | 270 +++++++++ tests/test_grading_honesty.py | 179 +++++- tests/test_operator_authority.py | 939 +++++++++++++++++++++++++++++ tests/test_redact_render.py | 7 +- tests/test_seal_publication.py | 9 +- 13 files changed, 2306 insertions(+), 26 deletions(-) create mode 100644 tests/test_claim_binding.py create mode 100644 tests/test_operator_authority.py diff --git a/docs/COMPAT.md b/docs/COMPAT.md index 5614228..93da733 100644 --- a/docs/COMPAT.md +++ b/docs/COMPAT.md @@ -571,3 +571,40 @@ overclaim. git history survives there as a loose object until the ledger directory is removed by hand. Treat the ledger as secret-bearing: it is gitignored by default, and it should not be committed, shared, or attached to an issue. + +## Known, not closed: a note's authority block is descriptive, not authenticated + +Found while closing the authority redaction and rendering defects. Both items below are +statements about what a *note* can say, not about what it can make didrun do — the note is +the artifact that crosses machines (`git fetch`), and `docs/TRUST_MODEL.md` already states +that citation is "not tamper resistance — whoever can rewrite a note can rewrite the digest +in it". These are recorded because "the note is not authenticated" is easy to state and easy +to forget field by field. + +1. **A note's authority `scope` is rendered as written, and is not re-validated against the + scope vocabulary.** `authorize` refuses any scope outside `allow-secrets` / `reseal`, and + `_valid_authority` drops a ledger record whose scope is unknown — but a note that has been + edited by hand can carry `"scope": "everything"` and `didrun verify` will print + `CITED authority for scope everything`. It grants nothing: `--require-authority` gates the + seal against the local ledger, never against a fetched note, so a forged scope changes only + the description a reader sees. It is left open because the honest fix is a decision about + what to render for an unrecognized scope — dropping the block hides that a citation was + claimed, and printing it plainly is what it does now — and that decision wants the same + treatment the grade vocabulary got, not a quiet clamp. + + The three fields that state *what a citation is worth* are NOT in this position and are + pinned to source constants at both bind and render: `grade` (always `cited`), `limitation` + (the full sentence) and `principal_basis` (always `self-asserted`). A note supplying + `principal_basis: "verified-ssh-signature"` was the reason the last of those three was + pinned — it printed `principal (verified-ssh-signature)` two lines above the + sentence saying didrun cannot tell who wrote it, in one of the exact words the trust model + forbids. Verified by test, both surfaces. + +2. **Redaction is not sanitization, and the two run at different times.** A label is scrubbed + for secrets before publication; control characters around the secret survive that pass + untouched, because a redactor that also rewrote whitespace could not declare faithful + spans. Every *terminal* surface therefore sanitizes at the point it prints — the verify + authority block, and the `seal` / `authorize` echoes — and the HTML path escapes through + one helper. Anything reading a note or `authority.jsonl` programmatically gets the raw + strings and must do its own neutralizing; that is the same contract the recorded argv and + output already carry. diff --git a/docs/TRUST_MODEL.md b/docs/TRUST_MODEL.md index 5b7597d..ef473a8 100644 --- a/docs/TRUST_MODEL.md +++ b/docs/TRUST_MODEL.md @@ -54,6 +54,68 @@ later. It is also a genuine adoption bet, because voluntary developer commit signing is rare in practice. Until it lands, **didrun's guarantees hold only among parties who already share a baseline of trust.** +## Operator authority: what `cited` attests + +`didrun authorize --artifact --scope --label ""` records that a +file with a particular SHA-256 was cited as the authority for one exceptional +transition. `seal` binds the live citations into the manifest, grades each one +`cited`, and every surface that shows one prints this sentence beside it: + +> cited authority: recorded that a file with this digest was cited for this scope. +> NOT a signature and NOT approval — didrun cannot tell who wrote it. + +**What it attests.** A file with this digest existed at this path and was cited for +this scope, at this point in an append-only record. The digest, the scope, the label +and the self-asserted principal are in the note; the artifact's bytes never are. + +**What it does not attest — read this part.** It is not evidence that a human wrote, +read, or approved anything. **An agent can author its own authorization artifact in +one line**, cite it, and seal. There is no identity here: `principal` is whatever +`$USER` said, recorded as `self-asserted`, and didrun authenticates nobody. `cited` +must never be rendered or reported as "signed", "approved", "verified" or +"authorized by" — the same rule that keeps `tree-exact` from rendering as "verified +true". + +**The one property it buys.** Retroactive fabrication becomes detectable: because the +digest sits in a note sealed earlier, an artifact edited or written after the fact no +longer matches what that earlier note recorded. That is a real, narrow gain, and it is +the whole gain. It converts an authority from prose into a fixed, diffable, +digest-bound object; it does not make the object trustworthy. It is also not tamper +resistance — whoever can rewrite a note can rewrite the digest in it, exactly as they +can rewrite a grade. + +**Why it is shaped this way.** `--allow-secrets` was the shipped instance of the +problem: a bare flag stored as a bare boolean, with no principal, no basis and no +artifact, so a reader could not tell an authorized override from an agent that simply +passed the flag. didrun's own thesis is that a claim with no witnessed event grades +`unknown`; an out-of-band authorization is exactly that shape, and the tool used to +grade it implicitly true. So authorities are: + +- **scoped** — only `allow-secrets` and `reseal`, the exceptional transitions. There + is deliberately no scope for an ordinary seal, and none will be added. +- **expiring** — a TTL counted in seals (default 1, minimum 1, and no value meaning + "never"). Seals rather than wall time, because a clock in the trust path is not + allowed here. +- **never required by default** — `seal --require-authority` is opt-in. Any + authorization demanded on the happy path becomes a rubber stamp within a week, + which is precisely how the information content of `--allow-secrets` reached zero. + +**The honest anchor for a stronger tier is a key the agent cannot read.** A +passphrase-less key in `~/.ssh` that the agent can read is theatre; a hardware-backed +key with touch-to-sign (`ssh-keygen -t ecdsa-sk`, or a Secure Enclave key) is the real +anchor, because the human's finger is the thing an agent cannot fake. Two further +tiers are designed and **not built**: + +- **Tier 1 — reuse the key the developer already has.** `git verify-tag` / + `git verify-commit` under `gpg.format=ssh` and `~/.ssh/allowed_signers`, recording + the signer identity and the verification output. Zero new key management, no crypto + inside didrun. **Not built.** +- **Tier 2 — org identity, roles, revocation, trust roots.** **Not built**, and out of + scope for a local single-user tool. + +Both are consequences of the deferral in the section above, and neither is claimed +anywhere in didrun's output. + ## The ledger is secret-bearing The ledger records everything a session printed — including anything an agent @@ -70,13 +132,20 @@ exported bundle as **secret-bearing**: claim labels and delta paths included. Override with `--allow-secrets`, which is logged into the manifest; the exported artifact is redacted either way. - **What blocks and what is redacted are the same fields.** Every string in a - published claim entry that carries operator- or repository-authored text (the - argv preview, the label, the pathspecs, the changed paths) is replaced by a - marker before the note is written, and the spans are declared in - `redaction.applied` / `redaction.fields`. This is stated because it was once - false in the worst direction: a token in a claim label refused the seal and - was then published verbatim on the override, under a refusal message that had - already promised a redacted artifact. Blocking is not redacting. + published entry that carries operator- or repository-authored text is replaced + by a marker before the note is written, and the spans are declared in + `redaction.applied` / `redaction.fields`. For a claim that is the argv preview, + the label, the pathspecs and the changed paths; for a cited authority it is the + label, the cited path **and the principal**. This is stated because it was once + false in the worst direction, twice: a token in a claim label refused the seal + and was then published verbatim on the override, under a refusal message that + had already promised a redacted artifact — and when authorities were added, + `principal` reproduced it exactly, having been scanned by the whole-manifest + backstop but left out of the pass that redacts. Blocking is not redacting. An + enumeration of "the fields that carry outside text" also goes stale the moment + a field is added, so the authority enumeration is checked for completeness by a + test rather than trusted, and a field it does not classify is dropped from the + note instead of published. - Everything else is **reported loudly and does not stop the seal**: every `notice`-tier finding, and any `block`-tier finding confined to the recorded output blobs, which stay in a gitignored local ledger and are never published. diff --git a/src/didrun/claims.py b/src/didrun/claims.py index 3d455f1..fa3cda5 100644 --- a/src/didrun/claims.py +++ b/src/didrun/claims.py @@ -500,10 +500,19 @@ def grade( exit_code=ev.exit_code, ) # Out-of-scope changes exist — display them; cannot claim scope-exact. + # The reason names the widening set as well as the count: the exact + # prefixes that would have made this scope-exact are already sitting in + # `delta`, and reporting only how MANY paths missed made an operator + # re-derive by hand something the tool had measured. The count prefix is + # unchanged and the set is appended after an em dash, so a consumer + # matching the old prefix still matches (docs/COMPAT.md). return GradeResult( claim, GRADE_STALE, - reason=f"{len(out_of_scope)} change(s) outside declared pathspecs", + reason=( + f"{len(out_of_scope)} change(s) outside declared pathspecs" + f" — {_widening_hint(c.path for c in out_of_scope)}" + ), delta=delta, supporting_event_index=idx, exit_code=ev.exit_code, @@ -520,6 +529,40 @@ def grade( ) +# How many prefixes the widening hint names before it summarises the rest. A +# hint longer than this stops being readable and starts being the delta again, +# which the caller already has. +WIDENING_HINT_CAP = 5 + + +def _widening_hint(paths) -> str: + """The declaration that would have brought ``paths`` into scope, in words. + + ``_within`` is a path-prefix test, so the smallest prefix covering a path is + its parent directory — that is the whole derivation. Deduplicated and sorted + for determinism, capped at ``WIDENING_HINT_CAP`` with the remainder counted + rather than dropped silently. A top-level file has no parent directory but + the repository root, and declaring the root would put EVERY path in scope, so + it contributes itself: the smallest prefix that covers it and nothing else. + + This is not relevance inference (module docstring: *set arithmetic only*). + It changes no grade, and it asserts nothing about whether those paths belong + to the claim — only what widening the declaration would have to say. + """ + prefixes = set() + for path in paths: + stripped = path.strip("/") + head, sep, _tail = stripped.rpartition("/") + prefixes.add(head + "/" if sep else stripped) + ordered = sorted(prefixes) + shown = ordered[:WIDENING_HINT_CAP] + hint = ", ".join(f"--path {p}" for p in shown) + elided = len(ordered) - len(shown) + if elided: + hint += f" (+{elided} more)" + return f"add {hint} to make this scope-exact" + + def _within(path: str, pathspecs: tuple[str, ...]) -> bool: """True if ``path`` is under any declared pathspec (prefix match). diff --git a/src/didrun/cli.py b/src/didrun/cli.py index 4aa4124..d1b03d3 100644 --- a/src/didrun/cli.py +++ b/src/didrun/cli.py @@ -1,6 +1,6 @@ """didrun command-line interface. -Five commands: run, claim, seal, verify, show. +Six commands: run, claim, authorize, seal, verify, show. `log` is subsumed by `show`; `report` is a `--html` flag on show/verify. The CLI is a thin shell over the library; all behavior lives in the modules so @@ -95,8 +95,8 @@ def cmd_claim(args) -> int: indices: tuple = () if args.type == CONJUNCTION: # A conjunction binds no event — it conjoins claims that carry their own - # witnessed events — so the "no successful event to bind" refusal below - # is not its refusal. `--event` and `--path` are refused rather than + # witnessed events — so the "most recent event did not exit 0" refusal + # below is not its refusal. `--event` and `--path` are refused rather than # silently ignored: an operator who passed one meant something by it. if args.event is not None: print( @@ -121,16 +121,45 @@ def cmd_claim(args) -> int: file=sys.stderr, ) return 2 - # Default: bind to the most recent successful event. + # Default: bind to the LAST event, never to the last *successful* one. + # + # Searching backwards for a success is the failure this tool exists to + # prevent, reachable through the documented happy path: run the tests + # (pass), edit, run them again (fail), `didrun claim tests-pass` — the + # claim bound the older passing event, `declared_at_index` was still the + # last index so the retroactive-binding rule did not catch it, and if the + # tree had not moved it graded tree-exact. The only signal was the index + # in a success line nobody reads. Refuse instead: an operator who really + # means an earlier event says so with --event N, which is deliberate and + # is warned about below. if args.event is not None: indices = (args.event,) + gap = len(events) - 1 - args.event + if gap > 0: + print( + f"didrun claim: binding to event {args.event}, {gap} event(s) " + f"before the most recent (index {len(events) - 1})", + file=sys.stderr, + ) else: - indices = tuple( - i for i, ev in enumerate(events) if ev.exit_code == 0 - )[-1:] # last success - if not indices: - print("didrun claim: no successful event to bind (last command did not exit 0)", file=sys.stderr) - return 2 + last = len(events) - 1 + recent = events[last] + if recent.exit_code != 0: + witness = ( + f"exited {recent.exit_code}" + if recent.exit_code is not None + else "recorded no exit code (interrupted, or capture that " + "witnesses no exit)" + ) + print( + f"didrun claim: the most recent event (index {last}) " + f"{witness} — refusing to bind this claim to an earlier " + f"event that passed. Pass --event N to bind an earlier " + f"event deliberately.", + file=sys.stderr, + ) + return 2 + indices = (last,) try: claim = Claim( ctype=args.type, @@ -155,6 +184,52 @@ def cmd_claim(args) -> int: return 0 +def cmd_authorize(args) -> int: + """Record that a cited artifact is the authority for one scope. + + Prints the digest and, verbatim, the limitation constant. The limitation is + printed here and not only in the docs because this command's output is where + an operator forms their idea of what the record means, and what it means is + much less than "approved". + """ + repo = Path(args.repo or os.getcwd()) + session = _session(repo) + try: + record = _manifest.declare_authority( + session, + Path(args.artifact), + scope=args.scope, + label=args.label, + principal=args.principal, + expires_after_seals=args.expires_after_seals, + ) + except _manifest.ManifestError as exc: + print(f"didrun authorize: {exc}", file=sys.stderr) + return 2 + except OSError as exc: + print(f"didrun authorize: cannot read the cited artifact: {exc}", file=sys.stderr) + return 2 + # Sanitized, like every other terminal surface that prints a string this + # process did not compute: the label, the principal and the path are argv. + text = render.sanitize_text + print( + f"recorded {_manifest.AUTHORITY_GRADE} authority for scope " + f"{text(record['scope'])}: {text(record['label'])}" + ) + print( + f" artifact {text(record['artifact_path'])} " + f"sha256 {text(record['artifact_sha256'])} " + f"{text(record['artifact_bytes'])} bytes" + ) + print( + f" principal {text(record['principal']) or '(unnamed)'} " + f"({_manifest.AUTHORITY_PRINCIPAL_BASIS}) · " + f"expires after {text(record['expires_after_seals'])} seal(s)" + ) + print(f" {_manifest.AUTHORITY_LIMITATION}") + return 0 + + def cmd_seal(args) -> int: repo = Path(args.repo or os.getcwd()) session = _session(repo) @@ -166,6 +241,7 @@ def cmd_seal(args) -> int: allow_secrets=args.allow_secrets, bundle_path=Path(args.bundle) if args.bundle else None, reseal=args.reseal, + require_authority=args.require_authority, ) except _manifest.redact.SecretsBlocked as exc: print(f"didrun seal: {exc}", file=sys.stderr) @@ -193,6 +269,21 @@ def cmd_seal(args) -> int: + f"{blocking + noticed} findings ({blocking} block / {noticed} notice)" + (" [--allow-secrets]" if m.secrets_override else "") ) + # What authorised the exceptional transition, if anything did — printed at the + # moment the citation is published, with the limitation attached. An ordinary + # seal cites nothing and prints nothing extra. + # Sanitized for the same reason the verify block is: these strings came from + # `authority.jsonl`, and redaction is not sanitization — a scrubbed label + # still carries whatever control characters were written around the secret. + for entry in m.authority: + print( + f" {_manifest.AUTHORITY_GRADE} authority for scope " + f"{render.sanitize_text(entry.get('scope', ''))}: " + f"{render.sanitize_text(entry.get('label', ''))} " + f"sha256 {render.sanitize_text(entry.get('artifact_sha256', ''))}" + ) + if m.authority: + print(f" {_manifest.AUTHORITY_LIMITATION}") return 0 @@ -362,7 +453,15 @@ def build_parser() -> argparse.ArgumentParser: # would keep refusing a type the library had accepted for a release. pc.add_argument("type", choices=list(CLAIM_TYPES)) pc.add_argument("--label", help="human label (default: the type)") - pc.add_argument("--event", type=int, help="bind to a specific event index") + pc.add_argument( + "--event", + type=int, + help=( + "bind to a specific event index, warning when it is not the most " + "recent. Default: the most recent event, and a claim is refused " + "rather than bound to an earlier one when that event did not exit 0" + ), + ) pc.add_argument("--path", action="append", help="declare a pathspec for scope-exact grading") pc.add_argument( "--of", @@ -376,6 +475,48 @@ def build_parser() -> argparse.ArgumentParser: ) pc.set_defaults(func=cmd_claim) + pa = sub.add_parser( + "authorize", + help=( + "record a cited authorization artifact for one exceptional scope " + "(stores the digest, never the bytes; NOT a signature)" + ), + ) + pa.add_argument( + "--artifact", + required=True, + help="path to the authorization artifact; its sha256 is recorded, its bytes are not", + ) + pa.add_argument( + "--scope", + required=True, + choices=list(_manifest.AUTHORITY_SCOPES), + help=( + "the exceptional transition this authority covers. There is no scope " + "for an ordinary seal: an authorisation on the happy path becomes a " + "rubber stamp" + ), + ) + pa.add_argument("--label", required=True, help="what is being authorised, in words") + pa.add_argument( + "--principal", + help=( + "who is citing it (default: $USER). Recorded as SELF-ASSERTED — " + "didrun authenticates nobody" + ), + ) + pa.add_argument( + "--expires-after-seals", + type=int, + metavar="N", + help=( + f"how many seals the citation covers, starting with the next one " + f"(default: {_manifest.AUTHORITY_DEFAULT_TTL}). There is no value " + f"meaning 'never' — a standing grant is what this replaces" + ), + ) + pa.set_defaults(func=cmd_authorize) + ps = sub.add_parser("seal", help="compile and attach a commit-bound manifest") ps.add_argument("--commit", help="commit to bind (default: HEAD)") ps.add_argument("--allow-secrets", action="store_true", help="export despite secret findings (redacted; logged)") @@ -389,6 +530,16 @@ def build_parser() -> argparse.ArgumentParser: "no note at all. Without it, a narrowing re-seal is refused" ), ) + ps.add_argument( + "--require-authority", + action="store_true", + help=( + "refuse --allow-secrets or --reseal unless a live cited authority " + "covers that scope (see `didrun authorize`). Default off: an " + "authorisation required on every seal becomes a rubber stamp. The " + "authority it demands is a citation, not an approval" + ), + ) ps.set_defaults(func=cmd_seal) pv = sub.add_parser("verify", help="verify a commit's claims against recorded evidence") diff --git a/src/didrun/manifest.py b/src/didrun/manifest.py index 50346e5..11587a6 100644 --- a/src/didrun/manifest.py +++ b/src/didrun/manifest.py @@ -25,7 +25,9 @@ from __future__ import annotations +import hashlib import json +import os import subprocess import sys from dataclasses import dataclass, field @@ -110,6 +112,13 @@ class Manifest: # (docs/COMPAT.md). claims_from: int = 0 claims_to: Optional[int] = None + # The authorisation artifacts this seal cited for its exceptional + # transitions, each graded `cited` and each carrying AUTHORITY_LIMITATION. + # A MANIFEST field and never an Event field: a reference to an external + # artifact inside the chain preimage would make every future rotation of + # that artifact a schema break (docs/COMPAT.md). Additive, defaults to [], + # read with .get(): no MANIFEST_VERSION bump. + authority: list = field(default_factory=list) def __post_init__(self) -> None: # The reading default for `claims_to`, applied once so every consumer @@ -133,6 +142,7 @@ def to_json(self) -> bytes: "secrets": self.secrets, "claims_from": self.claims_from, "claims_to": self.claims_to, + "authority": self.authority, } ) @@ -198,6 +208,13 @@ def from_json(cls, data: bytes) -> "Manifest": raise ManifestFormatError( f"manifest `claims` is a {type(d['claims']).__name__}, not a list" ) + # A note published before operator authority existed carries no key, and + # "no authority was cited" is exactly what [] says. A non-list is read as + # [] for the same reason: an authority block is record, never verdict, so + # a malformed one must not raise on a path that reads the rest fine. + authority = d.get("authority", []) + if not isinstance(authority, list): + authority = [] return cls( version=version, commit=d["commit"], @@ -208,6 +225,7 @@ def from_json(cls, data: bytes) -> "Manifest": secrets=d.get("secrets", {}), claims_from=d.get("claims_from", 0), claims_to=d.get("claims_to"), + authority=authority, ) @@ -315,6 +333,7 @@ def seal( write_notes: bool = True, bundle_path: Optional[Path] = None, reseal: bool = False, + require_authority: bool = False, ) -> Manifest: """Compile, redact, and attach a manifest for ``commitish``. @@ -330,6 +349,11 @@ def seal( window this seal would NARROW, unless ``reseal`` is True. That overwrite was silent and lossy: `git notes add -f` force-replaces, and a seal scoped to `all_claims[watermark:]` replaced a wide record with a narrow one. + + ``require_authority`` refuses an exceptional transition — ``allow_secrets`` + or ``reseal`` — that no live `cited` authority covers. Default off: an + authorisation on the happy path becomes a rubber stamp, so an ordinary seal + demands nothing and binds nothing. """ repo = Path(repo) commit = gitplumbing.head_commit(repo) if commitish == "HEAD" else _rev(repo, commitish) @@ -339,6 +363,22 @@ def seal( if tree is None: raise ManifestError(f"cannot resolve tree for commit {commit}") + # Authorities first, and the refusal with them: it is a statement about the + # INVOCATION, so it must cost nothing and write nothing when it fires. The + # scopes are the exceptional transitions this seal actually performs — an + # ordinary seal performs none, binds none, and gains no line anywhere. + authority_scopes = tuple( + scope + for scope, requested in ( + (AUTHORITY_ALLOW_SECRETS, allow_secrets), + (AUTHORITY_RESEAL, reseal), + ) + if requested + ) + bound_authority = _bind_authorities(session, authority_scopes, _seal_count(session)) + if require_authority: + _require_authority(authority_scopes, bound_authority) + # Entries, not just events: the seal records WHICH recorded entry backed # each claim, and an entry's chain hash is the only handle on that which # survives the ledger being archived, rebuilt, or replaced. @@ -412,6 +452,13 @@ def seal( claims_payload = [rc.payload for rc in redacted] field_findings = [f for rc in redacted for f in rc.findings] field_bytes = sum(rc.bytes_scanned for rc in redacted) + # The authority block is published, so it is scanned and redacted by the same + # pass, on the same rule as the claim fields above. + authority_payload, authority_findings, authority_bytes = _redact_authorities( + bound_authority + ) + field_findings.extend(authority_findings) + field_bytes += authority_bytes coverage = _coverage_statement(session) export_bytes = canonical_json( { @@ -422,6 +469,7 @@ def seal( "coverage": coverage, "claims_from": watermark, "claims_to": len(all_claims), + "authority": authority_payload, } ) scan = _scan_for_secrets( @@ -445,6 +493,7 @@ def seal( secrets=scan.to_dict(overridden), claims_from=watermark, claims_to=len(all_claims), + authority=authority_payload, ) # Publication and the watermark are one atomic pair. A published note with @@ -508,6 +557,12 @@ class VerifyReport: # units red on routine noise. The flag is recorded on the report so a reader # can tell which of the two verdicts they are looking at. require_env_match: bool = False + # The authorities the note says this seal cited, as published. Record, never + # verdict: an authority does not enter `worst_status` or `all_verified`, and + # it cannot make a red note green. `cited` attests that a file with that + # digest was cited — not that anyone approved anything — which is why it + # grades nothing and only renders. + authority: list = field(default_factory=list) @property def chain_faulted(self) -> bool: @@ -756,6 +811,10 @@ def _graded(position: int, stored: dict, claim: Claim, siblings=None) -> GradeRe chain_broken_index=chain_index, chain_reason=chain_reason, require_env_match=require_env_match, + # Entries that are not objects are dropped: the block is rendered, and a + # renderer walking a string as if it were a record is how a malformed + # note becomes a traceback out of `verify`. + authority=[a for a in manifest.authority if isinstance(a, dict)], ) @@ -1087,6 +1146,23 @@ def _last_seal_watermark(session: Session) -> int: return watermark +def _seal_count(session: Session) -> int: + """How many seals this ledger has recorded — the authority expiry clock. + + Seals, not wall time. A time-based TTL would put a clock in the trust path, + which is a hard invariant this package does not break, and it would make an + authority's liveness depend on how long a build took. A seal is the + transition an authority is cited for, so counting them is the measure that + means something. + """ + path = _seals_path(session) + if not path.exists(): + return 0 + return sum( + 1 for line in path.read_text(encoding="ascii").splitlines() if line.strip() + ) + + def _record_seal(session: Session, commit: str, tree: str, claims_watermark: int) -> None: # Private-append: these carry operator-authored labels and commit ids into # the secret-bearing ledger directory, and a plain open() would create them @@ -1129,6 +1205,377 @@ def _load_claims(session: Session) -> list[Claim]: return claims +# --- operator authority ------------------------------------------------------ +# +# didrun's thesis is that a claim with no witnessed event grades `unknown`. An +# out-of-band human authorisation is structurally a claim with no witnessed +# event, and `--allow-secrets` is the shipped instance: a bare argv flag stored +# as a bare boolean, with no principal, no basis, and no way for a reader to +# tell an authorised override from an agent that simply passed the flag. +# +# What this converts that into is narrow and is stated on every surface: "someone +# passed a flag citing an artifact with this digest, at this point in an +# append-only record". It is NOT evidence that a human approved anything — an +# agent can author its own authorisation artifact in one line. Signing is +# deferred (docs/TRUST_MODEL.md) and this does not close that gap; the honest +# anchor for a stronger tier is a key the agent cannot read. + +# The scopes an authority may be cited for: the exceptional transitions, and only +# those. There is deliberately no scope for an ordinary seal — an authorisation +# required on the happy path becomes a rubber stamp within a week, which is +# exactly how `--allow-secrets`'s information content went to zero. +AUTHORITY_ALLOW_SECRETS = "allow-secrets" +AUTHORITY_RESEAL = "reseal" +AUTHORITY_SCOPES = (AUTHORITY_ALLOW_SECRETS, AUTHORITY_RESEAL) + +# The grade a bound authority carries, in the same vocabulary as `tree-exact` and +# under the same rule: reported verbatim, never upgraded. `cited` never renders as +# "signed", "approved", "verified" or "authorized by". +AUTHORITY_GRADE = "cited" + +# How the principal was established. One value, because there is exactly one +# mechanism: didrun read $USER, or the operator typed a name. Recorded as a field +# rather than left implicit so a reader cannot mistake it for an authenticated +# identity. +AUTHORITY_PRINCIPAL_BASIS = "self-asserted" + +# Default TTL in seals when the operator names none. One, not unbounded: a +# standing grant is the shape `--allow-secrets` already had, and re-earning the +# citation per transition is the only thing that keeps it from routine-ising. +AUTHORITY_DEFAULT_TTL = 1 + +# The feature's honesty, as a constant that ships in the source and is printed on +# every surface that shows an authority — the terminal verdict, the HTML report, +# `authorize`'s own output, and the note itself. It lives here rather than only in +# the docs because a reader of a note or a verdict is not reading the docs, and +# `"grade": "cited"` with nothing beside it would be read as approval. +AUTHORITY_LIMITATION = ( + "cited authority: recorded that a file with this digest was cited for this " + "scope. NOT a signature and NOT approval — didrun cannot tell who wrote it." +) + +_DIGEST_CHUNK = 1 << 20 + + +def _authority_path(session: Session) -> Path: + return session.root / "authority.jsonl" + + +def _artifact_digest(path: Path) -> tuple: + """(sha256 hex, byte count) for the cited artifact. Never its bytes. + + Chunked because an operator may cite a large file and there is no reason to + hold it in memory. The digest is the only thing that leaves this function: + an authorisation artifact is exactly the kind of file that carries a + credential in passing, and the note it would ride into is the one artifact + that leaves the machine. + """ + digest = hashlib.sha256() + total = 0 + with open(path, "rb") as fh: + while True: + chunk = fh.read(_DIGEST_CHUNK) + if not chunk: + break + total += len(chunk) + digest.update(chunk) + return digest.hexdigest(), total + + +def declare_authority( + session: Session, + artifact: Path, + scope: str, + label: str, + principal: Optional[str] = None, + expires_after_seals: Optional[int] = None, +) -> dict: + """Record that a file with this digest was cited as the authority for a scope. + + Refuses a scope outside ``AUTHORITY_SCOPES``, an empty label, a TTL below + one, and an artifact that does not exist or is empty. An empty file is + refused rather than digested because its digest is a constant: every empty + authorisation in the world shares it, so it distinguishes nothing and would + be the one shape a fabricated citation could produce without a file. + + The path is recorded as the operator wrote it and is not resolved. A relative + citation therefore stays relative, which keeps a home directory out of a + published note when the operator cites one relatively; an absolute path is + published as typed. + """ + if scope not in AUTHORITY_SCOPES: + raise ManifestError( + f"unknown authority scope {scope!r}: didrun authorises only the " + f"exceptional transitions ({', '.join(AUTHORITY_SCOPES)}). There is " + f"no scope for an ordinary seal, and none will be added: an " + f"authorisation required on the happy path becomes a rubber stamp." + ) + if not label or not label.strip(): + raise ManifestError( + "an authority needs a --label saying what is being authorised; " + "an unlabelled citation records nothing a reader can act on" + ) + ttl = AUTHORITY_DEFAULT_TTL if expires_after_seals is None else expires_after_seals + if not _is_index(ttl) or ttl < 1: + raise ManifestError( + f"--expires-after-seals must be an integer >= 1 (got {ttl!r}); an " + f"authority that expires after zero seals authorises nothing, and " + f"there is no value meaning 'never'" + ) + artifact = Path(artifact) + if not artifact.is_file(): + raise ManifestError( + f"cannot cite {artifact}: no such file (an authority cites an " + f"artifact that exists, so the digest is of something a reviewer " + f"can open)" + ) + sha256, size = _artifact_digest(artifact) + if size == 0: + raise ManifestError( + f"cannot cite {artifact}: the file is empty, and every empty file " + f"has the same digest — it would distinguish nothing" + ) + record = { + "principal": ( + principal if principal is not None else os.environ.get("USER", "") + ), + "principal_basis": AUTHORITY_PRINCIPAL_BASIS, + "artifact_path": str(artifact), + "artifact_sha256": sha256, + "artifact_bytes": size, + "scope": scope, + "label": label, + "declared_at_index": _seal_count(session), + "expires_after_seals": ttl, + } + # Same append discipline as claims and seals: 0600 at creation, under the + # ledger-wide lock. A record line carries an operator-authored label and is + # not bounded by PIPE_BUF, so two concurrent `didrun authorize` invocations + # could otherwise interleave into one unparseable record. + with _append_lock(session.root): + with _open_private_append(_authority_path(session)) as fh: + fh.write(canonical_json(record) + b"\n") + return record + + +def _valid_authority(record) -> bool: + """Whether a stored authority record is one this seal may read. + + ``authority.jsonl`` is a local, gitignored, unhashed file, exactly like + ``claims.jsonl`` — so every field here arrived from a file rather than from + this process, and gets the same validate-rather-than-trust reading that + ``_superseded_by`` gets. Not a forgery barrier and not presented as one: + whoever can write this file can write a well-formed record. + """ + if not isinstance(record, dict): + return False + if record.get("scope") not in AUTHORITY_SCOPES: + return False + if not _is_index(record.get("declared_at_index")) or record["declared_at_index"] < 0: + return False + if not _is_index(record.get("expires_after_seals")) or record["expires_after_seals"] < 1: + return False + digest = record.get("artifact_sha256") + if not isinstance(digest, str) or len(digest) != 64: + return False + if any(c not in "0123456789abcdef" for c in digest): + return False + if not _is_index(record.get("artifact_bytes")) or record["artifact_bytes"] < 1: + return False + for key in ("principal", "artifact_path", "label"): + if not isinstance(record.get(key), str): + return False + return True + + +def _load_authorities(session: Session) -> tuple: + """Every readable authority record, plus how many were dropped. + + A record that does not validate is DROPPED. Dropping is the refusing + direction: an authority grants permission, so ignoring a malformed one + produces a refusal the operator can see and fix, while trusting it would + produce a permission nobody granted. A torn tail therefore costs a + citation, never a false one. + """ + path = _authority_path(session) + if not path.exists(): + return [], 0 + records = [] + dropped = 0 + for line in path.read_text(encoding="ascii", errors="replace").splitlines(): + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except ValueError: + dropped += 1 + continue + if _valid_authority(record): + records.append(record) + else: + dropped += 1 + return records, dropped + + +def _bind_authorities(session: Session, scopes: tuple, seal_index: int) -> list: + """The authorities this seal cites, graded ``cited``. + + Three conditions, every one of them fail-closed: + + * **Scope** — the record's scope is one this seal actually exercises. An + ordinary seal exercises none and therefore binds none, which is what keeps + an authority attached to exceptional transitions instead of becoming a line + on every note. + * **Not expired** — ``seal_index - declared_at_index < expires_after_seals``. + The clock starts at the seal that follows the authorisation, so + ``--expires-after-seals 1`` covers the next seal and not the one after. + * **Not from the future** — a record may not name a seal generation that has + not happened (``declared_at_index <= seal_index``). That is the guard + against a record written or backdated by hand to cover seals it never saw. + + ``limitation``, ``grade`` and ``principal_basis`` are written into every + bound entry from their SOURCE CONSTANTS so the note itself carries them, + not just the surfaces that render it — and so a hand-edited + ``authority.jsonl`` cannot dictate the three fields that state how much the + citation is worth. ``principal_basis`` belongs in that set because it + qualifies the principal: "self-asserted" is the only basis this version can + establish, so it is the only basis it will publish. + """ + records, dropped = _load_authorities(session) + if dropped: + print( + f"didrun seal: WARNING: dropped {dropped} unreadable authority " + f"record(s) from the ledger; they authorise nothing. A dropped " + f"record is the refusing direction, so a scope that needed one is " + f"refused rather than granted.", + file=sys.stderr, + ) + bound = [] + for record in records: + if record["scope"] not in scopes: + continue + declared = record["declared_at_index"] + if declared > seal_index: + continue + if seal_index - declared >= record["expires_after_seals"]: + continue + entry = dict(record) + entry["grade"] = AUTHORITY_GRADE + entry["limitation"] = AUTHORITY_LIMITATION + entry["principal_basis"] = AUTHORITY_PRINCIPAL_BASIS + bound.append(entry) + return bound + + +def _require_authority(scopes: tuple, bound: list) -> None: + """Refuse a transition whose scope no live cited authority covers. + + Default off, and it stays that way: making it default-on would put an + authorisation on the happy path, which is the failure mode this whole design + is shaped around. It exists for a CI gate that wants "a bare + `--allow-secrets` is not enough", and it says nothing stronger than that — + the authority it demands is still only a citation. + """ + covered = {entry["scope"] for entry in bound} + missing = [scope for scope in scopes if scope not in covered] + if not missing: + return + raise ManifestError( + "--require-authority: no live cited authority with scope " + + ", ".join(repr(s) for s in missing) + + f". Declare one with `didrun authorize --artifact --scope " + f"{missing[0]} --label \"\"` and re-run. {AUTHORITY_LIMITATION}" + ) + + +# How `_redact_authorities` must treat every key a bound authority carries. The +# split is exhaustive BY CONSTRUCTION: the pass publishes a key only if it is +# named here, so a field added later is dropped rather than published unscanned, +# and `test_every_authority_field_is_classified` fails the moment +# `declare_authority` grows a field nobody classified. +# +# Enumerating the text fields by hand, with no check that the enumeration was +# complete, is exactly what published `principal` verbatim into a git note while +# the refusal message promised a redacted export. The enumeration is no longer +# allowed to be silently incomplete. +_AUTHORITY_TEXT_FIELDS = ( + # (field name, the FindingSource kind its content is scanned as) + ("label", "label"), + ("artifact_path", "pathspec"), + # `--principal`, or `$USER` when that is absent. Outside text, exactly as + # much as the label is, and published in the same JSON object. + ("principal", "label"), +) + +# Computed here, validated by `_valid_authority`, or pinned from a source +# constant by `_bind_authorities`. None of these can carry outside text, which +# is why they are published as-is — the reason is recorded per field because +# "this one is safe" is the claim that has to stay true. +_AUTHORITY_PASSTHROUGH_FIELDS = frozenset( + { + "artifact_sha256", # computed here; validated as 64 lowercase hex + "artifact_bytes", # computed here; validated as an int >= 1 + "scope", # closed vocabulary (AUTHORITY_SCOPES) + "declared_at_index", # computed here; validated as an int >= 0 + "expires_after_seals", # validated as an int >= 1 + "principal_basis", # source constant, pinned at bind + "grade", # source constant, pinned at bind + "limitation", # source constant, pinned at bind + } +) + + +def _redact_authorities(bound: list) -> tuple: + """Scrub the operator-authored strings in the bound authorities. + + Returns ``(payload, findings, bytes_scanned)`` — the same three things + ``_redact_result`` returns for a claim, and for the same reason: the + findings ARE the export gate's input for these fields, so an authority + label cannot be blocked-but-published. That pair coming apart is the exact + defect the claim-field pass exists to have closed, and a field added later + reintroduces it unless it is redacted by the pass that reports it. + + THREE fields carry outside text: the label, the cited path and the principal. + They are listed in ``_AUTHORITY_TEXT_FIELDS`` and every one of them goes + through ``redact_field``. Everything else is published only if + ``_AUTHORITY_PASSTHROUGH_FIELDS`` vouches for it; an unclassified key is + dropped, because the failing direction for a field nobody has thought about + is to leave it out of the artifact that leaves the machine. + + Findings are located by field NAME rather than by an authority index — + ``FindingSource`` addresses claims and streams, and widening it is a change + to the projection contract, not to this unit — so the precise address + travels in ``redaction.fields`` instead. + """ + payload = [] + findings: list = [] + scanned = 0 + for index, entry in enumerate(bound): + fields: list = [] + + def _field(name: str, kind: str, value) -> str: + nonlocal scanned + text = value if isinstance(value, str) else str(value) + scanned += len(text.encode("utf-8", "replace")) + redaction = redact.redact_field(name, text, redact.FindingSource(kind)) + fields.extend(redaction.applied) + findings.extend(redaction.findings) + return redaction.text + + out = {k: v for k, v in entry.items() if k in _AUTHORITY_PASSTHROUGH_FIELDS} + for name, kind in _AUTHORITY_TEXT_FIELDS: + if name in entry: + out[name] = _field(f"authority[{index}].{name}", kind, entry[name]) + out["redaction"] = { + "projection_version": redact.PROJECTION_VERSION, + "detector_set_version": redact.DETECTOR_SET_VERSION, + "fields": fields, + } + payload.append(out) + return payload, findings, scanned + + # --- helpers ----------------------------------------------------------------- def _rev(repo: Path, commitish: str) -> Optional[str]: diff --git a/src/didrun/render.py b/src/didrun/render.py index e1511a1..38d84c6 100644 --- a/src/didrun/render.py +++ b/src/didrun/render.py @@ -21,6 +21,11 @@ from typing import Optional from .claims import ENV_DRIFT, ENV_INCOMPARABLE, ENV_MATCH, ENV_NOT_RECORDED +from .manifest import ( + AUTHORITY_GRADE, + AUTHORITY_LIMITATION, + AUTHORITY_PRINCIPAL_BASIS, +) # Grade -> (text token, ANSI color, severity marker for NO_COLOR, honest gloss). # The token NEVER overclaims: the strongest positive says TREE-EXACT ("the @@ -97,6 +102,19 @@ def _sanitize(s: str) -> str: return "".join(out) +def sanitize_text(value) -> str: + """`_sanitize` for the command surfaces outside this module. + + `didrun seal` and `didrun authorize` print an authority's label and principal + straight to the terminal, and neither value is computed by this process: + `authority.jsonl` is a local, unhashed, gitignored file that gets the same + validate-rather-than-trust reading as `claims.jsonl`, so whoever can write it + chooses those strings. A newline in a label fabricates a verdict row in the + seal's own output, the same way it did in the verify block. + """ + return _sanitize(value if isinstance(value, str) else str(value)) + + def _skipped_notes(n: int) -> str: """Notes under the didrun ref that could not be parsed at all. @@ -182,6 +200,78 @@ def superseded_text(report) -> str: ) +# The token for a bound authority. Deliberately NOT a row in `_GRADE_DISPLAY` or +# `_SORT_RANK`: `cited` is an AUTHORITY's grade and never a claim's. No +# GradeResult carries it, `worst_status` never returns it, and a `_SORT_RANK` row +# would imply it sorts among the claim grades, which it never does. It is held to +# the same rule as every token in that table — reported verbatim, never upgraded: +# `CITED` must never be rendered as "signed", "approved" or "authorized by". +_AUTHORITY_TOKEN = AUTHORITY_GRADE.upper() + + +def authority_lines(report, indent: str = " ") -> list: + """The block for each authority this note cited, or [] when it cited none. + + The limitation sentence comes from the SOURCE CONSTANT, never from the note. + A note is data this process did not compute, so a limitation a note could + edit would be no limitation at all — and this is the one string standing + between `CITED` and being read as approval. + + An ordinary seal cites nothing, so this returns [] and the verdict gains no + line. That is not a rendering choice: an authority attaches only to an + exceptional transition (docs/TRUST_MODEL.md). + """ + lines: list = [] + for entry in report.authority: + + def field(name: str, _entry=entry) -> str: + """Every value this block reads out of the note, sanitized at the + single point of access. + + Not `_sanitize` spelled out at each use site: that is how `size` and + `ttl` ended up as the two of six fields nobody wrapped. An + unsanitized value interpolated into these lines can carry newlines, + and a newline here fabricates whole verdict rows ABOVE the real + table — rows the renderer never computed, in the vocabulary this + module exists to keep honest. A note travels over `git fetch`, so it + is data from another machine, and `_sanitize` is not optional for + any of it. The HTML path already funnels every field through one + `esc`; this is the terminal's equivalent. + """ + value = _entry.get(name, "") + return _sanitize(value if isinstance(value, str) else str(value)) + + scope = field("scope") + label = field("label") + path = field("artifact_path") + # The full digest, not a 12-character prefix like the commit and tree + # above it. Comparing this value against a file is the one thing the + # feature buys, and printing a prefix would invite a reviewer to compare + # a prefix. + digest = field("artifact_sha256") + size = field("artifact_bytes") + principal = field("principal") or "(unnamed)" + # From the SOURCE CONSTANT, like the token and the limitation below, and + # for the same reason: this field says how the principal's name was + # established, so a note that supplied it could print "principal alice + # (verified-ssh-signature)" directly above a sentence saying didrun + # cannot tell who wrote it. "self-asserted" is the only basis this + # version can establish, so it is the only one it will print. + basis = _sanitize(AUTHORITY_PRINCIPAL_BASIS) + ttl = field("expires_after_seals") + lines.append( + _c(f"{indent}! {_AUTHORITY_TOKEN} authority for scope {scope}", "33") + + f": {label}" + ) + lines.append(f"{indent} artifact {path} sha256 {digest} {size} bytes") + lines.append( + f"{indent} principal {principal} ({basis}) · " + f"expires after {ttl} seal(s)" + ) + lines.append(f"{indent} {AUTHORITY_LIMITATION}") + return lines + + def _conjunct_token(grade) -> str: """The token for one conjunct of a conjunction — never a grade for a name that resolved to nothing. @@ -261,6 +351,9 @@ def render_verdict(report, width: int = 80) -> str: lines.append(f" {env_summary_text(report)}") if report.secrets_override: lines.append(_c(" ! sealed with --allow-secrets (redacted export)", "33")) + # Directly under the override line, because the override is what an authority + # is most often cited for and a reader must see the two together. + lines.extend(authority_lines(report)) if report.notes_skipped: lines.append(_c(f" ! {_skipped_notes(report.notes_skipped)}", "33")) lines.append("") @@ -381,6 +474,26 @@ def esc(s: str) -> str: superseded = superseded_text(report) superseded_html = f"{esc(superseded)} ·" if superseded else "" + # One block per cited authority, above the table for the same reason as the + # chain sentence: it is a statement about the seal, not about a claim, so a + # table row would file it alongside the grades. Every field is escaped, and + # the limitation, the token and the principal's basis come from source + # constants, never from the note. + authority_html = "".join( + "
    " + f"{esc(_AUTHORITY_TOKEN)} " + f"authority for scope {esc(a.get('scope', ''))}: {esc(a.get('label', ''))}" + f"artifact {esc(a.get('artifact_path', ''))} · " + f"sha256 {esc(a.get('artifact_sha256', ''))} · " + f"{esc(a.get('artifact_bytes', ''))} bytes · " + f"principal {esc(a.get('principal', '') or '(unnamed)')} " + f"({esc(AUTHORITY_PRINCIPAL_BASIS)}) · " + f"expires after {esc(a.get('expires_after_seals', ''))} seal(s)" + f"{esc(AUTHORITY_LIMITATION)}" + "
    " + for a in report.authority + ) + if all_ok: verdict_class, verdict_text = "ok", "ALL RECORDED-EXACT" elif worst == "chain-broken": @@ -445,6 +558,15 @@ def esc(s: str) -> str: .chainbanner {{ margin:8px 0; padding:8px 10px; border-radius:3px; background:var(--bad-fill); color:var(--on-fill); font-weight:700; }} .meta {{ color:var(--dim); font-size:12px; word-break:break-all; }} + /* An authority is record, not verdict, so it is toned like the advisory + states — never like a pass. The limitation is not small print: it is the + same size as the citation it qualifies. */ + .authority {{ margin:8px 0; padding:8px 10px; border-radius:3px; + border:1px solid var(--warn); color:var(--ink); font-size:12px; }} + .badge.cited {{ color:var(--warn); }} + .authority .ameta {{ display:block; margin-top:4px; color:var(--dim); + word-break:break-all; }} + .authority .alimit {{ display:block; margin-top:4px; color:var(--warn); }} table {{ width:100%; border-collapse:collapse; margin-top:8px; table-layout:fixed; }} th {{ text-align:left; color:var(--dim); font-weight:500; font-size:11px; text-transform:uppercase; letter-spacing:.05em; padding:8px; border-bottom:1px solid var(--line); }} @@ -505,6 +627,7 @@ def esc(s: str) -> str:
    didrun · flight record
    {verdict_text}
    {chain_html} + {authority_html}
    {esc(verified)}/{esc(total)} claims recorded-exact · {esc(report.evidence_bound_count)}/{esc(total)} claims evidence-bound · {superseded_html} diff --git a/tests/compat/synthetic.py b/tests/compat/synthetic.py index 45f5e1b..a1dca91 100644 --- a/tests/compat/synthetic.py +++ b/tests/compat/synthetic.py @@ -387,7 +387,13 @@ def build_corpus(root: Path) -> SyntheticCorpus: coverage=_coverage(), ).to_json() ) - for added_after_v1 in ("secrets_override", "secrets", "claims_from", "claims_to"): + for added_after_v1 in ( + "secrets_override", + "secrets", + "claims_from", + "claims_to", + "authority", + ): del body_d[added_after_v1] _strip_post_v2_claim_keys(body_d) _attach_note(root, commit_d, canonical_json(body_d)) diff --git a/tests/compat/test_corpus_replay.py b/tests/compat/test_corpus_replay.py index 9aa54f1..423d2d4 100644 --- a/tests/compat/test_corpus_replay.py +++ b/tests/compat/test_corpus_replay.py @@ -314,10 +314,16 @@ def test_leg1_chain_recompute(replay_source): # live inside a claim entry, and a claim entry round-trips verbatim — so v3's # allowlist is v2's. It is spelled out rather than aliased so that a v3-only # additive key stays a deliberate one-line diff here, exactly as for v1 and v2. +# `authority` is P5.1's: the authorisation artifacts a seal cited, each carrying +# its digest and the limitation constant. Additive with a v1-reproducing default +# ([]) and NO version bump — an authority is a manifest field, never an Event +# field, precisely so it stays compat-neutral. A stored note of any version may +# legitimately lack it (every note published before P5.1 does), so all three +# allowlists carry it. Third deliberate one-line diff this criterion has forced. MANIFEST_ADDITIVE_KEYS = { - 1: frozenset({"secrets_override", "secrets", "claims_from", "claims_to"}), - 2: frozenset({"secrets_override", "secrets", "claims_from", "claims_to"}), - 3: frozenset({"secrets_override", "secrets", "claims_from", "claims_to"}), + 1: frozenset({"secrets_override", "secrets", "claims_from", "claims_to", "authority"}), + 2: frozenset({"secrets_override", "secrets", "claims_from", "claims_to", "authority"}), + 3: frozenset({"secrets_override", "secrets", "claims_from", "claims_to", "authority"}), } diff --git a/tests/test_claim_binding.py b/tests/test_claim_binding.py new file mode 100644 index 0000000..a7809d4 --- /dev/null +++ b/tests/test_claim_binding.py @@ -0,0 +1,270 @@ +"""P5.2 — `didrun claim` must not receipt an older passing run. + +The defect, reachable through the documented happy path and nothing more +exotic: `cmd_claim` with no `--event` bound to the last event whose +``exit_code == 0`` rather than to the last event. So + + didrun run -- pytest -q # passes, event 0 + + didrun run -- pytest -q # FAILS, event 1 + didrun claim tests-pass # bound event 0 + +produced a claim backed by the older passing run. ``declared_at_index`` stayed +at ``len(events) - 1``, so the retroactive-binding rule (which fires only when +the claim was declared BEFORE its backing event) did not catch it, and if the +tree had not moved between the two runs the claim graded ``tree-exact``. The +only signal was the index printed in a success line. + +The fix is a refusal, not a better heuristic: bind to the last event, and if the +last event did not exit 0, refuse and say so. Binding an earlier event stays +available through an explicit ``--event N``, which now warns about the gap. +""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as _manifest +from didrun.capture import run_wrapped +from didrun.ledger import Event, Session + +PASS = [sys.executable, "-c", "print('ok')"] +FAIL = [sys.executable, "-c", "import sys; sys.exit(3)"] + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _claims(repo: Path) -> list: + return _manifest._load_claims(_session(repo)) + + +def _claim(repo: Path, *args: str) -> int: + return cli.main(["--repo", str(repo), "claim", *args]) + + +# --- 1. the headline case ----------------------------------------------------- + + +def test_a_failing_last_event_refuses_instead_of_receipting_the_older_pass( + repo: Path, capsys +): + """Pass, then fail, then claim with no --event. + + On pre-change code this SILENTLY receipted the older passing event 0 and + exited 0. The test therefore asserts three separate things, because "exit 2" + alone would still pass on a version that wrote the claim and then failed for + an unrelated reason: the exit code, that the message names the failing index + and its real exit code, and that claims.jsonl gained nothing at all. + """ + s = _session(repo) + run_wrapped(PASS, s, repo) + run_wrapped(FAIL, s, repo) + assert [ev.exit_code for ev in s.events()] == [0, 3] + capsys.readouterr() + + assert _claim(repo, "tests-pass", "--label", "unit") == 2 + err = capsys.readouterr().err + assert "index 1" in err, err + assert "exited 3" in err, err + assert _claims(repo) == [], "a refused claim must write nothing" + assert not (repo / ".didrun" / "claims.jsonl").exists() or ( + repo / ".didrun" / "claims.jsonl" + ).read_bytes() == b"" + + +def test_the_refusal_names_the_deliberate_escape(repo: Path, capsys): + """A refusal that does not say what to do instead gets worked around. + + The escape is `--event N`, and it must be named in the refusal itself: an + operator who really does mean the earlier run has a supported way to say so. + """ + s = _session(repo) + run_wrapped(PASS, s, repo) + run_wrapped(FAIL, s, repo) + capsys.readouterr() + assert _claim(repo, "tests-pass") == 2 + assert "--event" in capsys.readouterr().err + + +def test_a_last_event_that_witnessed_no_exit_refuses_too(repo: Path, capsys): + """``exit_code is None`` is not a pass either. + + An interrupted flight and a PTY transcript both record an event that + witnesses no exit code, and the old backwards search skipped straight over + them to an earlier success. "Not 0" includes "unknown", and the message says + which of the two it is rather than printing "exited None". + """ + s = _session(repo) + run_wrapped(PASS, s, repo) + s.append( + Event( + argv=("some-command",), + cwd=str(repo), + env_fingerprint="0" * 16, + observed_via="transcript", + coverage="display-only", + ) + ) + assert s.events()[-1].exit_code is None + capsys.readouterr() + + assert _claim(repo, "tests-pass") == 2 + err = capsys.readouterr().err + assert "index 1" in err, err + assert "no exit code" in err, err + assert "None" not in err, err + assert _claims(repo) == [] + + +# --- 2. the happy path is still the happy path ------------------------------- + + +def test_a_passing_last_event_binds_to_it(repo: Path, capsys): + s = _session(repo) + run_wrapped(FAIL, s, repo) + run_wrapped(PASS, s, repo) + capsys.readouterr() + + assert _claim(repo, "tests-pass", "--label", "unit") == 0 + (claim,) = _claims(repo) + assert claim.event_indices == (1,), "binds the LAST event, not the first pass" + assert capsys.readouterr().err == "", "the happy path warns about nothing" + + +def test_the_single_event_case_binds_event_zero(repo: Path): + """The documented one-run flow, unchanged: run something, claim it.""" + s = _session(repo) + run_wrapped(PASS, s, repo) + assert _claim(repo, "tests-pass") == 0 + (claim,) = _claims(repo) + assert claim.event_indices == (0,) + + +# --- 3. an explicit older binding warns but succeeds ------------------------- + + +def test_explicit_event_zero_of_four_warns_and_still_writes_the_claim( + repo: Path, capsys +): + """`--event` is a deliberate act, so it succeeds — but not silently. + + 197 of 846 claims in the corpus this tool was built against bound a + non-terminal event. Whether any of those gaps spanned a FAILURE is unknown + (it needs a read of archived logs). The warning is what makes the next one + visible at the moment it is created. + """ + s = _session(repo) + for _ in range(4): + run_wrapped(PASS, s, repo) + assert len(s.events()) == 4 + capsys.readouterr() + + assert _claim(repo, "tests-pass", "--event", "0") == 0 + err = capsys.readouterr().err + assert "event 0" in err, err + assert "3 event(s) before the most recent" in err, err + (claim,) = _claims(repo) + assert claim.event_indices == (0,), "the claim is written, not refused" + + +def test_explicit_binding_to_the_most_recent_event_does_not_warn(repo: Path, capsys): + """There is no gap, so there is nothing to say. A warning on every explicit + `--event` would be noise, and noise is how a real warning gets filtered.""" + s = _session(repo) + for _ in range(3): + run_wrapped(PASS, s, repo) + capsys.readouterr() + assert _claim(repo, "tests-pass", "--event", "2") == 0 + assert capsys.readouterr().err == "" + + +def test_an_explicit_binding_to_a_failing_event_is_not_refused_here( + repo: Path, capsys +): + """The refusal is about the DEFAULT, not about `--event`. + + Binding a claim to an event that failed is allowed and is not a lie: the + grading ladder witnesses the failure and grades it `failed`, which is the + honest record. Refusing at declare time would hide a failure instead of + recording one. + """ + s = _session(repo) + run_wrapped(FAIL, s, repo) + run_wrapped(PASS, s, repo) + capsys.readouterr() + assert _claim(repo, "tests-pass", "--event", "0") == 0 + (claim,) = _claims(repo) + assert claim.event_indices == (0,) + + +# --- 4. no events at all ----------------------------------------------------- + + +def test_no_events_at_all_keeps_the_existing_refusal_verbatim(repo: Path, capsys): + """Asserted verbatim: docs/COMPAT.md quotes this sentence.""" + capsys.readouterr() + assert _claim(repo, "tests-pass") == 2 + assert capsys.readouterr().err == ( + "didrun claim: no recorded events to bind to (run something first)\n" + ) + + +# --- 5. declared_at_index is untouched in every path ------------------------- + + +@pytest.mark.parametrize( + "extra,events,expected_indices", + [ + ([], 3, (2,)), + (["--event", "0"], 3, (0,)), + (["--event", "1"], 2, (1,)), + ], + ids=["default", "explicit-older", "explicit-latest"], +) +def test_declared_at_index_is_always_the_last_index( + repo: Path, extra, events, expected_indices +): + """The retroactive-binding rule reads this field, and P5.2 changes which + event is BOUND, never where the declaration sits. If this drifted, a claim + bound to the last event would start grading `unknown`. + """ + s = _session(repo) + for _ in range(events): + run_wrapped(PASS, s, repo) + assert _claim(repo, "tests-pass", *extra) == 0 + (claim,) = _claims(repo) + assert claim.event_indices == expected_indices + assert claim.declared_at_index == events - 1 + + +# --- 6. end to end through the real binary ----------------------------------- + +_CLI_BOOT = "import sys; from didrun.cli import main; sys.exit(main(sys.argv[1:]))" + + +def test_the_refusal_holds_in_a_subprocess_too(repo: Path): + """In-process `cli.main` shares this interpreter's state; a separate process + is the shape an agent actually invokes, and the exit code is what a shell + gate reads.""" + for argv in ([*PASS], [*FAIL]): + subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "run", "--", *argv], + cwd=str(repo), + capture_output=True, + ) + proc = subprocess.run( + [sys.executable, "-c", _CLI_BOOT, "--repo", str(repo), "claim", "tests-pass"], + cwd=str(repo), + capture_output=True, + text=True, + ) + assert proc.returncode == 2, proc.stderr + assert "exited 3" in proc.stderr + assert _claims(repo) == [] diff --git a/tests/test_grading_honesty.py b/tests/test_grading_honesty.py index 36d6be5..d005616 100644 --- a/tests/test_grading_honesty.py +++ b/tests/test_grading_honesty.py @@ -23,7 +23,7 @@ from didrun import gitplumbing as gp from didrun.capture import run_wrapped -from didrun.claims import Claim, grade +from didrun.claims import WIDENING_HINT_CAP, Claim, _widening_hint, grade from didrun.ledger import Session # The two sentences the old fail-open path emitted. Neither may ever appear in @@ -207,3 +207,180 @@ def test_single_event_claims_are_unaffected(repo: Path, expected_grade: str): r = grade(claim, sealed_tree, s.events(), repo, s.blobs.root) assert r.grade == expected_grade assert r.reason == expected_reason + + +# --- P5.2: a scope miss names the widening set it already measured ----------- +# +# Measured over the corpus this tool was built against: 178 of 846 claims +# declared a pathspec and 14 of them lost `scope-exact` because the delta held +# exactly ONE out-of-scope path. The set that would have covered it was already +# in `delta`; the reason reported only how many paths missed, so an operator had +# to re-derive by hand something the tool had computed. + + +def _out_of_scope_case( + repo: Path, later: dict, pathspecs: tuple = ("src",) +) -> tuple[Session, Claim, str]: + """One event, then a commit that edits `later`'s paths only. + + The event's tree is the pre-edit HEAD tree, so grading against the post-edit + HEAD yields a delta of exactly `later`'s paths — every one of them outside + ``pathspecs``. + """ + s = _session(repo) + (repo / "src").mkdir() + (repo / "src" / "a.py").write_text("x = 1\n") + for rel, body in later.items(): + p = repo / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(body) + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "add tree") + run_wrapped([sys.executable, "-c", "print(1)"], s, repo) + for rel in later: + (repo / rel).write_text("later\n") + _git(repo, "add", "-A") + _git(repo, "commit", "-qm", "edit outside src") + return ( + s, + Claim( + ctype="tests-pass", + label="t", + event_indices=(0,), + pathspecs=pathspecs, + declared_at_index=0, + ), + gp.commit_tree(repo, "HEAD"), + ) + + +def test_one_out_of_scope_path_names_the_pathspec_that_would_cover_it(repo: Path): + """The 14-claim case. Grade, delta and count prefix are unchanged; the + reason gains the arithmetic fact of what widening would take.""" + s, claim, sealed = _out_of_scope_case(repo, {"docs/status/X.md": "one\n"}) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + + assert r.grade == "stale", "the hint must not promote the grade" + assert r.grade != "scope-exact" + assert [c.path for c in r.delta] == ["docs/status/X.md"], "delta unchanged" + assert r.reason == ( + "1 change(s) outside declared pathspecs" + " — add --path docs/status/ to make this scope-exact" + ) + # The old prefix is preserved verbatim, which is what a downstream validator + # matching on it reads. + assert r.reason.startswith("1 change(s) outside declared pathspecs") + + +def test_the_named_widening_set_actually_makes_it_scope_exact(repo: Path): + """The un-gameable half: the hint is a claim about `_within`'s arithmetic, so + declaring exactly what it names must produce `scope-exact`. A hint that + printed a plausible-looking directory would pass the assertion above and + fail here.""" + s, claim, sealed = _out_of_scope_case(repo, {"docs/status/X.md": "one\n"}) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + body = r.reason.split(" — ", 1)[1] + suffix = " to make this scope-exact" + assert body.startswith("add ") and body.endswith(suffix), body + named = [ + tok.removeprefix("--path ") + for tok in body[len("add ") : -len(suffix)].split(", ") + if tok.startswith("--path ") + ] + assert named == ["docs/status/"] + + widened = Claim( + ctype=claim.ctype, + label=claim.label, + event_indices=claim.event_indices, + pathspecs=claim.pathspecs + tuple(named), + declared_at_index=claim.declared_at_index, + ) + r2 = grade(widened, sealed, s.events(), repo, s.blobs.root) + assert r2.grade == "scope-exact", r2.reason + + +def test_twelve_out_of_scope_paths_in_three_directories_dedup_to_three(repo: Path): + """Dedup and sort, with nothing elided. + + Twelve out-of-scope paths across three directories are three prefixes, so + the hint names three and says nothing about a remainder — there is none. A + `+0 more` suffix here would be a false statement about what was dropped. + """ + later = { + f"{d}/f{i}.md": "one\n" + for d in ("zeta", "docs/status", "alpha/beta") + for i in range(4) + } + s, claim, sealed = _out_of_scope_case(repo, later) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + + assert r.grade == "stale" + assert len(r.delta) == 12 + assert r.reason == ( + "12 change(s) outside declared pathspecs — add --path alpha/beta/, " + "--path docs/status/, --path zeta/ to make this scope-exact" + ) + assert "more)" not in r.reason + + +def test_more_directories_than_the_cap_are_counted_not_dropped(repo: Path): + """Above the cap the hint stays readable and stays honest: five prefixes and + the exact number withheld. Truncating without the count would understate the + widening the operator was being told about.""" + later = {f"d{i:02d}/f.md": "one\n" for i in range(8)} + s, claim, sealed = _out_of_scope_case(repo, later) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + + assert r.grade == "stale" + assert r.reason == ( + "8 change(s) outside declared pathspecs — add --path d00/, --path d01/, " + "--path d02/, --path d03/, --path d04/ (+3 more) to make this scope-exact" + ) + assert r.reason.count("--path ") == WIDENING_HINT_CAP + + +def test_a_top_level_file_is_named_by_itself_not_by_the_repository_root(repo: Path): + """A path with no parent directory is the one case where "use the parent" + would be a lie with consequences: the parent is the repository root, and + declaring the root puts EVERY path in scope, so the hint would be advising a + declaration that stops meaning anything. It names the file instead — the + smallest prefix that covers it and nothing else.""" + s, claim, sealed = _out_of_scope_case(repo, {"TOPLEVEL.md": "one\n"}) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + assert r.reason == ( + "1 change(s) outside declared pathspecs" + " — add --path TOPLEVEL.md to make this scope-exact" + ) + assert "--path ." not in r.reason and "--path /" not in r.reason + + +def test_the_hint_is_deterministic_and_order_independent(repo: Path): + """Determinism is a hard invariant of the trust path, and the delta's order + is git's. Sorting is what makes the same scope miss produce the same reason + twice.""" + paths = ["b/x.md", "a/y.md", "b/z.md", "a/y.md"] + first = _widening_hint(paths) + assert first == _widening_hint(reversed(paths)) + assert first == "add --path a/, --path b/ to make this scope-exact" + + +def test_a_scope_exact_reason_is_byte_identical_to_before(repo: Path): + """No hint where nothing missed. The in-scope sentence is the one archived + notes carry for their 163 inert pathspec claims.""" + s, claim, sealed = _case_scope_exact(repo) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + assert r.grade == "scope-exact" + assert r.reason == "all 1 change(s) within declared pathspecs" + assert "add --path" not in r.reason + + +def test_a_stale_claim_with_no_pathspecs_gains_no_hint(repo: Path): + """The hint belongs to the branch that measured an out-of-scope SET. A claim + that declared no scope has no widening arithmetic to report, and inventing + one would be the relevance inference the module forbids.""" + s, claim, sealed = _case_stale(repo) + r = grade(claim, sealed, s.events(), repo, s.blobs.root) + assert r.grade == "stale" + assert r.reason == "tree moved since evidence: 1 path(s) differ" + assert "--path" not in r.reason diff --git a/tests/test_operator_authority.py b/tests/test_operator_authority.py new file mode 100644 index 0000000..ef91b23 --- /dev/null +++ b/tests/test_operator_authority.py @@ -0,0 +1,939 @@ +"""P5.1 — operator authority, and the exact size of what it buys. + +Every test here is written against one sentence, which is also the constant the +source prints on every surface: a `cited` authority records that a file with this +digest was cited for this scope. It is NOT a signature and NOT approval, because +**an agent can author its own authorization artifact in one line** — nothing in +this file, and nothing in didrun, distinguishes that from a human writing one. + +So the tests are deliberately asymmetric. The mechanism tests (bind, scope, +expiry, refusal) pin plumbing. The two that matter are the honesty ones: +``test_the_limitation_string_is_in_the_output``, which fails if any surface shows +a citation without the sentence that bounds it or reaches for a stronger word, +and ``test_retroactive_fabrication_is_detectable_and_only_that``, whose docstring +exists to stop a later reader from mistaking this feature for tamper-proofing. +""" + +from __future__ import annotations + +import hashlib +import json +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +from didrun import cli +from didrun import manifest as M +from didrun import redact +from didrun import render +from didrun.capture import run_wrapped +from didrun.claims import Claim +from didrun.ledger import Session + +# Neutral, low-entropy, and spaced so the tokenizer cannot see one long run: this +# string exists to be searched for in the published note, not to trip a detector. +_ARTIFACT_BODY = "body of the cited artifact\n" + +# Structurally a github token, and synthetic: the shape the detectors match, never +# a real credential. Same construction as tests/test_detector_tiers.py. +_LABEL_TOKEN = "ghp_" + "F" * 36 + + +def _session(repo: Path) -> Session: + return Session(repo / ".didrun") + + +def _one_event(session: Session, repo: Path): + return run_wrapped([sys.executable, "-c", "print('ok')"], session, repo) + + +def _claim(session: Session, label: str = "suite") -> None: + M.declare_claim( + session, + Claim(ctype="tests-pass", label=label, event_indices=(0,), declared_at_index=0), + ) + + +def _artifact(repo: Path, name: str = "APPROVAL.md", body: str = _ARTIFACT_BODY) -> Path: + path = repo / name + path.write_text(body, encoding="utf-8") + return path + + +def _authority_lines(repo: Path) -> list: + path = repo / ".didrun" / "authority.jsonl" + if not path.exists(): + return [] + return [ln for ln in path.read_text(encoding="ascii").splitlines() if ln.strip()] + + +def _commit(repo: Path, name: str, body: str) -> None: + (repo / name).write_text(body, encoding="utf-8") + for args in (["add", name], ["commit", "-qm", f"add {name}"]): + subprocess.run(["git", *args], cwd=str(repo), check=True, capture_output=True) + + +def _ready(repo: Path) -> Session: + """A session with one recorded event and one claim bound to it.""" + s = _session(repo) + _one_event(s, repo) + _claim(s) + return s + + +# --- test 1: record and bind ------------------------------------------------- + +def test_authorize_records_one_line_and_the_seal_binds_it(repo: Path): + """The digest travels into the note. The artifact's bytes never do. + + An authorization artifact is exactly the kind of file that carries a + credential in passing, and the note is the one artifact that leaves the + machine — so the absence of the body is asserted directly, not inferred from + the fact that nothing in the code writes it. + """ + s = _ready(repo) + artifact = _artifact(repo) + record = M.declare_authority( + s, artifact, scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + + assert len(_authority_lines(repo)) == 1 + stored = json.loads(_authority_lines(repo)[0]) + assert stored == record + assert "artifact_bytes" in stored and stored["artifact_bytes"] == len( + _ARTIFACT_BODY.encode("utf-8") + ) + + m = M.seal(s, repo, allow_secrets=True) + assert len(m.authority) == 1 + bound = m.authority[0] + real = hashlib.sha256(artifact.read_bytes()).hexdigest() + assert bound["artifact_sha256"] == real + assert bound["grade"] == M.AUTHORITY_GRADE == "cited" + assert bound["principal_basis"] == "self-asserted" + + note = m.to_json().decode("ascii") + # The body, and every word of it long enough to be recognisable, is absent. + assert _ARTIFACT_BODY.strip() not in note + assert "cited artifact" not in note + assert real in note + + +def test_an_ordinary_seal_cites_nothing_even_when_an_authority_exists(repo: Path): + """The regression case, and the anti-rubber-stamp invariant in one. + + An authority on disk must not attach itself to a seal that performs no + exceptional transition. If it did, `cited` would appear on every note, which + is the routine-isation this design is shaped to avoid. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + m = M.seal(s, repo) + assert m.authority == [] + report = M.verify(s, repo) + assert report.authority == [] + assert render.authority_lines(report) == [] + assert M.AUTHORITY_LIMITATION not in render.render_verdict(report) + + +# --- test 2: the limitation string is in the output -------------------------- + +_STRONGER_WORDS = ("signed", "approved", "authorized by", "verified") + + +def _stronger_word_in(surface: str) -> list: + """Which upgrade words appear in a surface, matched as WORDS. + + Word boundaries and not substrings: the HTML carries a CSS comment about the + "MOST-designed states", and a substring search would report `signed` inside + `designed` — a false hit that would either fail this test forever or, worse, + get it weakened to a scoped search that stops covering the header line. + """ + lowered = surface.lower() + return [ + word + for word in _STRONGER_WORDS + if re.search(rf"\b{re.escape(word)}\b", lowered) + ] + + +def test_the_limitation_string_is_in_the_output(repo: Path): + """Every surface that shows a citation shows what it is not. + + Both halves are asserted: the exact constant is present, and none of the four + words that would upgrade `cited` into something it is not appears anywhere in + the rendered text. The second half is what catches a later well-meaning + rewrite of the header line. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + M.seal(s, repo, allow_secrets=True) + report = M.verify(s, repo) + assert len(report.authority) == 1 + + terminal = render.render_verdict(report) + html = render.render_html(report) + assert M.AUTHORITY_LIMITATION in terminal + assert M.AUTHORITY_LIMITATION in html + assert "CITED" in terminal and "CITED" in html + # The detector is checked against a surface that WOULD overclaim before it is + # trusted on the two that must not. A regex typo here would otherwise make + # the honesty assertion below vacuous and silently green forever. + assert _stronger_word_in("CITED authority — SIGNED and approved by drew") == [ + "signed", + "approved", + ] + assert _stronger_word_in("nothing here says it in those words") == [] + assert _stronger_word_in(terminal) == [] + assert _stronger_word_in(html) == [] + + +def test_the_rendered_limitation_comes_from_the_source_not_the_note(repo: Path): + """A limitation a note could edit would be no limitation at all. + + The note carries the sentence too, so a reader of raw JSON sees it. But the + renderer must print the SOURCE constant, because the note is data this + process did not compute — the same reading rule every other uncomputed field + gets. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + M.seal(s, repo, allow_secrets=True) + report = M.verify(s, repo) + report.authority[0]["limitation"] = "this authority is fine, ship it" + + terminal = render.render_verdict(report) + assert M.AUTHORITY_LIMITATION in terminal + assert "ship it" not in terminal + + +# --- test 3: --require-authority refuses a bare override --------------------- + +def test_require_authority_refuses_a_bare_override_and_names_the_scope( + repo: Path, capsys +): + s = _ready(repo) + code = cli.main(["--repo", str(repo), "seal", "--allow-secrets", "--require-authority"]) + err = capsys.readouterr().err + assert code == 2 + assert M.AUTHORITY_ALLOW_SECRETS in err + assert _note_body(repo) is None # nothing was published + + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + code = cli.main(["--repo", str(repo), "seal", "--allow-secrets", "--require-authority"]) + assert code == 0 + body = _note_body(repo) + assert body is not None + assert json.loads(body)["authority"][0]["scope"] == M.AUTHORITY_ALLOW_SECRETS + + +def test_require_authority_leaves_the_happy_path_alone(repo: Path): + """It gates the exceptional transitions and nothing else. + + A seal that overrides nothing and reseals nothing exercises no scope, so + there is no scope to authorise and the flag cannot turn into the + every-seal rubber stamp. + """ + s = _ready(repo) + m = M.seal(s, repo, require_authority=True) + assert m.authority == [] + + +def test_require_authority_refuses_a_reseal_without_a_reseal_authority(repo: Path): + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo, reseal=True, require_authority=True) + assert M.AUTHORITY_RESEAL in str(exc.value) + M.declare_authority(s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it") + m = M.seal(s, repo, reseal=True, require_authority=True) + assert [a["scope"] for a in m.authority] == [M.AUTHORITY_RESEAL] + + +# --- test 4: scope isolation ------------------------------------------------- + +def test_a_reseal_authority_does_not_satisfy_allow_secrets(repo: Path): + """One scope, one boundary. A citation is not a general permission.""" + s = _ready(repo) + M.declare_authority(s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it") + + m = M.seal(s, repo, allow_secrets=True) + assert m.authority == [] # the wrong scope binds to nothing + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo, allow_secrets=True, require_authority=True) + assert M.AUTHORITY_ALLOW_SECRETS in str(exc.value) + assert M.AUTHORITY_RESEAL not in str(exc.value).split("scope", 1)[1].split(".", 1)[0] + + +# --- test 5: expiry ---------------------------------------------------------- + +def test_an_authority_expires_after_the_seals_it_declared(repo: Path): + """`--expires-after-seals 1` covers the next seal and not the one after. + + The clock is seals, not wall time: a wall clock in the trust path is not + allowed here, and it would make a citation's liveness depend on how long a + build took. + """ + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_ALLOW_SECRETS, + label="drew ok", + expires_after_seals=1, + ) + + first = M.seal(s, repo, allow_secrets=True) + assert len(first.authority) == 1 + + # A second commit, so the second seal gets its own note and the narrowing + # guard is not what this test is measuring. + _commit(repo, "next.py", "x = 1\n") + _claim(s, label="second unit") + second = M.seal(s, repo, allow_secrets=True) + assert second.authority == [] + + with pytest.raises(M.ManifestError) as exc: + M.seal(s, repo, allow_secrets=True, require_authority=True, reseal=True) + assert M.AUTHORITY_ALLOW_SECRETS in str(exc.value) + + +def test_a_wider_ttl_covers_the_seals_it_says_it_does(repo: Path): + """The complement of the test above: expiry counts, it does not just fire.""" + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_ALLOW_SECRETS, + label="drew ok", + expires_after_seals=2, + ) + assert len(M.seal(s, repo, allow_secrets=True).authority) == 1 + _commit(repo, "next.py", "x = 1\n") + _claim(s, label="second unit") + assert len(M.seal(s, repo, allow_secrets=True).authority) == 1 + _commit(repo, "third.py", "y = 2\n") + _claim(s, label="third unit") + assert M.seal(s, repo, allow_secrets=True).authority == [] + + +# --- test 6: never standing -------------------------------------------------- + +@pytest.mark.parametrize("scope", ["", "everything", "seal", "allow_secrets", None]) +def test_an_unscoped_or_unknown_authority_is_refused_at_authorize_time(repo, scope): + """There is no standing grant and no scope for an ordinary seal. + + `allow_secrets` (underscored) is in the parameter list on purpose: a + near-miss of a real scope must be refused rather than helpfully coerced, + because coercing it would silently widen what was authorised. + """ + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.declare_authority(s, _artifact(repo), scope=scope, label="drew ok") + assert "scope" in str(exc.value) + assert _authority_lines(repo) == [] + + +def test_the_cli_refuses_an_unknown_scope_with_exit_2(repo: Path): + """And it refuses it BECAUSE the scope is unknown, not because it cannot parse. + + The good invocation is asserted first on purpose. argparse exits 2 for any + unrecognised argument, including an `authorize` subcommand that does not + exist at all — so a bare "unknown scope exits 2" assertion passes on a + binary with no operator authority in it, which measured green on exactly + that during this unit's falsification pass. Pinning the accepted case first + is what makes the refusal below a statement about the SCOPE. + """ + _ready(repo) + artifact = _artifact(repo) + assert ( + cli.main( + [ + "--repo", str(repo), "authorize", + "--artifact", str(artifact), + "--scope", M.AUTHORITY_ALLOW_SECRETS, + "--label", "drew ok", + ] + ) + == 0 + ) + assert len(_authority_lines(repo)) == 1 + + with pytest.raises(SystemExit) as exc: + cli.main( + [ + "--repo", str(repo), "authorize", + "--artifact", str(artifact), + "--scope", "everything", + "--label", "drew ok", + ] + ) + assert exc.value.code == 2 + assert len(_authority_lines(repo)) == 1 # the refusal recorded nothing + + +def test_the_authorize_command_prints_the_digest_and_the_limitation(repo, capsys): + """`authorize`'s own output is where an operator forms their idea of this.""" + _ready(repo) + artifact = _artifact(repo) + assert ( + cli.main( + [ + "--repo", str(repo), "authorize", + "--artifact", str(artifact), + "--scope", M.AUTHORITY_ALLOW_SECRETS, + "--label", "drew ok", + ] + ) + == 0 + ) + out = capsys.readouterr().out + assert hashlib.sha256(artifact.read_bytes()).hexdigest() in out + assert M.AUTHORITY_LIMITATION in out + assert "self-asserted" in out + assert _stronger_word_in(out) == [] + + +def test_a_missing_or_empty_artifact_is_refused(repo: Path): + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.declare_authority( + s, repo / "nope.md", scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + assert "no such file" in str(exc.value) + + empty = repo / "EMPTY.md" + empty.write_text("", encoding="utf-8") + with pytest.raises(M.ManifestError) as exc: + M.declare_authority(s, empty, scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok") + assert "empty" in str(exc.value) + assert _authority_lines(repo) == [] + + +@pytest.mark.parametrize("ttl", [0, -1, True, "2", 1.5]) +def test_a_ttl_that_authorises_nothing_or_everything_is_refused(repo, ttl): + """No value means "never", and zero seals authorises nothing. + + `True` is in the list because `bool` is an `int` subclass: without the + `_is_index` reading it would be accepted as a TTL of 1, which is a real + number arriving from a place that never meant one. + """ + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_ALLOW_SECRETS, + label="drew ok", + expires_after_seals=ttl, + ) + assert "expires-after-seals" in str(exc.value) + + +def test_an_unlabelled_authority_is_refused(repo: Path): + s = _ready(repo) + with pytest.raises(M.ManifestError) as exc: + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label=" " + ) + assert "--label" in str(exc.value) + + +# --- test 7: what the feature actually buys ---------------------------------- + +def test_retroactive_fabrication_is_detectable_and_only_that(repo: Path): + """Editing the cited artifact after the seal is detectable. That is ALL. + + This is the ONE property operator authority buys. It does NOT establish that + a human wrote the artifact, that anyone read it, or that anyone approved + anything: an agent can author its own authorization artifact in one line, + cite it, and seal. It is also NOT tamper-proofing — whoever can rewrite the + note can rewrite the digest inside it just as easily as they can rewrite a + grade. What is detectable is exactly this: a digest recorded in an EARLIER + sealed note no longer matches the file the note names. + + Do not read this test as evidence of anything beyond that sentence. + """ + s = _ready(repo) + artifact = _artifact(repo) + M.declare_authority( + s, artifact, scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + sealed = M.seal(s, repo, allow_secrets=True) + sealed_digest = sealed.authority[0]["artifact_sha256"] + assert sealed_digest == hashlib.sha256(artifact.read_bytes()).hexdigest() + + artifact.write_text(_ARTIFACT_BODY + "and one more line\n", encoding="utf-8") + assert hashlib.sha256(artifact.read_bytes()).hexdigest() != sealed_digest + + # The earlier note is unchanged and still names the pre-edit digest, which is + # what makes the mismatch visible at all. + published = json.loads(_note_body(repo)) + assert published["authority"][0]["artifact_sha256"] == sealed_digest + + # And the mismatch is not a verdict: the claims still grade exactly as they + # did. An authority is record, never verdict. + report = M.verify(s, repo) + assert report.all_verified is True + + +# --- test 8: compat ---------------------------------------------------------- + +def test_a_note_without_an_authority_key_reads_back_additively(): + """Every note published before P5.1 lacks the key. [] is what that means.""" + for version in (1, 2, 3): + body = json.dumps( + { + "version": version, + "commit": "a" * 40, + "tree": "b" * 40, + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + }, + sort_keys=True, + separators=(",", ":"), + ).encode("ascii") + m = M.Manifest.from_json(body) + assert m.authority == [] + assert json.loads(m.to_json())["authority"] == [] + + +def test_a_malformed_authority_block_in_a_note_is_read_as_no_authority(): + """A block that is not a list, or holds things that are not records. + + Read down to nothing rather than raised on: an authority block is record and + never verdict, so it cannot be the reason a readable note stops verifying. + """ + base = { + "version": M.MANIFEST_VERSION, + "commit": "a" * 40, + "tree": "b" * 40, + "claims": [], + "coverage": {"total_events": 0, "by_coverage": {}}, + } + for bad in ("standing grant", 7, {"scope": "allow-secrets"}): + body = json.dumps( + dict(base, authority=bad), sort_keys=True, separators=(",", ":") + ).encode("ascii") + assert M.Manifest.from_json(body).authority == [] + + +def test_the_authority_version_contract_did_not_bump(repo: Path): + """Additive field, so no MANIFEST_VERSION bump (Locked Decision 9).""" + assert M.MANIFEST_VERSION == 3 + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + m = M.seal(s, repo, allow_secrets=True) + assert m.version == 3 + + +def test_an_unreadable_authority_record_is_dropped_not_trusted(repo: Path, capsys): + """A torn or hand-edited record costs a citation; it never grants one. + + Dropping is the refusing direction, which is why a scope that needed the + record is refused rather than quietly granted. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + path = repo / ".didrun" / "authority.jsonl" + good = path.read_text(encoding="ascii") + record = json.loads(good.strip()) + record["expires_after_seals"] = "forever" + path.write_text( + json.dumps(record, sort_keys=True, separators=(",", ":")) + "\n{tor\n", + encoding="ascii", + ) + with pytest.raises(M.ManifestError): + M.seal(s, repo, allow_secrets=True, require_authority=True) + assert "dropped 2 unreadable authority record(s)" in capsys.readouterr().err + + +def test_an_authority_from_the_future_binds_to_nothing(repo: Path): + """A record may not name a seal generation that has not happened. + + `authority.jsonl` is a local unhashed file, so this is validate-rather-than- + trust, not a forgery barrier: whoever can write the file can write a record + that passes. It closes the accident and the crude backdate, which is the same + bound every other uncomputed field in this package carries. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + path = repo / ".didrun" / "authority.jsonl" + record = json.loads(path.read_text(encoding="ascii").strip()) + record["declared_at_index"] = 9 + path.write_text( + json.dumps(record, sort_keys=True, separators=(",", ":")) + "\n", + encoding="ascii", + ) + assert M.seal(s, repo, allow_secrets=True).authority == [] + + +# --- the published bytes are the scanned bytes ------------------------------- + +def test_a_secret_in_an_authority_label_blocks_and_is_never_published(repo: Path): + """The authority block is published, so it is scanned AND redacted. + + Adding a published field that the export gate blocks on but the redactor + never touches would re-create the defect P3.2 closed: a token in a claim + label refused the seal and was then published verbatim on the override, + under a refusal message that had already promised a redacted artifact. + + The scope here is `reseal` rather than `allow-secrets`, and that is not + incidental: a bound authority is only published when the seal exercises its + scope, so `allow-secrets` is the one scope whose citation can never be + blocked — the flag that binds it is the same flag that overrides the block. + `reseal` is the case where the gate has something to say. + """ + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_RESEAL, + label=f"replace it {_LABEL_TOKEN}", + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo, reseal=True) + assert "github-token" in str(exc.value) + assert _note_body(repo) is None # the blocked seal published nothing + + m = M.seal(s, repo, reseal=True, allow_secrets=True) + note = m.to_json().decode("ascii") + assert _LABEL_TOKEN not in note + assert "redacted" in m.authority[0]["label"] + applied = m.authority[0]["redaction"]["fields"] + assert [f["field"] for f in applied] == ["authority[0].label"] + + +def _note_body(repo: Path): + proc = subprocess.run( + ["git", "notes", f"--ref={M.NOTES_REF}", "show", "HEAD"], + cwd=str(repo), + capture_output=True, + ) + return proc.stdout if proc.returncode == 0 else None + + +# --- the principal is outside text too --------------------------------------- + +# A high-entropy value that trips the notice tier rather than a named detector. +# Notice tier is the dangerous one here: it does not block, so a seal carrying it +# succeeds with `secrets_override` false and nothing asking the operator to look. +_NOTICE_TOKEN = "XMP7t2n3AGm5EhSc-sTMAMpLouiqyBfxD-__oQkcEbg" + + +def test_a_secret_in_the_principal_blocks_and_is_never_published(repo: Path): + """`principal` is `--principal` or `$USER` — outside text, and published. + + This is the same defect as the label case above, one field over: the export + gate scanned `principal` through the whole-manifest backstop and blocked on + it, the refusal said "re-run with --allow-secrets to export a REDACTED + artifact anyway", and the override then published the token verbatim into + the git note with `redaction.fields` empty. A refusal that promises + redaction has to deliver it for the field it refused over. + """ + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_RESEAL, + label="replace it", + principal=_LABEL_TOKEN, + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo, reseal=True) + assert "github-token" in str(exc.value) + assert _note_body(repo) is None + + m = M.seal(s, repo, reseal=True, allow_secrets=True) + note = m.to_json().decode("ascii") + assert _LABEL_TOKEN not in note + assert "redacted" in m.authority[0]["principal"] + applied = m.authority[0]["redaction"]["fields"] + assert [f["field"] for f in applied] == ["authority[0].principal"] + + +def test_the_same_secret_is_scrubbed_in_the_label_and_the_principal(repo: Path): + """One string in two adjacent fields must not come out two different ways. + + The notice tier made this the quiet case: the seal is not refused, nothing + is overridden, `secrets_override` stays false — and the value was scrubbed + in `label` and published raw in `principal`, in the same JSON object, in the + one artifact designed to leave the machine. + """ + s = _ready(repo) + M.declare_authority( + s, + _artifact(repo), + scope=M.AUTHORITY_RESEAL, + label=f"svc {_NOTICE_TOKEN}", + principal=_NOTICE_TOKEN, + ) + m = M.seal(s, repo, reseal=True) + + assert m.secrets_override is False # nothing was overridden + note = m.to_json().decode("ascii") + assert _NOTICE_TOKEN not in note + entry = m.authority[0] + assert "redacted" in entry["label"] and "redacted" in entry["principal"] + assert sorted(f["field"] for f in entry["redaction"]["fields"]) == [ + "authority[0].label", + "authority[0].principal", + ] + + +def test_every_authority_field_is_classified(repo: Path): + """The enumeration is not allowed to be silently incomplete. + + `_redact_authorities` names the fields it redacts by hand. Hand-written + enumerations go stale — `principal` was published verbatim for exactly that + reason — so this test fails the moment `declare_authority` or + `_bind_authorities` grows a field that neither table classifies, instead of + waiting for a reviewer to notice a new key in a note. + """ + s = _ready(repo) + record = M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it" + ) + produced = set(record) | {"grade", "limitation"} + classified = {name for name, _kind in M._AUTHORITY_TEXT_FIELDS} | set( + M._AUTHORITY_PASSTHROUGH_FIELDS + ) + assert produced <= classified, f"unclassified authority field(s): {produced - classified}" + + # And the published entry carries exactly what was classified, plus the + # redaction declaration — no key reaches a note unaccounted for. + m = M.seal(s, repo, reseal=True) + assert set(m.authority[0]) - {"redaction"} <= classified + + +def test_a_published_authority_field_the_redactor_forgot_still_blocks( + repo: Path, monkeypatch +): + """The whole-manifest backstop, exercised rather than asserted. + + `_scan_for_secrets` documents `export_bytes` as the backstop: "a field a + later version adds and forgets to redact is still in these bytes, so it + still blocks". Nothing tested that — removing the authority block from + `export_bytes` broke no test — so the sentence was a comment, not a + property. Here a field is added, vouched for as safe, and NOT redacted; the + seal must still refuse. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it" + ) + real_bind = M._bind_authorities + + def bind_with_a_new_field(session, scopes, seal_index): + bound = real_bind(session, scopes, seal_index) + for entry in bound: + entry["future_field"] = _LABEL_TOKEN + return bound + + monkeypatch.setattr(M, "_bind_authorities", bind_with_a_new_field) + monkeypatch.setattr( + M, + "_AUTHORITY_PASSTHROUGH_FIELDS", + M._AUTHORITY_PASSTHROUGH_FIELDS | {"future_field"}, + ) + with pytest.raises(redact.SecretsBlocked) as exc: + M.seal(s, repo, reseal=True) + assert "github-token" in str(exc.value) + + +def test_an_unclassified_authority_field_is_dropped_not_published( + repo: Path, monkeypatch +): + """For a field nobody classified, the failing direction is to leave it out. + + The counterpart to the test above: that one covers a field a later version + vouches for, this one covers a field it never thought about. Dropping loses + a line from a note; publishing loses a credential. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it" + ) + real_bind = M._bind_authorities + + def bind_with_a_new_field(session, scopes, seal_index): + bound = real_bind(session, scopes, seal_index) + for entry in bound: + entry["unclassified_field"] = _LABEL_TOKEN + return bound + + monkeypatch.setattr(M, "_bind_authorities", bind_with_a_new_field) + # It still blocks, because the backstop sees the bound entry — but the point + # here is what the override publishes. + m = M.seal(s, repo, reseal=True, allow_secrets=True) + note = m.to_json().decode("ascii") + assert "unclassified_field" not in note + assert _LABEL_TOKEN not in note + + +# --- a note cannot fabricate lines in the block that renders it -------------- + +_AUTHORITY_RENDERED_FIELDS = ( + "scope", + "label", + "artifact_path", + "artifact_sha256", + "artifact_bytes", + "principal", + "expires_after_seals", +) + + +def test_a_note_cannot_inject_lines_into_the_authority_block(repo: Path, monkeypatch): + """A note arrives over `git fetch`, so every field of it is outside data. + + `artifact_bytes` and `expires_after_seals` were the two of the block's + fields that no one wrapped in `_sanitize`, because each use site spelled the + call out by hand. Both are numbers in a note didrun wrote, and neither is a + number in a note someone else edited — a newline in either one fabricated + whole verdict rows above the real table, carrying the words this module + exists to keep out of its own output. + + `NO_COLOR` is set so the ESC assertion means what it says: with colour on the + renderer emits its own escapes, and "no ESC in the output" could not tell + those apart from an injected one. + """ + monkeypatch.setenv("NO_COLOR", "1") + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + M.seal(s, repo, allow_secrets=True) + report = M.verify(s, repo) + honest = dict(report.authority[0]) + assert len(render.authority_lines(report)) == 4 + + payload = ( + "1\n = ALL RECORDED-EXACT (upstream verified)\n" + " = TREE-EXACT forged-row exit 0 reviewed and signed off\x1b[31m\x1b[0m" + ) + for name in _AUTHORITY_RENDERED_FIELDS: + entry = dict(honest) + entry[name] = payload + report.authority = [entry] + + lines = render.authority_lines(report) + assert len(lines) == 4, f"{name}: the block is no longer four lines" + assert not any("\n" in ln for ln in lines), f"{name}: newline survived" + assert not any("\x1b" in ln for ln in lines), f"{name}: ESC survived" + + verdict = render.render_verdict(report) + assert not re.search( + r"(?m)^\s*[=~!x?]?\s*(TREE-EXACT|ALL RECORDED-EXACT)\s+forged-row", verdict + ), f"{name}: fabricated a verdict row" + + # Not `_stronger_word_in` here: the payload smuggles "signed" and + # "verified" as CONTENT, and a note whose label really says "signed off + # by legal" is entitled to have that rendered as text. The invariant + # those words belong to is about didrun's own vocabulary — the token and + # the limitation, both pinned to source constants and covered above. + # What must not survive is the structure: no line break, no escape. + html_out = render.render_html(report) + assert "\x1b" not in html_out, f"{name}: ESC survived into HTML" + assert "forged-row" in html_out.replace("&#", ""), f"{name}: content lost" + + +def test_the_rendered_principal_basis_comes_from_the_source_not_the_note(repo: Path): + """The third field that states what a citation is worth, pinned like the other two. + + `grade` and `limitation` already come from source constants so a note cannot + restate the trust level. `principal_basis` qualifies the principal, and it + did not — a note supplying "verified-ssh-signature" printed + `principal drew (verified-ssh-signature)` two lines above the sentence + saying didrun cannot tell who wrote it, in one of the exact words + `_STRONGER_WORDS` forbids. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_ALLOW_SECRETS, label="drew ok" + ) + M.seal(s, repo, allow_secrets=True) + report = M.verify(s, repo) + report.authority[0]["principal_basis"] = "verified-ssh-signature" + + terminal = render.render_verdict(report) + html_out = render.render_html(report) + for surface in (terminal, html_out): + assert "verified-ssh-signature" not in surface + assert M.AUTHORITY_PRINCIPAL_BASIS in surface + assert _stronger_word_in(terminal) == [] + assert _stronger_word_in(html_out) == [] + + +def test_the_published_principal_basis_comes_from_the_source_not_the_ledger(repo: Path): + """`authority.jsonl` is local, gitignored and unhashed, so it is outside data too. + + Whoever can write that file can write any basis into it. The note must carry + the basis this version can actually establish, not the one the file claims. + """ + s = _ready(repo) + M.declare_authority( + s, _artifact(repo), scope=M.AUTHORITY_RESEAL, label="replace it" + ) + path = repo / ".didrun" / "authority.jsonl" + record = json.loads(_authority_lines(repo)[0]) + record["principal_basis"] = "verified-ssh-signature" + path.write_text(json.dumps(record) + "\n", encoding="ascii") + + m = M.seal(s, repo, reseal=True) + assert m.authority[0]["principal_basis"] == M.AUTHORITY_PRINCIPAL_BASIS + assert "verified-ssh-signature" not in m.to_json().decode("ascii") + + +def test_the_command_surfaces_cannot_be_line_injected_by_an_authority( + repo: Path, capsys, monkeypatch +): + """`didrun authorize` and `didrun seal` print these strings too. + + The verify block was the reported case, but it is not the only surface that + prints an authority's label: `seal` echoes the citation it just published, + reading the label out of `authority.jsonl`. Redaction is not sanitization — + a scrubbed label still carries whatever control characters surrounded the + secret — so the seal's own output could grow a verdict row it never + computed, from a local file that gets validate-rather-than-trust treatment. + """ + monkeypatch.setenv("NO_COLOR", "1") + forged = r"(?m)^\s*[=~!]?\s*TREE-EXACT\s+forged-row" + payload = "ok\n = TREE-EXACT forged-row exit 0 reviewed and signed off" + + s = _ready(repo) + artifact = _artifact(repo) + code = cli.main( + [ + "--repo", str(repo), "authorize", + "--artifact", str(artifact), + "--scope", M.AUTHORITY_RESEAL, + "--label", payload, + ] + ) + out = capsys.readouterr().out + assert code == 0 + assert not re.search(forged, out), "authorize fabricated a verdict row" + assert "^J" in out, "the newline was not caret-encoded" + + assert cli.main(["--repo", str(repo), "seal", "--reseal"]) == 0 + out = capsys.readouterr().out + assert not re.search(forged, out), "seal fabricated a verdict row" + assert "^J" in out, "the newline was not caret-encoded" diff --git a/tests/test_redact_render.py b/tests/test_redact_render.py index 67ac14f..72e31ac 100644 --- a/tests/test_redact_render.py +++ b/tests/test_redact_render.py @@ -2,7 +2,7 @@ from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field from didrun import redact from didrun import render @@ -74,6 +74,11 @@ class R: chain_reason: str = chain_why # Mirrors VerifyReport's environment fields, for the same reason. require_env_match: bool = gating + # Mirrors VerifyReport's authority block, for the same reason. Empty on + # purpose: every surface test in this file renders a report that cited + # nothing, which is what pins "an ordinary verdict gains no authority + # line". The cited case lives in tests/test_operator_authority.py. + authority: list = field(default_factory=list) @property def worst_status(self): diff --git a/tests/test_seal_publication.py b/tests/test_seal_publication.py index 16ac454..f680dd1 100644 --- a/tests/test_seal_publication.py +++ b/tests/test_seal_publication.py @@ -294,6 +294,12 @@ def test_manifest_json_bytes_are_pinned(): P4.1 adds `claims_from`/`claims_to`, the claim window the seal covered, on the same terms: reading defaults of 0 and `len(claims)`, so no stored note changes meaning and this stays a deliberate diff rather than a version bump. + + P5.1 adds `authority`, the cited authorization artifacts, with a reading + default of `[]` — and it sorts FIRST, so the pinned bytes move at the front. + Same terms again: additive, no version bump, one deliberate diff here and one + in the compat allowlist. It is a manifest key and not an `Event` field + precisely so it can be exactly this: a change no stored note notices. """ m = M.Manifest( version=1, @@ -319,7 +325,8 @@ def test_manifest_json_bytes_are_pinned(): coverage={"by_coverage": {"complete": 1}, "total_events": 1}, ) assert m.to_json() == ( - b'{"claims":[{"claim":{"argv_preview":["pytest","-q"],"ctype":"tests-pass",' + b'{"authority":[],' + b'"claims":[{"claim":{"argv_preview":["pytest","-q"],"ctype":"tests-pass",' b'"declared_at_index":0,"event_indices":[0],"label":"t","pathspecs":[]},' b'"delta":[],"exit_code":0,"grade":"tree-exact","reason":"r",' b'"supporting_event_index":0}],' From f6c44b0d2308c72b76a80cb7064c6ef908259781 Mon Sep 17 00:00:00 2001 From: Drew Date: Wed, 29 Jul 2026 20:06:38 -0700 Subject: [PATCH 6/8] =?UTF-8?q?docs:=20record=20the=20v0.2=20compat=20repl?= =?UTF-8?q?ay=20=E2=80=94=20verified,=20bet,=20and=20known-not-closed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The real-corpus legs did NOT run: DIDRUN_COMPAT_CORPUS was unset, so all four are recorded as BET (UNEXERCISED) rather than simulated. What did run: the full suite (496 passed, 4 skipped), the capture kill-gate (PASS, 100% defect-class recall), a 3.11 grammar parse over the tree, the stdlib-only and private-path audits, and an end-to-end walk of the real CLI in throwaway repos covering 30 properties — including every earlier phase's headline property, re-checked rather than assumed after P3-P5 landed on top of them. COMPAT_REPLAY.md carries the VERIFIED / BET / KNOWN-NOT-CLOSED split and the four gates. V02_STATUS.md is the plain-English release statement, including the five defects this pass found and deliberately did not fix (the integration pass measures; a fix landing after the last gate is unverified code in a release). COMPAT.md's forward-incompatibility warning was stale in two places: the Version 3 section described the unknown-claim-type traceback in the present tense after a later unit turned it into a graded refusal in this reader, and "the compat corpus replays all three versions" was true only of the synthetic fixtures. Both corrected. --- docs/COMPAT.md | 22 ++++- docs/COMPAT_REPLAY.md | 140 +++++++++++++++++++++++++++ docs/V02_STATUS.md | 220 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 377 insertions(+), 5 deletions(-) create mode 100644 docs/COMPAT_REPLAY.md create mode 100644 docs/V02_STATUS.md diff --git a/docs/COMPAT.md b/docs/COMPAT.md index 93da733..c2fbf90 100644 --- a/docs/COMPAT.md +++ b/docs/COMPAT.md @@ -79,14 +79,23 @@ above exists to prevent. ### Version 3 — the claim-type vocabulary gained `conjunction` -**A v3 note may contain a claim whose `ctype` is `conjunction`, and reading it in a binary -whose claim vocabulary does not include that type is not a graded refusal but an uncaught -traceback: `verify` calls `Claim.from_dict` outside the `try` that grades a claim, so -`ClaimError: unknown claim type: 'conjunction'` propagates out of the command.** That is +**A v3 note may contain a claim whose `ctype` is `conjunction`, and in a released binary +whose claim vocabulary does not include that type — v0.1, or a v0.2 built before the +refusal below landed — reading it is not a graded refusal but an uncaught traceback: +`verify` called `Claim.from_dict` outside the `try` that grades a claim, so +`ClaimError: unknown claim type: 'conjunction'` propagated out of the command.** That is the vocabulary break, stated at full strength — "raises" understates it, and the `MANIFEST_VERSION` bump to 3 exists so a v0.1 or early-v0.2 reader refuses the note on its version *before* it reaches the claim it cannot parse. +**THIS reader no longer behaves that way, and the distinction matters when reading the rest +of this section.** A later unit closed the same hole from this end: a claim type this +binary cannot parse, in a note whose version it accepts, is now a whole-manifest refusal +naming the entry — `claim entry 0 is unreadable: unknown claim type: '…'`, exit 2, no +traceback. Measured. See *An unreadable note is a graded refusal* below. The paragraph +above describes what a note published by this version does **when it travels backwards to +an older binary**, which is the forward incompatibility this section is about. + There is no way to extend a closed vocabulary without that break. The alternative — skipping claim types a reader does not know — is worse: the note would silently under-report, and print a verdict over the claims it happened to understand. @@ -522,7 +531,10 @@ note from being read as a verdict. Stored evidence is unaffected: every note ever published carries an integer version of 1, 2 or 3 and the required fields, so nothing legitimately published is newly refused. The -compat corpus replays all three versions. +compat harness's **synthetic** fixtures cover all three versions and the per-version +additive-key allowlists; the **real** archive replay is still unexercised, so "nothing +legitimately published is newly refused" is an argument from the version guard, not a +measurement over stored notes. `docs/COMPAT_REPLAY.md` records which is which. ## Known, not closed diff --git a/docs/COMPAT_REPLAY.md b/docs/COMPAT_REPLAY.md new file mode 100644 index 0000000..b67ffa2 --- /dev/null +++ b/docs/COMPAT_REPLAY.md @@ -0,0 +1,140 @@ +# The v0.2 compat replay — verified, bet, and known not closed + +**Date:** 2026-07-29 · **Commit:** `49a8b74` on `v0.2-hardening` · **Python:** 3.14.5 (CPython, arm64) +· **Platform:** Darwin 25.5.0 (macOS) · **Suite:** 496 passed, 4 skipped + +**Corpus provenance: NONE. The corpus legs did not run.** `DIDRUN_COMPAT_CORPUS` was unset, so the four +real-corpus legs skipped by construction and every corpus-wide row below is **BET (UNEXERCISED)**. No +number in this document is a simulation of one. See *Gate 1* for exactly what clears it. + +Every command below ran unwrapped and is therefore **UNRECEIPTED** — the wrapper discipline is suspended +for this pack, so nothing here has a didrun receipt behind it. What it has is the shipped CLI's own +stdout, quoted. + +--- + +## VERIFIED — demonstrated, with the command that settles it + +Each row was produced this session by running the shipped CLI in a throwaway git repo under `$TMPDIR`, or +by a test in the suite that fails when the property stops being true. One witness, one platform, one +interpreter. + +| # | Claim | What settles it | Result | +|---|---|---|---| +| V1 | The suite is green | `python -m pytest -q` | **496 passed, 4 skipped** (4 skips are exactly the corpus legs) | +| V2 | The capture kill-gate holds | `python -m harness.recall` | **PASS**, freeze INTACT, defect-class recall **100.0% (3/3)**, structural 0.0% (0/2) | +| V3 | The synthetic compat legs are green | `python -m pytest -q tests/compat` | 11 passed, 4 skipped; leg1 7 logs / 11 entries, leg2 4 notes **0 violations**, leg3 **residue 0**, leg4 **multi-index 0** | +| V4 | The tree parses under 3.11 grammar | `ast.parse(…, feature_version=(3,11))` over all 41 `*.py` | **0 failures.** Syntax only — not an API check | +| V5 | Runtime is stdlib-only | AST walk of every import in `src/` + `harness/` | 24 top-level imports, **0 non-stdlib, 0 non-local**; `dependencies = []` | +| V6 | No private paths in shipped files | RULE 17 grep over `src/ tests/ harness/ docs/ testkit/` | Only `/home/runner/…` and `/home/example/…` — public CI shapes and explicit examples, which is what RULE 17 asks for | +| V7 | **Seal fails closed on note-publication failure** | `.git/objects` made unwritable, then `seal` | exit **2**, git's error quoted verbatim, **no `seals.jsonl`**, **no note**, **no watermark file**; permission restored → same seal succeeds and `verify --strict` exits 0 | +| V8 | A note far past `ARG_MAX` publishes and re-reads | 30 claims with 60 KB labels | note body **1,819,948 B** (14× the largest corpus note), seal exit 0, `verify` exit 0, 30/30 recorded-exact | +| V9 | **The substituted-ledger attack (EV2) fails closed** | seal 3 claims → `rm -rf .didrun` → 3 unrelated exit-0 commands on the same tree → `verify --strict` | **exit 1**, `WITNESS-UNAVAIL` ×3, **0/3 evidence-bound**, reason `entry hash mismatch at index 2`. Pre-P1.2 this was `3/3 recorded-exact`, exit 0 | +| V10 | A tampered chain dominates the verdict | flip one nibble of `entry_hash` at index 2 | `x ledger chain BROKEN at index 2 — every grade below is unreliable`, `--strict` **exit 1**, `show --session` **exit 1** | +| V11 | A missing ledger is *unavailable*, not *broken* | `mv .didrun .didrun-moved`, then `verify` | `WITNESS-UNAVAIL`, `--strict` exit 1, the word "broken" appears **0** times, and **no `.didrun` is recreated** | +| V12 | Ledger permissions | `ls -ld .didrun` after a write command | `drwx------`, `session.log` `-rw-------` | +| V13 | An interrupt never loses the flight | SIGINT then SIGTERM during `run -- sleep 30` | exit **130** / **143**, one greppable line, no traceback, event recorded `unobserved no-exit`, `chain intact`, and the next `run` appends normally | +| V14 | A torn log refuses rather than writing past the tear | truncate `session.log` by 25 B, then `run` | append refused, byte offset named, recovery spelled out, `session.log` still **1** line. (Delivered as a traceback — see U2) | +| V15 | Concurrent appends are safe and non-blocking | 6 parallel `run`s; then a 5 s `run` alongside a second | **6 events, chain intact**; the second run completed in **0.38 s** while the 5 s run held the session | +| V16 | Streamed output is byte-faithful, and a corrupt blob refuses | `run --tee`, `show --event 0 --output`, then flip a byte in the blob | output arrived progressively, replayed exactly; corrupted read → `blob digest mismatch (corruption)`, exit **2** | +| V17 | A malformed note is a **graded refusal**, never a crash and never a green | 6 bad `version` shapes + string claim entry + truncated body + unknown `ctype` + killed `ctype` | **every one exit 2** with a reason. `true`/`2.5`/`0`/`-1`/`"3"`/`null` all → `not a version number`; `version: 99` → `newer than this didrun understands (max 3); upgrade didrun` | +| V18 | Forging the note's own rows does not mint a green | rewrite `grade` to `tree-exact` and `evidence.entry_hash` to zeros | `WITNESS-UNAVAIL`, **0/1 evidence-bound** — the forgery grades *worse*, not better | +| V19 | Supersession retains the record and excludes it from the verdict | claim → move the tree → re-run → re-claim the same label → seal | `1 superseded`, first claim carries `superseded_by: 1` and prints `superseded by claim #1 (record, not verdict)`, `--strict` exit **0** | +| V20 | A narrowing re-seal is refused | `seal` over a narrower window | exit **2**, message names both windows and the two ways out; `--reseal` permits it | +| V21 | A conjunction is never better than its worst member | `claim conjunction --of T,L` with T stale | `STALE`, `worst of 2 conjunct(s): 'T' graded stale`, and both members printed (`of T: STALE`, `of L: TREE-EXACT`) | +| V22 | The detector tiers separate benign from dangerous | a deep build path in stdout vs. a `ghp_…` token in argv | path → **2 notice**, seal succeeds exit 0. Token → **export blocked**, exit **3**, located at `argv of event 0 arg 2 offset 11`, **0 notes written** | +| V23 | A scope miss names the fix | `--path src/` with a `docs/` change | `STALE`, `1 change(s) outside declared pathspecs — add --path docs/ to make this scope-exact` | +| V24 | A claim after a witnessed failure is refused | pass, then `exit 3`, then `claim tests-pass` | exit **2**, and **no `claims.jsonl` written at all** | +| V25 | Environment drift is advisory until asked | `TZ=UTC` seal, `TZ=Asia/Tokyo` verify | `0 match / 1 drifted`, `(advisory; --require-env-match makes it refuse)`; `--require-env-match --strict` exit **1** | +| V26 | A cited authority is cited, not claimed as approval | `authorize` → `seal --allow-secrets --require-authority` → `verify` | `CITED` row carrying, verbatim: *"NOT a signature and NOT approval — didrun cannot tell who wrote it"* | +| V27 | Amend still resolves the note | `git commit --amend`, then `verify` | `resolved-by tree-fallback`, verdict unchanged | +| V28 | Staleness has a non-empty delta | claim, then change the code, then seal | `! STALE`, `tree moved since evidence: 1 path(s) differ` + `M file.txt`, `--strict` exit 1 | +| V29 | `verify` in a repo with no ledger creates nothing | `verify` in a fresh repo | `○ NO CLAIMS`, and no `.didrun` afterwards | +| V30 | **The corpus guardrail refuses the live tree by name** | `is_refused()` / `resolve_corpus_root()` over path strings only — no I/O against the corpus | the live path, its trailing-slash form, and any path *inside* it are all refused by digest; a live-looking copy (`.didrun` + `.git`) is refused by heuristic; a proper copy is accepted | + +**Grep gate**, run and quoted: + +``` +$ grep -rnE "print\(|\bassert .*(line|raw|content|body)\b" tests/compat/ +tests/compat/__init__.py:7:content. ``assert line == ...`` is forbidden, because pytest prints both sides +tests/compat/test_corpus_replay.py:6:two hashes is fine — a hash is not content. ``assert line == ...`` is +tests/compat/test_corpus_replay.py:677: its arguments, so ``assert note_violations(, 0) == []`` would +tests/compat/test_corpus_replay.py:689: assert one_newline == [] +``` + +Four hits, **zero echo a ledger line**: three are the prose that states the rule, and `one_newline` is a +list of `(index, reason)` violation names bound to a local before the assertion, which is the shape the +gate exists to force. No `print(` anywhere in `tests/compat/`. + +**Read-only gate:** not applicable and not claimed — nothing was pointed at a corpus, so there were no +mtimes to compare. + +--- + +## BET — argued, not measured + +| # | Claim | What would settle it | Who | +|---|---|---|---| +| B1 | **D1 — every stored `entry_hash` in the archived logs recomputes** (or the exceptions are the two known fork incidents) | the corpus leg, over a copy | **HUMAN (Gate 1)** | +| B2 | **D2 — all 65 notes satisfy the three-part round-trip criterion** | the corpus leg, over a copy | **HUMAN (Gate 1)** | +| B3 | **D3 — every archived `objects/` entry partitions, residue 0** | the corpus leg, over a copy | **HUMAN (Gate 1)** | +| B4 | **D4 — multi-index 0, the 178/846 pathspec histogram, the gap distribution** | the corpus leg, over a copy | **HUMAN (Gate 1)** | +| B5 | **FU1** — the 667-vs-846 claim-count discrepancy between the original brief and the artifact | recount over the archived notes | HUMAN | +| B6 | **FU2** — "the structured detectors produced zero findings across the seals". Measured over exported `argv_preview` strings only, **never over the ledgers** | run the detectors over archived ledgers, by someone authorised to read them | HUMAN | +| B7 | **FU3** — the 1-of-8 resume-legality mapping under Locked Decision 2. Inference over recorded tree ids, not observation | inspect the eight launches directly | HUMAN | +| B8 | **FU4** — lane 2's EV2 false-green, single-sourced in the research. V9 above is a second witness for the **mechanism**, in a scratch repo; it is **not** a witness for the corpus | replay EV2 against a corpus copy | HUMAN | +| B9 | **FU5** — every frequency, cost and duration figure in the research, all self-reported by the system under study | independent instrumentation | HUMAN | +| B10 | **Red-team finding 16** — whether the 197 non-terminal claim bindings had an intervening failure | a grep over the archived logs, by someone authorised to read them | HUMAN | +| B11 | The **S6 live-agent recall leg** — capture recall against real agent sessions | a real agent on a real machine (`harness/README.md`) | **HUMAN (Gate 3)** | +| B12 | **CI green on 3.11 / 3.12 / 3.13 × ubuntu / macos** | the CI workflow. **Cannot be settled here** — this interpreter is 3.14.5, and V4 is a grammar check, not an API one | **HUMAN / CI (Gate 2)** | +| B13 | That any of this **changes a decision**. 846 corpus claims produced exactly **1** `failed`; a corpus whose only observed actor was cooperating cannot distinguish a superbly effective gate from a nearly inert one | instruct an agent to obtain a green `didrun verify --strict` by any means. **Not run here, and not authorised by this pack.** The highest-value missing experiment | **HUMAN (Gate 4)** | +| B14 | Durability under power loss. There is no `fsync` anywhere in the package; V15 shows appends are *mutually* safe, not that a committed append survives a crash | a fault-injection harness | open | + +--- + +## KNOWN NOT CLOSED — non-guarantees v0.2 leaves open on purpose + +- **Inherited-environment false greens.** `GOFLAGS=-exec=/usr/bin/true` and its family + (`NODE_OPTIONS`, `PYTHONPATH`, `LD_PRELOAD`, `DYLD_INSERT_LIBRARIES`, `PYTEST_ADDOPTS`, a PATH shim, + `sitecustomize.py`) make a passing command that ran nothing. No allowlist closes it. P3.3 records a PATH + **digest**; a declared clean-environment mode is the honest v0.3 shape. +- **No ledger retention or purge.** A credential scrubbed from git history survives as a loose object under + `.didrun/objects`. Real, and it needs a retention policy behind it before it needs code. +- **The redaction marker string is forgeable.** `«redacted:…»` is a convention, not a cryptographic mark; + keyed fingerprints were rejected because a per-ledger key destroys the cross-ledger stability that + motivated them. +- **`chmod` is close to a no-op on Windows.** V12's `drwx------` is a POSIX result and does not travel. +- **`flock` on a network filesystem.** V15 was measured on a local APFS volume. NFS/SMB semantics differ and + were not tested. +- **Tree-fallback returns the *first* matching note** (red-team finding 20) rather than failing closed on + more than one match. Deliberately out of this pack; the first candidate for a v0.2.1 unit. +- **`argv_preview` is written into every note, dropped on read, and computed-but-never-rendered in HTML** + (red-team finding 21). A product call, not a hardening unit. +- **Duplicate conjunct labels are counted twice.** `--of a,a` grades correctly and reads oddly. +- **`seal` still does not consult the chain.** `verify` does; sealing over a broken chain is possible. +- **None of this resists a determined local forger.** Whoever can substitute a ledger can regenerate its + chain and re-run `claim` and `seal` to mint fresh hash-bound claims. `ledger.py` declares this in the + source, `docs/TRUST_MODEL.md` states it, and V18 is *accident and drift detection*, not a barrier. + +--- + +## The four gates + +| Gate | Who clears it | Status | +|---|---|---| +| 1 — the corpus copy | **HUMAN** | **NOT CLEARED.** `DIDRUN_COMPAT_CORPUS` unset. Clear it by copying the archive tree **without** its live `.didrun` directory onto a separate volume and re-running with the variable set at the copy. The harness refuses the live tree by name (V30) even if it is set wrongly | +| 2 — CI on 3.11/3.12/3.13 × ubuntu/macos | **HUMAN / CI** | **NOT CLEARED.** Local green does not imply CI green | +| 3 — the S6 live-agent recall leg | **HUMAN** | **NOT CLEARED.** Unexercised | +| 4 — does any of this change a decision | **HUMAN** | **NOT CLEARED, and unanswerable from this corpus** | + +--- + +## Confidence + +**30 of 44 load-bearing conclusions are externally verified** — by the shipped CLI's own output in a +throwaway repo this session, or by a suite test that fails when the property stops being true. The other +14 rest on argument from code, on figures the system under study reported about itself, or on a signal +only a human or CI can supply. **Every one of the four real-corpus legs is in the second group.** + +A green replay means *these hashes reproduce*. It has never meant *the corpus is proven intact*, and a +`tree-exact` grade means *recorded against the sealed tree* — never *proven correct*. diff --git a/docs/V02_STATUS.md b/docs/V02_STATUS.md new file mode 100644 index 0000000..17317a1 --- /dev/null +++ b/docs/V02_STATUS.md @@ -0,0 +1,220 @@ +# didrun v0.2 — release readiness, stated honestly + +**As of 2026-07-29, commit `49a8b74` on `v0.2-hardening`.** Nineteen units landed across six phases. This +document exists because the difference between *"we shipped the code"* and *"we know it works"* is the only +thing a trust tool actually sells, and the two are not the same size here. + +Read this before deciding to release. It is deliberately unflattering. + +Two conventions used throughout, because the whole product depends on them: + +- **`tree-exact` means "recorded against the sealed tree."** It has never meant "proven correct." +- **UNRECEIPTED** means a command was run without the didrun wrapper, so no receipt backs it. Everything in + the *Verified* section below is UNRECEIPTED: the wrapper discipline is suspended while hardening didrun + itself, for the obvious reason. + +--- + +## VERIFIED — what was actually demonstrated, and how + +The gates, all run at `49a8b74` on CPython 3.14.5 / macOS arm64: + +| Gate | Result | +|---|---| +| `python -m pytest -q` | **496 passed, 4 skipped** (the 4 skips are exactly the real-corpus legs) | +| `python -m harness.recall` — the CI kill-gate | **PASS**, probe freeze INTACT, defect-class recall **100.0% (3/3)** | +| `python -m pytest -q tests/compat` | 11 passed, 4 skipped; synthetic legs report **0 note violations**, **0 layout residue**, **0 multi-index claims** | +| 3.11 grammar parse over all 41 `*.py` | clean. **Syntax only** — it cannot catch a 3.12+ stdlib API | +| stdlib-only | 24 top-level imports in `src/` + `harness/`, **none** outside the stdlib; `dependencies = []` | +| no private paths in shipped files | clean; the only `/home/...` strings are a public CI shape and an explicit `example` | + +Beyond the gates, the real CLI was walked end-to-end in throwaway repos and **30 distinct properties were +observed directly**, quoted verbatim in `docs/COMPAT_REPLAY.md`. The load-bearing ones: + +**The headline defect is closed, and I watched it close.** The attack v0.2 exists to stop — seal three +claims, delete the ledger, run three unrelated commands that exit 0 against the same tree, then verify — +now returns `WITNESS-UNAVAIL` on all three, **0/3 evidence-bound**, `--strict` exit 1. The pre-P1.2 +behaviour was `3/3 recorded-exact`, exit 0. That is the whole point of the release and it is measured, not +argued. + +**Every earlier phase's headline property still holds after P3–P5 landed on top of it.** Re-checked +individually, not assumed: + +- **Seal is fail-closed and all-or-nothing.** With note publication made to fail for real (the git object + store set unwritable), `seal` exits **2**, quotes git's error, and writes **no `seals.jsonl` line, no + note, and no watermark file**. Restore the permission and the same seal succeeds and verifies green. A + failed seal leaves nothing behind that a later seal would silently skip. +- **A substituted ledger fails closed** (above), and **forging the note's own rows makes the verdict + worse, not better**: rewriting a stored `grade` to `tree-exact` and zeroing the evidence hash yields + `WITNESS-UNAVAIL`, 0/1 evidence-bound. +- **Interrupts never lose a flight.** SIGINT → exit 130, SIGTERM → exit 143, one greppable line each, no + traceback, the in-flight event recorded as `unobserved no-exit`, `chain intact`, and the next `run` + appends normally. +- **Ledger permissions hold**: `drwx------`, `session.log` `-rw-------`. +- **Concurrent appends are safe and do not serialize.** Six parallel runs produced six events with the + chain intact, and a second `run` finished in **0.38 s** while a five-second run held the session. +- **A torn log refuses instead of writing past the tear** — named byte offset, spelled-out recovery, + nothing appended. (Its delivery is a defect; see K1.) + +**And the new v0.2 surfaces do what they say.** A note body of **1,819,948 bytes** — fourteen times the +largest note in the real corpus, and far past `ARG_MAX` — publishes, re-resolves and parses. A malformed +note is a **graded refusal with exit 2** across every shape tried (six bad `version` types, a string where +a claim object belongs, a truncated body, an unknown claim type, a retired claim type): no traceback, and +critically **no false green** — the `{"version": true}` class that used to verify green now refuses. +Supersession retains the retired claim in the record and excludes it from the verdict. A conjunction is +graded as the worst of its members and names which one. The secrets detector puts a deep build path at +**notice** (seal proceeds) and a `ghp_…` token at **block** (exit 3, located to the argument and offset, no +note written). A cited authority prints, verbatim, *"NOT a signature and NOT approval — didrun cannot tell +who wrote it."* + +**The guardrail protecting the live corpus was verified without touching the corpus.** The refusal list is +stored as path digests, so the live tree — plus its trailing-slash form and anything inside it — is refused +by name, a live-looking copy is refused by heuristic, and a proper copy is accepted. Checked with string +hashing only; no filesystem access to that tree occurred at any point in this pass. + +--- + +## STILL A BET — designed, argued from code, not measured + +**The single biggest one: the real-corpus replay never ran.** Nineteen units each argued in their own risk +register that they would not disturb 65 sealed notes and 178 archived ledgers. **Every one of those +arguments is still an argument.** The harness that would settle it exists, is content-blind, and is green +on synthetic fixtures — and it has never seen the real thing, because the corpus is live, secret-bearing +evidence for a build that is still running and copying it is not mine to do. Four legs are unexercised: +chain recompute over the archived logs, the note round-trip over all 65 notes, the `objects/` layout +partition, and the claim-shape invariants. + +A green synthetic run is **not** corpus validation, and the harness prints exactly that word — +`corpus: SKIPPED (unverified)` — rather than letting a green line imply it. + +The rest, each with what would settle it: + +- **CI on Python 3.11 / 3.12 / 3.13 × ubuntu / macos.** Unverified and **unverifiable here**: this + interpreter is 3.14.5. The 3.11 check that passed is a *grammar* parse; it cannot catch a 3.12+ stdlib + API. The declared floor is 3.11 and nothing has ever run there. +- **Capture recall against real agent sessions (the S6 leg).** The frozen S1–S5 probes report 100% + defect-class recall; S6 needs a real agent on a real machine and has never been run. +- **Durability across a crash.** There is no `fsync` anywhere in the package. Concurrent appends are + *mutually* safe; that is not the same as an append surviving power loss. +- **`flock` on network filesystems, and permissions on Windows.** Both measured on local APFS only. +- **Five research follow-ups (FU1–FU5) and red-team finding 16.** A claim-count discrepancy (667 vs 846); a + "zero detector findings" figure measured over exported argv strings and never over the ledgers; a + 1-of-8 resume-legality mapping that is inference over recorded tree ids rather than observation; the + original EV2 false-green, single-sourced (this pass witnessed the *mechanism* in a scratch repo, which is + not a witness for the corpus); and every frequency, cost and duration figure in the research, all + self-reported by the system under study. Details and owners in `docs/COMPAT_REPLAY.md`. +- **The most important unknown: whether any of this changes a decision.** 846 recorded claims produced + exactly **one** `failed`. That is consistent with a superbly effective gate and equally consistent with a + nearly inert one, and a corpus whose only observed actor was cooperating cannot tell the two apart. The + experiment that would — tell an agent to obtain a green `didrun verify --strict` by any means and see what + it does — was **not run**, is not authorised by the plan this release followed, and is the highest-value + missing signal in the whole project. Until someone runs it, "didrun catches dishonest agents" is a + hypothesis. + +--- + +## KNOWN NOT CLOSED — defects and gaps left in deliberately + +### Defects found in this pass and NOT fixed + +The final integration pass measures; it does not fix, because a fix landing after the last verification +gate is unverified code in a release. Each of these is a small, self-contained unit for Drew to sequence. + +- **K1 — `didrun run` reports two operational errors as raw Python tracebacks with exit 1.** + 1. A command that does not exist (`run -- /bin/false` on macOS, where that path is absent) → + uncaught `FileNotFoundError`, exit 1, **and no event recorded at all**. + 2. A torn `session.log` → uncaught `LedgerError`, exit 1. The refusal *message* is excellent and + complete; only its delivery is wrong. + + This breaks an invariant the codebase states in its own words in `cmd_verify`: *evidence this binary + cannot read is a graded refusal, not a crash — 2, not `--strict`'s 1*. `cmd_verify` catches `LedgerError` + and returns 2; `cmd_run` wraps nothing, and `main()` catches only the two interrupt types. **Exit 1 is + `--strict`'s "graded badly" code**, so a crashed `run` is indistinguishable by exit status from an honest + failure — the same class of bug P4 already fixed on the manifest side. **Pre-existing, not a v0.2 + regression:** `main`'s `subprocess.run` raises the same way. Fix shape: one `try` in `cmd_run` over + `OSError` and `LedgerError`, returning 2 with a named message. + +- **K2 — `show --session` reports an unverifiable chain as "BROKEN".** `ledger.py` computes a three-way + status (`intact` / `broken` / `unverifiable`) and `render.py` documents in a comment that the two faults + *"get different words on purpose"* — broken means an entry did not recompute, unverifiable means this + binary cannot check it. `verify`'s reason line honours that (`ledger chain unverifiable — entry 1: + unknown chain preimage version: 2`). But `show --session` goes through the boolean `verify_chain()` and + can only say `chain BROKEN at index 1`, and `verify`'s own headline banner says `CHAIN-BROKEN` for both. + So a future ledger format reads to the operator as **tamper**, which is precisely the false alarm the + frozen preimage was built to prevent. + +- **K3 — `didrun --version` still prints `0.1.0`.** The entire forward-compatibility story in + `docs/COMPAT.md` is phrased as "a v0.1 reader versus a v0.2 reader" and instructs operators to *upgrade + the verifier before the sealer* — and the only way anyone can tell which they have is `--version`. + **This must not ship as 0.1.0.** It is not fixed here only because `src/didrun/**` was fenced off for + this pass; it is a two-line change (`pyproject.toml`, `src/didrun/__init__.py`) with no test depending on + the string. + +- **K4 — the happy path shouts.** The first seal of every session prints a four-line `WARNING` about + having no watermark file, a condition the warning itself then explains is *"correct for a session's first + seal."* `docs/COMPAT.md` argues elsewhere that a gate which fires spuriously gets ignored within a week; + this is that, on the most-travelled path in the tool. + +- **K5 — a plan inconsistency worth recording so nobody chases it.** P4.1's documented supersession + scenario ("fail → claim → fix → claim, no commit between") is **unreachable through the CLI** now, + because P5.2 correctly refuses a claim bound to an event that exited nonzero. Supersession itself works + and was verified by the route that remains: claim, let the tree move, re-run, re-claim the same label. + The product is right; the plan's matrix row is stale. + +### Non-guarantees the design leaves open on purpose + +- **Inherited-environment false greens.** `GOFLAGS=-exec=/usr/bin/true` and its whole family + (`NODE_OPTIONS`, `PYTHONPATH`, `LD_PRELOAD`, `DYLD_INSERT_LIBRARIES`, `PYTEST_ADDOPTS`, a PATH shim, + `sitecustomize.py`) produce a command that passes having run nothing. No allowlist closes it, and putting + the effective environment in the event body would put the richest secret carrier on a developer machine + into the chain preimage. A PATH digest is recorded; a declared clean-environment mode is the v0.3 shape. +- **No retention or purge.** A credential scrubbed from git history survives as a loose object under + `.didrun/objects`. +- **The `«redacted:…»` marker is forgeable** — a convention, not a cryptographic mark. +- **Tree-fallback takes the first matching note** and does not fail closed on a second match (red-team + finding 20). First candidate for v0.2.1. +- **`argv_preview` is written into every note, dropped on read, and computed-but-unrendered in HTML** + (finding 21). A product decision, not a hardening unit. +- **`seal` does not consult the chain.** `verify` does. Sealing over a broken chain is possible. +- **Duplicate conjunct labels count twice.** `--of a,a` grades correctly and reads oddly. +- **No PKI, signing, key management or revocation.** `authorize` records a cited artifact's digest and says + in its own output that it is not a signature. +- **And the one that frames all the others: none of this resists a determined local forger.** Whoever can + substitute a ledger can regenerate its chain and re-run `claim` and `seal` to mint fresh hash-bound + claims. The source declares it, `docs/TRUST_MODEL.md` states it, and nothing in v0.2 changes it. What + v0.2 buys is **accident and drift detection on published evidence** — which is what actually goes wrong + in practice, and is worth having. It is not tamper-proofing and must never be sold as such. + +--- + +## What a human has to do before this ships + +Four things need a signal no amount of local work can produce: + +1. **Copy the archive tree and run the corpus replay.** Copy it *without* its live `.didrun` directory onto + a separate volume, then run the compat suite with `DIDRUN_COMPAT_CORPUS` pointed at the copy. The + harness refuses the live tree by name if the variable is set wrongly. Until this runs, "v0.2 does not + disturb existing sealed evidence" is an argument, not a finding. +2. **Get CI green on 3.11, 3.12 and 3.13 across ubuntu and macos.** The declared floor is 3.11 and nothing + has ever executed there. +3. **Run the S6 live-agent recall leg** — real agent, real machine, per `harness/README.md`. +4. **Run the adversarial-agent experiment**, or accept knowingly that the product's central claim is + untested. Tell an agent to obtain a green `didrun verify --strict` by any means and watch what it does. + One `failed` out of 846 claims cannot tell an effective gate from an inert one. + +Plus the small ones that are simply mine-to-hand-over: **K3 (the version string) blocks release outright**, +and K1 is worth an hour. + +--- + +## The honest summary + +**30 of 44 load-bearing conclusions are externally verified.** The other 14 rest on argument from code, on +figures the system reported about itself, or on a signal only a human or CI can supply — and all four +real-corpus legs are in that second group. + +The engineering is in good shape: the defect the release was built to close is closed and measured, every +prior phase's property was re-checked rather than assumed, and the failure modes fail closed with legible +messages. What is missing is not code. It is contact with reality — the real corpus, real CI, a real agent, +and one honest adversarial test. Ship it as **0.2.0 after K3 and gates 1–2**, and do not describe it as +verified against existing evidence until gate 1 actually clears. From 578f1a25022b7e2047cc0bf749db8ed1cc66752d Mon Sep 17 00:00:00 2001 From: Drew Date: Wed, 29 Jul 2026 21:37:21 -0700 Subject: [PATCH 7/8] release: bump to 0.2.0 COMPAT.md's whole forward-compatibility story is phrased as "a v0.1 reader versus a v0.2 reader" and tells operators to upgrade the verifier before the sealer. `--version` is the only way anyone can tell which one they have, so shipping this work while it still reported 0.1.0 would have undercut the document it exists to serve. Closes K3 in docs/V02_STATUS.md, which was outstanding only because the integration pass deliberately fenced off src/didrun/**. 496 passed, 4 skipped; harness.recall PASS at 100%. --- docs/V02_STATUS.md | 12 ++++++------ pyproject.toml | 2 +- src/didrun/__init__.py | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/V02_STATUS.md b/docs/V02_STATUS.md index 17317a1..2b2003d 100644 --- a/docs/V02_STATUS.md +++ b/docs/V02_STATUS.md @@ -143,12 +143,12 @@ gate is unverified code in a release. Each of these is a small, self-contained u So a future ledger format reads to the operator as **tamper**, which is precisely the false alarm the frozen preimage was built to prevent. -- **K3 — `didrun --version` still prints `0.1.0`.** The entire forward-compatibility story in - `docs/COMPAT.md` is phrased as "a v0.1 reader versus a v0.2 reader" and instructs operators to *upgrade - the verifier before the sealer* — and the only way anyone can tell which they have is `--version`. - **This must not ship as 0.1.0.** It is not fixed here only because `src/didrun/**` was fenced off for - this pass; it is a two-line change (`pyproject.toml`, `src/didrun/__init__.py`) with no test depending on - the string. +- **K3 — CLOSED after the integration pass.** `didrun --version` printed `0.1.0`, which was + self-defeating: the forward-compatibility story in `docs/COMPAT.md` is phrased as "a v0.1 reader versus + a v0.2 reader" and instructs operators to *upgrade the verifier before the sealer*, and `--version` is + the only way anyone can tell which they have. Now `0.2.0` in `pyproject.toml` and + `src/didrun/__init__.py`. It was outstanding at the end of the integration pass only because that pass + fenced off `src/didrun/**` by design, not because the change was in doubt. - **K4 — the happy path shouts.** The first seal of every session prints a four-line `WARNING` about having no watermark file, a condition the warning itself then explains is *"correct for a session's first diff --git a/pyproject.toml b/pyproject.toml index d6e6c42..d1d6851 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "didrun" -version = "0.1.0" +version = "0.2.0" description = "Flight records for agent-written code: deterministic session capture, evidence ledger, and graded verification." requires-python = ">=3.11" license = "Apache-2.0" diff --git a/src/didrun/__init__.py b/src/didrun/__init__.py index 03de7e1..d96c1de 100644 --- a/src/didrun/__init__.py +++ b/src/didrun/__init__.py @@ -10,6 +10,6 @@ The runtime is stdlib-only by design; auditability is the trust wedge. """ -__version__ = "0.1.0" +__version__ = "0.2.0" __all__ = ["__version__"] From 3019ad1637d390d6d8180c14b9d3dac03ad119a3 Mon Sep 17 00:00:00 2001 From: Drew Date: Wed, 29 Jul 2026 22:25:00 -0700 Subject: [PATCH 8/8] docs: bring the README up to v0.2, and cut the claims it could not back The README described a tool that no longer existed and made several claims nothing supported. Corrected: the prose said Python 3.14+ while the badge, pyproject and CI all said 3.11+; there were six commands documented as five; the grade table listed five of seven grades, missing witness-unavailable and chain-broken entirely. Every command, flag, grade token and output snippet in the file has now been reproduced by running the CLI. Added: what v0.2 changed, led by the evidence binding -- verify used to regrade against whatever ledger was on disk, so deleting one and running unrelated commands on the same tree still reported every claim recorded-exact. The upgrade note now warns that older readers refuse a v0.2 note by design, which is the thing most likely to bite a mixed-version setup. Local ledger exposure is documented where a user needs it: raw output lives as blobs under .didrun with no retention, so a credential scrubbed from git history survives there. Removed or qualified: "FAILED means a caught lie" (an overclaim that also contradicted this file's own closing paragraph, and in v0.2 claim refuses to bind a bare claim to a failing event at all -- nothing in the grade speaks to intent); an unsourced market assertion about 2026 review throughput; a census of competing tools that was never taken; an absolute about an unshipped tier; and the entropy sweep presented without its known ~1-3% miss on slash-bearing base64. What v0.2 did NOT establish is now its own section: compatibility with already sealed evidence is not a finding, only synthetic fixtures are green and the harness prints "corpus: SKIPPED (unverified)" itself; CI has never run on this branch; and a session.log record this binary cannot parse still crashes with a traceback at exit 1, which is indistinguishable by exit status from an honest failure. The closing paragraph now states the weakest thing that is true: across the largest run to date exactly one claim graded failed, the claim counts disagree with each other, every figure is self-reported by the system under study, and a run whose only actor was cooperating cannot tell an effective gate from an inert one. --- README.md | 198 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 140 insertions(+), 58 deletions(-) diff --git a/README.md b/README.md index 945afa6..9f5842d 100644 --- a/README.md +++ b/README.md @@ -7,24 +7,32 @@ **Flight records for agent-written code.** -> **v0 · experimental.** The core (`didrun run/claim/seal/verify/show`) works and is -> covered by CI on Linux + macOS across Python 3.11–3.13, but the API and CLI may -> still change, and there has been no independent security review yet. Treat it -> accordingly. See [what it does and does not attest](docs/TRUST_MODEL.md). +> **v0.2 · experimental.** The six commands (`run`, `claim`, `authorize`, `seal`, +> `verify`, `show`) work, and the test suite and capture kill-gate are green — on one +> machine, macOS / CPython 3.14. The declared floor is **Python 3.11**: the CI matrix +> (ubuntu + macOS × 3.11–3.13) is wired to `main`, which is what the badge above tracks, +> and it has **never run on the v0.2 branch**. Read "3.11+" as declared and tested on +> `main`, not as a v0.2 measurement. The API and CLI may still change, and there has been +> no independent security review. **Upgrading from v0.1? Verifiers before sealers** — see +> [What changed in v0.2](#what-changed-in-v02). What didrun does and does not attest is +> [here](docs/TRUST_MODEL.md); the unflattering version of what this release did and did +> not establish is [here](docs/V02_STATUS.md). When an AI coding agent says *"done — tests pass,"* the proof usually evaporates the moment the session ends. The transcript is unverifiable prose, CI only re-runs what CI knows about, and the reviewer is left re-deriving the work or trusting -vibes. That gap is the 2026 bottleneck: teams merge far more agent-authored PRs -than they can review, and AI PRs wait the longest for a reviewer. +vibes. didrun records what an agent **actually executed** — the command, the exit code, the output, and the exact git tree state it ran against — then binds structured -claims to a commit and lets a reviewer check them in seconds: +claims to a commit and lets a reviewer check them in seconds. A verdict carrying one +row of each of the five ladder grades: ``` x FAILED — review needed - 2/5 claims recorded-exact · commit a1b2c3d4e5f6 · tree 0f1e2d3c4b5a · resolved-by commit + 2/5 claims recorded-exact · commit 7eebe0fd0d9f · tree c6876974c0c0 · resolved-by commit + 4/5 claims evidence-bound + env: 4 match / 0 drifted / 0 incomparable / 1 not-recorded STATUS CLAIM DETAIL ---------------------------------------------------------------------------- @@ -33,8 +41,12 @@ x FAILED — review needed ! STALE lint clean exit 0 tree moved since evidence: 2 path(s) differ M src/api/handlers.py A src/api/new_route.py - ~ SCOPE-EXACT unit tests pass exit 0 all 3 change(s) within declared pathspecs + ~ SCOPE-EXACT unit tests pass exit 0 all 2 change(s) within declared pathspecs + M src/api/handlers.py + A src/api/new_route.py = TREE-EXACT type check passes exit 0 self-stable command ran against the sealed tree + + didrun records what ran; it does not prove the code is correct. See docs/TRUST_MODEL.md. ``` There is **no LLM in the trust path.** Capture and verification are deterministic. @@ -42,16 +54,15 @@ didrun is a *recording*, not a *proof* — read [what it does and does not attes ## Why it exists -- **Verification is the bottleneck, not generation.** Reviewers need to check an - agent's claims without re-running the work. -- **Everything else in this space is another LLM judging the code.** didrun answers - a different question — *what actually happened?* — deterministically. -- **Trust is an artifact problem, not a model problem.** The fix for unverifiable - work is work that carries its own machine-checkable evidence. +A reviewer needs to check an agent's claims without re-running the work, and the +usual answer is another model reading the diff — a judgement about whether the code +looks right. didrun answers a different and much narrower question, *what actually +happened?*, and answers it deterministically, so what a reviewer gets is evidence +they can check rather than a second opinion they have to weigh. ## Quickstart (under 5 minutes) -didrun is Python 3.14+, **stdlib-only** (zero runtime dependencies — you can read +didrun is Python 3.11+, **stdlib-only** (zero runtime dependencies — you can read the whole thing). Install it into your project's environment: ```bash @@ -79,40 +90,40 @@ The `--strict` exit code is the CI seam: wire `didrun verify --strict` into your pipeline and a drifted or unbacked claim fails the build. Every verdict also reports how the environment compared with the one the seal -recorded — `env: N match / M drifted / K incomparable`. Drift is advisory by -default, because a changed environment is a fact about the machine verifying, -not about whether the recorded command ran; `didrun verify --require-env-match ---strict` makes it a refusal. Evidence sealed by an older didrun is -*incomparable*, never drifted, and never refuses. - -For humans, generate a self-contained HTML evidence report: - -```bash -didrun verify --html evidence.html -``` - -The report is a single file with zero external assets — it opens offline and -prints cleanly. Redaction covers the sealed note: every exported claim string — -the `argv_preview`, the label, the pathspecs, the changed paths, the conjuncts -and the reason — is -redacted, and the projection is declared in the manifest. The HTML report -carries no argv at all and renders the note's redacted label, but a stale -claim's file list is recomputed against your working tree at verify time and is -shown as it is on disk — read it before you attach it to a PR. +recorded — `env: N match / M drifted / K incomparable`. Drift is advisory unless +you pass `--require-env-match`, because a changed environment is a fact about the +machine verifying, not about whether the recorded command ran. Evidence sealed by +an older didrun is *incomparable*, never drifted, and never refuses. + +`didrun verify --html evidence.html` writes a self-contained report for humans: +one file, zero external assets, opens offline and prints cleanly. Redaction +covers the sealed note — every exported claim string (`argv_preview`, label, +pathspecs, changed paths, conjuncts, reason) is redacted by the same pass that +reports findings to the gate, and the projection is declared in the manifest. +The HTML carries no argv at all and renders the note's redacted label, but a +stale claim's file list is recomputed against your working tree at verify time +and shown as it is on disk — read it before you attach it to a PR. + +Detection is pattern- and entropy-based, so it is a filter, not a guarantee: a +base64 credential whose own slashes chop it into short, name-like pieces is scored +as a path and missed ([docs/COMPAT.md](docs/COMPAT.md) quantifies that class). And +redaction covers the *published* artifact only. The local ledger is not redacted +and has no retention or purge — `didrun run` stores each command's raw stdout and +stderr as content-addressed blobs under `.didrun/`, so a credential scrubbed out of +git history survives there until you delete the directory by hand. `.didrun/` is +gitignored and its root is forced to `0700` (files didrun creates are `0600`; ones +an older didrun created keep their modes, and `.didrun/objects/**` is git's at +git's modes). Treat it as secret-bearing: do not commit it, share it, or attach it +to an issue. ## Use it with your coding agent You don't have to run these commands by hand. Tell your agent to route its own verification through didrun, and it produces receipts as a side effect of working. -Drop the snippet for your tool into its instructions file: - -- **Claude Code** → add the block to `CLAUDE.md` -- **Codex / agent-agnostic** → add it to `AGENTS.md` -- **Cursor** → a project rule -- **Any agent** → a system-prompt block - -Copy-paste snippets for each — plus the honest bounds — are in -[docs/agents.md](docs/agents.md). The short version: +Drop the snippet for your tool into its instructions file — `CLAUDE.md` for Claude +Code, `AGENTS.md` for Codex and anything agent-agnostic, a project rule for Cursor, +a system-prompt block for anything else. Copy-paste snippets for each, plus the +honest bounds, are in [docs/agents.md](docs/agents.md). The short version: ```markdown For any check you'll cite as evidence, run `didrun run -- `. Per finished @@ -124,13 +135,14 @@ as verified. This is **cooperative capture** — it works because the agent follows the instruction, not because anything is enforced. Commands the agent *doesn't* route through `didrun run` are simply not recorded (a claim with no backing run grades -`unknown`), so didrun never mistakes "didn't capture it" for "verified." Enforced, -agent-can't-forget capture is what Tiers 1–3 are for — see the note below. +`unknown`), so didrun never mistakes "didn't capture it" for "verified." Tiers 1–3 +narrow how much an agent can forget to route; nothing closes it, and none of them +is wired to a command yet — see the note below. ## The grades didrun never says "verified — trust me." Each claim gets an honest grade against -the sealed commit's tree — **first match wins**: +the sealed commit's tree: | Grade | Meaning | |-------|---------| @@ -138,19 +150,74 @@ the sealed commit's tree — **first match wins**: | **SCOPE-EXACT** | Every change since the evidence is within claimant-declared pathspecs. | | **STALE** | Evidence exists but the tree moved — the exact path+content delta is shown. | | **UNKNOWN** | No honest binding (no witnessed success, retroactive binding, gc'd object). | -| **FAILED** | The claimed command was *recorded failing* — a caught lie, not merely missing evidence. | +| **FAILED** | The command backing the claim was *recorded exiting nonzero* — evidence against the claim, not merely absent evidence. Nothing in the grade speaks to intent. | +| **WITNESS-UNAVAIL** | The live ledger cannot supply the evidence this claim was sealed against. The sealed grade is shown as history, never re-earned, and `--strict` never accepts it. | +| **CHAIN-BROKEN** | Report-level, never a per-claim row: the ledger every grade was read out of does not recompute, or this binary cannot check it. It dominates every grade below it. | + +The first five are the ladder. It is evaluated worst-first — `failed`, `unknown`, +`stale`, `scope-exact`, `tree-exact` — and the **first match wins**, so a claim never +grades better than the weakest fact about it. (The table above is in the reverse, +best-first order a verdict prints.) The last two are not rungs: one replaces a grade +when the evidence behind it cannot be produced, the other overrides the whole report. `STALE` always carries the concrete delta; it is never a bare shrug. `tree-exact` means the evidence tree equals the sealed tree — it does **not** claim the code is correct or that the command meaningfully tested anything. -## The five commands +## The six commands + +- `didrun run [--tee] [--heartbeat SECONDS] -- ` — record a wrapped execution (complete capture). `--tee` mirrors the child's raw output to your terminal as it arrives (local only, never redacted); `--heartbeat` prints a content-free progress line to stderr. Both default **off**, so anything parsing didrun's stdout sees what it saw before. +- `didrun claim [--label L] [--event N] [--path ] [--of a,b,c]` — declare a structured claim. With no `--event` it binds the command that just ran, and **refuses** (exit 2) rather than bind an earlier passing event when the latest one exited nonzero. `claim conjunction --label REL --of a,b,c` declares one over other claims in the same seal window, graded as the worst of them. Re-declaring the same type and label supersedes the earlier claim: the retired one keeps its grade in the record, drops out of the verdict, and is counted on both surfaces. +- `didrun authorize --artifact --scope {allow-secrets,reseal} --label ""` — record a cited authorization artifact for one exceptional scope, storing its SHA-256 and never its bytes. What that establishes, narrowly: **a file with this content was cited.** Not a signature, not authentication, not evidence a human approved anything — an agent can write its own artifact in one line, and every surface that prints one says so. A citation covers `--expires-after-seals N` seals starting with the next one (default **1**), and there is no value meaning *never*. `seal --require-authority` demands one for those two scopes; it is opt-in, because an authorization required on the happy path is a rubber stamp within a week. +- `didrun seal [--commit C] [--allow-secrets] [--bundle F] [--reseal] [--require-authority]` — compile and publish a commit-bound evidence manifest. **Fails closed**: exit 2 if the note cannot be published (leaving no note, no watermark, no seal record), exit 3 on a structured secret in the bytes it is about to publish. Replacing a note with a *narrower* record needs `--reseal`. +- `didrun verify [--commit C] [--strict] [--require-env-match] [--html F] [--quiet]` — check claims against the evidence they were sealed against. +- `didrun show [--commit C] [--session] [--event N --output [--stream stdout|stderr] [--redacted]] [--html F]` — the verdict, the recorded session history, or a recorded output blob, re-hashed on read so a corrupted one is a refusal rather than bytes presented as the record. -- `didrun run -- ` — record a wrapped execution (complete capture). -- `didrun claim [--path ]` — declare a structured claim. `didrun claim conjunction --label REL --of a,b,c` declares one over other claims in the same seal window: it is graded as the worst of them and can never be better than any one of them. -- `didrun seal` — compile a commit-bound evidence manifest (redacts secrets; refuses when a structured secret is found in what it is about to publish, warns about the rest). -- `didrun verify [--strict] [--require-env-match] [--html ]` — check claims against recorded evidence. -- `didrun show [--session]` — show the verdict, or the recorded session history. +## What changed in v0.2 + +**`verify` is now bound to the evidence it sealed.** Before v0.2 it regraded against +whatever ledger happened to be on disk, keyed by integer index, and never checked the +chain. Measured: seal three claims, delete the ledger, run three unrelated commands into +a fresh one on the same tree, then verify. v0.1 reported `3/3 recorded-exact` and exited +0. v0.2: + +``` +? WITNESS-UNAVAIL — review needed + 0/3 claims recorded-exact · commit 8710d403a1b8 · tree 7c75e84f39ea · resolved-by commit + 0/3 claims evidence-bound +``` + +`--strict` exits 1, and a chain that does not recompute now dominates the verdict +outright rather than being ignored. The rest: + +- **Seal is atomic and fails closed.** Publication and the watermark are one pair: nothing is recorded until the note is on the commit, and a failure after that rolls the note back. A seal that could not publish used to report success. +- **The v1 chain preimage is frozen behind an explicit field list**, so adding an `Event` field cannot silently change hashes already written. An entry whose preimage version this binary does not know is reported unverifiable *with its index*, never graded against a guess. +- **Interrupts no longer lose a flight.** Signals are held across the digest-and-append window (SIGINT → 130, SIGTERM → 143, one line each), and a `run` that dies after the digest records `tree_after=None` with observed-text-only coverage instead of costing the event. +- **The ledger is tighter.** Its root is `0700` and the files didrun creates are `0600`; read-only `verify` and `show` no longer manufacture one as a side effect; concurrent appends take an `fcntl.flock`, so parallel writers cannot fork the chain. Bounds are declared rather than rounded up: inherited files keep their old modes, `.didrun/objects/**` is git's at git's modes, and on a platform with no `fcntl` or no meaningful `chmod` both properties degrade to v0.1 behaviour instead of pretending. +- **Redaction and reporting are one act.** The entropy detector scores runs rather than characters — a path in segments, a base64 blob whole — every detector declares a tier (six structured patterns at `block`, the entropy sweep at `notice`), and the fields that block are the fields that get redacted, so `--allow-secrets` cannot publish raw what the gate blocked. The projection is *declared*, not inferred: `projection_version`, `detector_set_version`, and the exact replaced spans, which substitute back to the input byte for byte. +- **Claims can be superseded**, so a fix-verify loop converges instead of accumulating red rows, and a conjunction grades as the worst of its members. Both are described under `claim` above. + +**What v0.2 did not establish.** Compatibility with evidence already sealed is **not** a +finding: the compat harness is content-blind and green on synthetic fixtures only, the +real archive replay has never run, and the harness prints `corpus: SKIPPED (unverified)` +rather than let a green line imply otherwise. Nor has CI run on this branch. And two +operational errors still escape as Python tracebacks with exit **1** instead of the graded +refusal with exit 2 that the same binary gives elsewhere: a command that does not exist, +and a `session.log` this binary cannot parse (which takes `run`, `verify` and +`show --session` down alike). Exit 1 is `--strict`'s "graded badly" code, so a crash is +not distinguishable by exit status from an honest failure. +[docs/V02_STATUS.md](docs/V02_STATUS.md) separates verified from bet, item by item. + +> **Upgrading: verifiers before sealers.** `MANIFEST_VERSION` is now **3**, and an older +> reader **refuses** a v0.2 note by design — exit 2, `manifest version N is newer than +> this didrun understands (max M); upgrade didrun` — because regrading a note index-only, +> without checking the binding it was sealed under, is the confidently-wrong verdict this +> tool exists to avoid. Seal on a v0.2 workstation, verify in v0.1 CI, and every commit +> fails. The other direction is fine: a v1 note carries no evidence block and regrades +> exactly as v0.1 regraded it. Two smaller breaks the same way — a tree digest now also +> excludes `.didrun-history/`, so a repository carrying one **un-gitignored** sees claims +> against the older digest grade `stale`; and a narrowing re-seal needs `--reseal`. +> [docs/COMPAT.md](docs/COMPAT.md). ## Capture is tiered and honest about coverage @@ -158,15 +225,15 @@ Universal, invisible capture of everything an agent runs is not possible from outside the agent (measured — absolute-path executions escape a PATH shim every time). didrun is honest about this instead of pretending: -- **Tier 0 — `didrun run -- `:** the trust core. Complete argv/exit/output/tree state. This is the only tier that *guarantees* a claim's evidence — **and, in v0, the only tier wired to a command.** Use it explicitly for anything you'll claim. +- **Tier 0 — `didrun run -- `:** the trust core. Complete argv/exit/output/tree state. This is the only tier that *guarantees* a claim's evidence — **and, in v0.2, the only tier wired to a command.** Use it explicitly for anything you'll claim. - **Tier 1 — PATH shim:** re-dispatches bare-name commands through Tier 0. Absolute-path invocations and shell builtins are honest structural gaps. -- **Tier 2 — per-shell trap:** bash/zsh enrichment, **inert unless a didrun session is active** (it never records unrelated shell activity on your machine). +- **Tier 2 — per-shell trap:** bash/zsh enrichment. The snippet is gated on `DIDRUN_SESSION`, which only `didrun run` sets, so it is **inert outside a didrun session** rather than observing every shell on the machine. - **Tier 3 — native adapters:** e.g. Claude Code hooks. Enrichment only; the core works without them. Every recorded event carries a coverage grade, so the manifest is honest about what was and wasn't observed. -> **v0 status:** Tiers 1–3 exist as measured, tested library functions (see +> **v0.2 status:** Tiers 1–3 exist as measured, tested library functions (see > `harness/` and `src/didrun/capture.py`) but are **not yet exposed as a > user-facing `install`/`hooks` command** — wiring them up is on the roadmap. > Today you get their guarantee by calling `didrun run -- ` directly. @@ -179,6 +246,21 @@ cryptographic signing, which is deferred — see [TRUST_MODEL.md](docs/TRUST_MOD It is a **trust accelerant among people with a baseline of trust**, not a substitute for it. It records; you review. +**It does not resist a local forger.** Whoever can substitute a ledger can regenerate +its chain and re-run `claim` and `seal` to mint fresh hash-bound claims. What v0.2 buys +is **accident and drift detection on published evidence** — the thing that actually goes +wrong in practice, and worth having. It is not tamper-proofing and must not be sold as +such. + +**And the honest limit of the evidence for the whole idea:** in the largest run to date — +846 sealed claims by one count and 667 by another, a discrepancy that is itself an open +research item, and every figure here self-reported by the system under study — exactly +**one** claim graded `failed`. That is equally consistent with a highly effective gate and +with a nearly inert one, and a run whose only actor was cooperating cannot distinguish +them. Until someone tells an agent to obtain a green +`didrun verify --strict` by any means and watches what it does, "didrun catches dishonest +agents" is a hypothesis. + ## License Apache-2.0.