v0.2 hardening: bind verdicts to sealed evidence, fail closed on publication - #1
Merged
Conversation
P0 of the v0.2 hardening pack -- the foundation every later phase needs, so that adding fields and grades later cannot invalidate evidence already sealed. seal: publication and the watermark are now one atomic pair. _attach_note checks git's exit code instead of discarding it (a seal could report success when publication had failed), the body travels on stdin via -F - rather than -m (a real note body hit 1.6 MB, past ARG_MAX), and everything after publication runs inside a rollback that restores the commit's prior note. ledger: freeze the v1 chain preimage behind an explicit per-version field list, so future Event fields cannot silently change the hash of records already written. from_dict now filters unknown keys instead of cls(**d), so a newer ledger no longer crashes an older reader. Verified by recomputing a v1 entry hash under pre-change code: byte-identical. manifest: refuse unknown manifest versions rather than parsing them optimistically; carry the fingerprint version in band. tests: add the preimage golden, the seal-publication suite, and a content-blind compat replay harness that reads ledger structure and git metadata only, gated on DIDRUN_COMPAT_CORPUS so it skips cleanly when absent. 53 -> 96 passing, 4 env-gated skips; harness.recall still PASS at 100%.
…rrupted flights P1 and P2 of the v0.2 hardening pack. P1 -- the reason v0.2 exists. verify --strict was not bound to the evidence it sealed: it regraded claims against whatever ledger was on disk now, keyed by integer index, never called verify_chain, and sealed no event hash, tree or fingerprint into the note. Measured on v0.1: seal three claims, delete the ledger, run three unrelated commands (one /usr/bin/true) into a fresh ledger on the same tree, and it reported 3/3 recorded-exact, exit 0. That scenario now reports witness-unavailable on every claim and exits 1. Claims now carry an evidence block naming the entry that backed them, verify checks that binding, a broken chain dominates the verdict, and tree_delta returns None rather than [] when the delta cannot be computed -- an archived ledger used to produce the confident false sentence "evidence tree equals sealed tree". Manifest version 2; v1 notes still regrade exactly as before, down to the reason string. P2 -- operability. The ledger root is 0700 and read-only verify no longer creates one as a side effect. Output can stream (--tee/--heartbeat) instead of appearing only in a summary line after the command ends. Appends take an flock so concurrent writers cannot corrupt the chain. Interrupt safety needed two fixes, both found by reproducing the loss rather than reasoning about it. Signals are now held across the digest-and-append window, so a second Ctrl-C cannot unwind the recorder between the drain and the record (measured 12/30 flights lost before, 0/28 after). And the after-digest never costs the event on any path: a terminal Ctrl-C reaches the whole process group and kills the git the digest shells out to, even when the child exited cleanly, which lost a completed command's event 11/20 times. tree_after is None there and coverage says observed-text-only rather than claiming complete. COMPAT.md said a v0.1 reader parses a v2 note without crashing. Measured, it exits 2 -- correctly, by the refuse-never-coerce rule. Documented as the real forward incompatibility it is: upgrade the verifier before the sealer. 96 -> 193 passing, 4 env-gated skips; harness.recall still PASS at 100%.
This lands the v0.2 secrets and evidence work, plus the two defects an independent verification pass found in it. The detector tiering stands as designed: the six structured patterns block, the entropy sweep only notices. Measured over a real archive, the structured detectors had never fired once and 65 of 66 seals carried an override, so a gate that only ever produced false positives is a gate that gets switched off. A refusal is now scoped to the bytes that actually leave the machine, and findings carry a location instead of a bare count. The first defect was in the fix for that gate's false positives. Removing `/` from the entropy sweep's token class stopped a filesystem path being scored as one long token, and that part was measured. What was not measured is that `/` is also 1 of the 64 characters of base64: a credential whose slash-free runs are all shorter than the 24-character floor stopped being a candidate at all, and since replacement is driven off the findings, it stopped being redacted too. An AWS-shaped secret in a recorded argv went into refs/notes/didrun verbatim under a seal line reading "0 findings". Reproduced end to end before touching anything. So the separator is no longer decided in the character class. A run is taken over the class including `/`, and a per-run test decides whether that `/` is a separator or an alphabet member: a path is a sequence of names, base64 is not, and a name is mostly letters that are mostly lower case. One name is enough to call a run a path, which is the conservative direction — it can only return a run to segment scoring, so it cannot add a false positive the previous behaviour did not already have. Measured at exactly that: 0 new false positives over 46,403 real paths, with recall on slash-bearing base64 back from 74-87% to 97-99%. Both directions are now in MEASUREMENTS.md, and the sentence there that presented a one-directional measurement as two-directional is corrected rather than removed. The second defect was that blocking and redacting were different sets. Widening the scan to the export domain caught secrets in claim labels and delta paths and refused the seal over them — but nothing scrubbed those fields, so the override published them raw, under a refusal message that had already promised "export a redacted artifact anyway". Every exported claim string is now redacted by the same pass that reports it to the gate, so the two cannot drift apart again; the projection is declared per field in `redaction.fields`, and `scan_domain` says the fields were scanned. The seal still blocks exactly as before, which has its own test, because a fix that quietly opened the gate would be worse than the leak. Both fixes carry regression tests that were falsified: reverting each one turns the relevant tests red (8 and 4 respectively) and restoring it turns them green. Also here, from the same phase: the v2 environment fingerprint and its advisory-by-default drift comparison, claims bound to the recorded entry by chain hash so an archived or substituted ledger is visible rather than silent, interrupted flights recorded instead of lost, and the exported redaction projection made exactly reconstructible. Known and deliberately not closed, recorded in COMPAT.md rather than dropped: an HTML report recomputes a stale claim's file list against the working tree, so that list is not the note's redacted copy; the entropy sweep still misses roughly 1-3% of slash-bearing base64 by design and resists no attacker who knows the rule; and `reason`/`coverage` are exported unredacted because they carry only generated text today, which is a property of the generators and not an enforced invariant. Gates: 320 passed, 4 skipped. harness.recall VERDICT PASS, defect-class recall 100%, freeze INTACT. All changed files parse under Python 3.11. No runtime dependencies added. No private paths or internal names in shipped files. Gate runs are UNRECEIPTED - this repo keeps no didrun ledger.
Two claim-layer additions and one remediation of the reader they exposed. Supersession. A repair re-declares the same gate, and a later claim carrying the same (ctype, label) now retires every earlier one in the same seal window. That is what lets a fix clear a failed or stale claim with no intervening commit; before this the only escapes were seal-early, which needs a commit, and re-seal, which truncated the record, so discarding the ledger was the rational third. Supersession subtracts from the verdict and never from the record: every claim in the window is still graded and published, the retired entry keeps its grade and reason verbatim, and both surfaces mark it and count it. Retrying a gate until it passes is therefore visible rather than prevented, which is the honest bound and strictly better than the invisible truncation it replaces. The mark is validated on read, not trusted, and an all-superseded window grades unknown rather than falling through all([]) to a vacuous green. Conjunction. A new claim type that grades no evidence of its own: it names other claims in the same window by label and grades as the worst of them, so it can never come out better than any one of its members. It binds no event, and verify resolves it in a second pass against the results in that same note -- never against the live claims.jsonl, which would let a claim declared after the seal change an old commit's verdict. This substitutes for nothing. Every conjunct keeps its own witnessed exit code and its own tree comparison, and the design review's refusal to accept a transition marker in place of a recorded execution still stands untouched. The vocabulary change bumps MANIFEST_VERSION to 3, which is the first bump that is not additive in either direction, and docs/COMPAT.md states the break at full strength: an older reader meeting a v3 note raises out of the command rather than refusing gracefully. Because the bump makes the version guard the thing that whole story rests on, the guard was measured -- and it did not hold. Over 22 malformed note shapes the reader left an uncaught traceback on eleven and exited 1, the same code --strict uses for a note that graded badly, so a corrupt note could not be told from an honest failure by exit status. Four shapes were accepted outright: a note whose version was the JSON value true verified green at exit 0, because bool is an int subclass and True > 3 is False, and 2.5, 0 and -1 did the same. So an unreadable note is now a graded refusal with a reason naming what is wrong, for every shape rather than the remembered ones. version is read with the same rule the supersession mark gets and must be an integer of at least 1; the required fields are checked; and a claim type this binary cannot parse is a refusal instead of a ClaimError escaping verify, which closes from this end the hole COMPAT.md documents from the other. The two refusals stay distinguishable because the tree-fallback scan depends on it: a malformed body is skipped and counted so one foreign note cannot hide every good one, while a version above this reader's maximum propagates, since scanning past it would report an older note's verdict as if it were current. That split used to happen by accident, the scan catching bare Exception. This is fail-closed reading and not a forgery barrier; whoever can rewrite a note can rewrite a grade directly. Nothing legitimately published is newly refused -- every note in the world carries an integer version of 1, 2 or 3 and the required fields, and the compat corpus replays all three. Also redacts reason for every claim rather than exporting it as generated text. A conjunction's reason names a conjunct, which is operator-authored, so the field stopped being generated-only; covering it for all claim types means a later version that interpolates something into it is covered without anyone remembering. And a narrowing re-seal is refused unless --reseal is passed: git notes add -f force-replaces, and a seal scoped to all_claims[watermark:] was silently publishing a narrow note over a wide one and destroying a record that exists nowhere else. Three known-not-closed items are recorded in docs/COMPAT.md rather than dropped: a duplicate conjunct label is counted twice in the reason, the no-events refusal for a conjunction borrows a message about binding events, and a conjunction's redacted reason is not guaranteed to contain its redacted conjunct label as a substring since the two are scanned as separate fields. None can produce a grade the evidence does not back. Suite 435 passed 4 skipped, up from 405 with no regressions; harness.recall freeze INTACT, defect-class recall 100.0%, VERDICT PASS; the changed files parse at feature_version 3.11 and the refusal paths were exercised under a real 3.11.14; runtime dependencies remain empty.
Three changes to what a claim can honestly say, and one remediation pass over the surface the third one added. `didrun claim` with no `--event` used to bind to event 0 whatever event 0 was. In a session that ran a passing command and then a failing one, that produced a fully green sealed receipt over a suite which had just exited non-zero. It now binds the command that actually just ran, refuses when that command failed, and names the index, the exit code and `--event N` in the refusal, so the operator who really did mean an earlier event has a way to say so. A claim that lands `scope-exact` now says which pathspec would have made it tree-exact, naming the directories and files that fell outside what was declared. The hint is absent when the scope is already exact, so the reason line for an exact claim is unchanged. `didrun authorize` records that a file with a given SHA-256 was cited as the authority for one exceptional transition — `allow-secrets` or `reseal`, with no scope for an ordinary seal, because an authorisation on the happy path becomes a rubber stamp. It expires after a stated number of seals, and `--require-authority` refuses a bare override that no live citation covers. It grades `cited` and it buys exactly one property: retroactive fabrication becomes detectable, because the digest sits in a note sealed earlier. It is not a signature, it is not approval, and an agent can author its own authorization artifact in one line — the limitation sentence is printed beside every citation from a source constant so a note cannot edit it. The remediation. Publishing a new block of operator text re-opened a defect this repo had already closed once for claim labels: `principal` is `--principal` or `$USER`, it was published verbatim, and the pass that redacts authority fields never touched it. A token there refused the seal via the whole-manifest backstop under a message promising a redacted export, and then `--allow-secrets` wrote it into the git note unchanged with `redaction.fields` empty. Worse in the quiet case: one high-entropy value passed as both label and principal came out scrubbed in the first field and raw in the second, in the same JSON object, with nothing overridden. `principal` now goes through the same `redact_field` pass as the label and the cited path; the enumeration of which fields carry outside text is checked for completeness by a test instead of trusted, and a field neither table classifies is dropped from the note rather than published unscanned. The whole-manifest backstop that had been carrying this alone is now exercised too — removing the authority block from the scanned bytes used to break no test. Two of the six fields in the terminal's authority block were interpolated without `_sanitize`, because every use site spelled the call out by hand. `artifact_bytes` and `expires_after_seals` are numbers in a note didrun wrote and arbitrary strings in a note someone else edited, and a newline in either one fabricated whole verdict rows above the real table — rows the renderer never computed, carrying the words this vocabulary forbids, plus raw escape bytes into the operator's terminal. A note crosses machines over `git fetch`, so every field of it is outside data; the block now reads all of them through one sanitizing accessor, the way the HTML path already funnelled everything through one `esc`. The `seal` and `authorize` echoes had the same hole from the ledger side and are sanitized too — redaction is not sanitization, and a scrubbed label still carries whatever control characters surrounded the secret. `principal_basis` joins `grade` and `limitation` as a field pinned to its source constant at both bind and render. It qualifies the principal, so a note supplying "verified-ssh-signature" printed a claim of verified identity two lines above the sentence saying didrun cannot tell who wrote it. Left open and written down in docs/COMPAT.md rather than fixed quietly: a note's authority `scope` is still rendered as written and not re-validated against the scope vocabulary. It grants nothing — `--require-authority` gates against the local ledger, never against a fetched note — but a hand-edited note can misname the transition it claims to cover, and choosing what to render for an unrecognised scope deserves the same deliberate treatment the grade vocabulary got.
…osed The real-corpus legs did NOT run: DIDRUN_COMPAT_CORPUS was unset, so all four are recorded as BET (UNEXERCISED) rather than simulated. What did run: the full suite (496 passed, 4 skipped), the capture kill-gate (PASS, 100% defect-class recall), a 3.11 grammar parse over the tree, the stdlib-only and private-path audits, and an end-to-end walk of the real CLI in throwaway repos covering 30 properties — including every earlier phase's headline property, re-checked rather than assumed after P3-P5 landed on top of them. COMPAT_REPLAY.md carries the VERIFIED / BET / KNOWN-NOT-CLOSED split and the four gates. V02_STATUS.md is the plain-English release statement, including the five defects this pass found and deliberately did not fix (the integration pass measures; a fix landing after the last gate is unverified code in a release). COMPAT.md's forward-incompatibility warning was stale in two places: the Version 3 section described the unknown-claim-type traceback in the present tense after a later unit turned it into a graded refusal in this reader, and "the compat corpus replays all three versions" was true only of the synthetic fixtures. Both corrected.
COMPAT.md's whole forward-compatibility story is phrased as "a v0.1 reader versus a v0.2 reader" and tells operators to upgrade the verifier before the sealer. `--version` is the only way anyone can tell which one they have, so shipping this work while it still reported 0.1.0 would have undercut the document it exists to serve. Closes K3 in docs/V02_STATUS.md, which was outstanding only because the integration pass deliberately fenced off src/didrun/**. 496 passed, 4 skipped; harness.recall PASS at 100%.
The README described a tool that no longer existed and made several claims nothing supported. Corrected: the prose said Python 3.14+ while the badge, pyproject and CI all said 3.11+; there were six commands documented as five; the grade table listed five of seven grades, missing witness-unavailable and chain-broken entirely. Every command, flag, grade token and output snippet in the file has now been reproduced by running the CLI. Added: what v0.2 changed, led by the evidence binding -- verify used to regrade against whatever ledger was on disk, so deleting one and running unrelated commands on the same tree still reported every claim recorded-exact. The upgrade note now warns that older readers refuse a v0.2 note by design, which is the thing most likely to bite a mixed-version setup. Local ledger exposure is documented where a user needs it: raw output lives as blobs under .didrun with no retention, so a credential scrubbed from git history survives there. Removed or qualified: "FAILED means a caught lie" (an overclaim that also contradicted this file's own closing paragraph, and in v0.2 claim refuses to bind a bare claim to a failing event at all -- nothing in the grade speaks to intent); an unsourced market assertion about 2026 review throughput; a census of competing tools that was never taken; an absolute about an unshipped tier; and the entropy sweep presented without its known ~1-3% miss on slash-bearing base64. What v0.2 did NOT establish is now its own section: compatibility with already sealed evidence is not a finding, only synthetic fixtures are green and the harness prints "corpus: SKIPPED (unverified)" itself; CI has never run on this branch; and a session.log record this binary cannot parse still crashes with a traceback at exit 1, which is indistinguishable by exit status from an honest failure. The closing paragraph now states the weakest thing that is true: across the largest run to date exactly one claim graded failed, the claim counts disagree with each other, every figure is self-reported by the system under study, and a run whose only actor was cooperating cannot tell an effective gate from an inert one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the evidence-integrity gaps found while dogfooding didrun on a 15-day autonomous build.
Local install is intentionally NOT upgraded yet — an autonomous build is mid-run against didrun 0.1.0.