Skip to content

fix(kernel): EvidenceEmitter stores a step's events in call order (N123) - #573

Merged
LamaSu merged 7 commits into
masterfrom
fix/n123-emitter-call-order
Oct 6, 2026
Merged

LamaSu merged 7 commits into
masterfrom
fix/n123-emitter-call-order

Conversation

@LamaSu

@LamaSu LamaSu commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Row: N123 (steward #6352). EvidenceEmitter.addEvent stored events in the order their SHA-256 digests finished, not the order it was called.

The bug

addEvent awaited hashEvent (an async crypto.subtle.digest) before stepEv.events.push. So when calls overlap on one step, a slow digest puts an event after later ones. Master's runPrintJob makes such calls (printer-job.ts:196), and so do the IPP binding tests: they flaked in #474, #456 and #6307.

Reproduced at this branch's base. All five failures are in n123-before-fix.log:

  • refvertical's test (#6343);
  • three new gated tests;
  • the new production-path case: runPrintJob with a plain emitter, where execution_started's digest takes 3 s, gave [progress, started, progress, completed].

What it does and does not reach:

  • hashBundle hashes the sorted event hashes (spec canonical.ts:107-111). So the bundle hash, the kernel signature and verifyBundleHash do not depend on order.
  • The gateway's evidence envelope re-sorts events by id (evidence-envelope.ts:61, :85).
  • What the race does change is the order of a bundle's events array as returned and archived. Two public verifiers read that order and report timestamp_ordering_violation: evaluateEvidence (verifier oracle/evidence-evaluator.ts:123-128) and VerifierNode (network/verifier-node.ts:298-304).

The fix (packages/kernel/src/evidence-emitter.ts)

Each step keeps a chain, StepEvidence.stored.

  • An event is still hashed at its call, from the event as called. It is stored only after the step's earlier events have been stored or have failed.
  • The digests still run concurrently.
  • An event the step refuses (the LO-EV-9 binding checks) still rejects at once and takes no place.
  • A failed digest rejects only its own call, in its turn, and is never an unhandled rejection in the meantime.
  • Steps never wait on each other.

Every caller now gets call order: master's runPrintJob, JobRunner (already ordered at the caller since #502), the printer line (#521, likewise), scripts, and third-party kernels using the exported API.

Tests

Merging

🤖 Generated with Claude Code

LamaSu added 6 commits October 3, 2026 19:07
addEvent awaited each event's SHA-256 digest before storing it, so overlapping
calls on one step stored in digest-finish order: a slow digest put an event
after later ones. master's runPrintJob makes such calls (printer-job.ts:196),
so a print's signed bundle could list progress before started, and the IPP
binding test flaked (#474, #456, #6307).

Each step now keeps a chain: an event is hashed at its call, as before, and
stored only after the step's earlier events are stored or have failed. The
hashes still run concurrently, a refused event still rejects at once and
takes no place, a failed hash rejects only its own call, and steps never wait
on each other.

Tests: refvertical's reproduction (#6343) and four gated cases in
evidence-emitter-order.test.ts. The IPP binding test drops #571's ordered
emitter (carried here) and gains a production-path case: runPrintJob with a
plain emitter and a slow execution_started digest keeps the printer's order,
in the signed bundle too.
…he printer line's hunk

The printer line (#521 to #541) rewrites the abort case right after where it was, so the two
changes touched and conflicted. Moved unchanged.
…k 259)

Three HIGH findings on #573 (astra pack 259). Each was reproduced at dce82e8,
and on master's emitter, before this change:

- finalizeBundle hashed the live events array, awaited the signer, then copied
  the array, so an add stored meanwhile appeared in bundle.events without being
  covered by bundleHash. It now waits for every add called before it, refuses
  a step that lost an accepted add, and hashes and returns ONE snapshot. An add
  called later is stored after the bundle and never enters it.
- registerStep replaced a step while adds were pending, and those adds then
  resolved into a record nothing reads. It now refuses while adds are pending.
  A step whose adds have settled is still replaced, so a later run starts a
  fresh record. An add pending when its step is cleaned up now fails instead of
  storing.
- Steps were keyed by the joined string job:step, so ("a:b", "c") and
  ("a", "b:c") shared one record and one chain. They are now two map levels.

Tests: seven lifecycle cases in evidence-emitter-order.test.ts, six of which
failed before the fix. Mutations: 15 of 15 killed. Kernel: 1140 of 1140.
…s (pack 261)

Pack 261 found pack 259's HIGH 1 still open. The snapshot copied only the
array, so its event objects were the stored ones that callers still hold:
addEvent returns them and getEvents hands them out. A change made through such
a reference while the bundle was signed reached the bundle. Reproduced at
239b944: the bundle's event was the caller's object, and after its hash was
changed during signing, verifyBundleHash returned false.

The snapshot is now a structuredClone of the stored events, so the bundle holds
copies no caller can reach. A new test changes the caller's event (its hash and
payload) while the signer is gated. The bundle keeps the original, and both its
bundle hash and its event hash verify.

Mutations: 16 of 16 killed (a shallow snapshot is the new mutant).
Kernel: 1141 of 1141.
…resh full CI on a green master (steward #6465)
… clean head tree and a fresh full CI (steward #6548; a plain merge)
LamaSu added a commit that referenced this pull request Oct 4, 2026
@LamaSu
LamaSu marked this pull request as ready for review October 6, 2026 22:47
@LamaSu
LamaSu merged commit bf9f53a into master Oct 6, 2026
12 of 15 checks passed

This branch had an error being deployed

1 failed deployment
trusted-checks — 2808b1f1 Deployed Oct 6, 2026 by LamaSu via post-verdicts #167
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant