Hold the order-canonical recompute to the kernel's findings

[iemyashasvi]

Summary

A replay of a recording by the same image that made it failed 99 of 100 correlations, every one at $.rawConnectorRequest.value. The system under test echoes the connector request it built as a serialized JSON document; the headers object inside is iterated from a hash map, so its key order is seeded per construction and differs on every call. Same headers, same values, different bytes.

The kernel's diff_json reads that string as the document it is and reported nothing. The orchestrator's order-canonical recompute read it as bytes and reported one blocking leaf. classify_http_body_diff replaced the kernel's rows with the recompute's whenever bodies were present, so the less-informed engine had the final say.

The change

The recompute exists to decide WHERE a difference belongs once ordering is resolved, so a permutation is one difference at its collection rather than one per scattered position. It was never meant to decide WHETHER a difference exists; the kernel is the authority on that, and it alone applies the run's body allowlist.

The recompute's rows are now held to the kernel's findings, in both directions:

• It may not invent one. A recompute row survives only where some kernel row speaks about the same region. Regions are compared as declarations via the existing canonical_set_path, so either engine may describe one difference deeper (the kernel decoded an embedded document), shallower (the recompute stopped at the encoding), or at a different index of the same collection (alignment moved a changed member).
• It may not erase one. A reconciliation that would leave a non-empty kernel set with nothing is refused, and the kernel's own rows stand. Only a broken artifact produces that shape; the scorer must still not turn it into a pass.
• What it sets aside is named. Each path the recompute reported and the kernel judged the same notes OrderNondeterministicWarning (without counting a divergence, as ReplyCanonAbsorbed does), increments order_nondeterminism_warnings, and pushes a warning naming the path and why it did not count. A suite that leans on the kernel's judgment on every response no longer reads identically to one whose bodies agreed.

This fixes the class, not the instance: no encoding has to be taught to two engines for them to agree, and an allowlisted path can no longer block through an engine that never saw the allowlist.

On the counter: order_nondeterminism_warnings was declared, folded from a kind nothing emitted, reported on every scorecard, and hardcoded to zero. Its doc described retired UPDATE inherited-row ordering noise. It is repurposed with the doc stating what it meant before and that every scorecard since wrote zero, so no non-zero value has ever carried the old meaning.

What still blocks

Probed against the changed code:

Case	Kernel rows	Outcome
Value changed inside an embedded document	1	blocks
Array reordered inside an embedded document	3	blocks
Repeated same-named form keys reordered	1	blocks
Ordinary value change	1	blocks
Object key order inside an embedded document	0	not blocking, named as a warning
Distinct form keys reordered	0	not blocking, named as a warning
Top-level array reordered	2	absorbed by default, named, as before

The only differences newly tolerated are the two the kernel deliberately equates, JSON object member order and distinct-key order in a form body, and each is named on the scorecard.

Tests

Eight new tests. The invariant in both directions: if the kernel found nothing, nothing blocks; if the recompute accounts for none of the kernel's findings, the kernel's rows stand. The measured incident is pinned as the pipeline builds it, and its path is named. A real change inside the same echoed document still blocks. Path coverage is by segment, not string prefix. At scorecard level, the counter folds from the kind, counter_disagreements() is empty, and the warning names the path.

just verify (fmt check, clippy with warnings denied, workspace tests) is clean.

Relation to #94

#94 taught the recompute to decode embedded strings by importing the kernel's policy. This supersedes it with a change that removes the possibility of the two engines disagreeing at all, adds no string sniffing to the recompute, and is strictly more conservative (an array reordered inside an embedded document still blocks here). #94's decoding arm composes with this and could follow as a precision improvement, since with this in place it can no longer invent a block.

[maverox]

Reviewed by checking out the branch and running it: just verify is green, 950 passing.

The diagnosis is right and the change is the right shape. Two engines read the same bodies; only one of them sees through encodings and only one of them applies the run's allowlist; letting the less-informed one have the final say is the defect. Holding the recompute to re-describing rather than deciding fixes the class rather than the instance.

The allowlist consequence is the strongest part and slightly undersold in the description: the allowlist is applied inside the kernel, so before this change an allowlisted path could block through an engine that had never seen it. That is a correctness bug independent of the ordering incident, and it falls out of the same fix.

Verified rather than taken on trust

The tests are load-bearing. Two mutations, each killed by the test that should kill it:

mutation	killed by
reconciliation removed (Some(rows) — the bug restored)	a_recompute_row_the_kernel_never_reported_does_not_block, a_hash_ordered_map_inside_an_echoed_document_does_not_block
segment-aware prefix replaced with inner.starts_with(outer)	coverage_is_by_path_segment_not_by_string_prefix

canonical_set_path supports the alignment argument: it rewrites indices to [] and strips a trailing [], so $.items[1] and $.items[2] both canonicalise to $.items. The "alignment moved a changed member" case works as described.

One gap, and I think it should be closed before merge

The invariant is stated in one direction — the recompute may re-shape the kernel's findings and may not invent one. The code permits the other direction: it may erase one.

classify_http_body_diff does

let rows = canonical.as_deref().unwrap_or(&diff.body_diff);

so the reconciled rows replace the kernel's rather than merging with them. If reconciliation empties a non-empty kernel set, the kernel's own findings are discarded and nothing blocks.

This is reachable. Probed on the branch with bodies differing only at $.alpha while the kernel reports $.beta:

kernel rows = ["$.beta"]
reconciled  = []            <- kernel's finding erased, nothing blocks

Whether it is reachable in real data depends on the claim in the doc comment — "this engine recurses the same structure the kernel does, so every region the kernel can report in already has a row here". That is the assumption the safety of dropping rests on, and it is asserted rather than tested. order_canonical_diff is an independent traversal of the raw bodies, not a filter over the kernel's rows, so nothing in the types guarantees it.

Suggested addition, four lines, reusing the existing None path which already means "use the kernel's rows":

let reconciled = reconciled_with_kernel(rows, &diff.body_diff);
// The invariant holds in BOTH directions: the recompute may re-shape the
// kernel's findings and may not invent one — and may not erase one either.
// If nothing it produced speaks about any region the kernel reported, the
// re-shaping has failed and the kernel's own rows stand.
if reconciled.is_empty() && !diff.body_diff.is_empty() {
    return None;
}
Some(reconciled)

I applied this locally: all five new tests still pass, the probe above passes, 449 orchestrator tests green. It costs nothing and makes the invariant symmetric.

Smaller notes, neither blocking

• A kernel row at the root keeps every recompute row, since path_covers("$", x) is true for any x. That is the conservative direction and probably intended, but it is not stated anywhere.
• reconciled_with_kernel is O(recompute x kernel) per body. Fine at observed sizes; worth remembering if bodies grow.

Happy to approve once the erase direction is closed, or to hear why it cannot arise — if the superset property is guaranteed somewhere I have missed, a line saying so in the doc comment would do the same job.

[maverox]

Follow-up on my review above, and this one changes my recommendation rather than adding to it.

The tolerated differences are silenced, not absorbed

This repo already has a settled pattern for "an ordering difference that carries no meaning": absorb it and name it. ValueCanonAbsorbed for args/result, ReplyCanonAbsorbed for the HTTP reply path — both emitted, both counted, both keyed by who said the order carries no meaning.

The table in the description shows this change does not follow it:

Case	Blocks
Top-level array reordered	absorbed by default, named, as before
Object key order inside an embedded document	no
Distinct form keys reordered	no

The first is absorbed and named. The two new ones are dropped by filter — no kind, no counter, no ledger row, nothing. The PR's own test confirms it: assert!(classification.canon_absorbed.is_empty()).

That matters here more than it would elsewhere, because this codebase has already paid for the lesson: a kind without a folded summary field is invisible by construction, which is why 41 ReplyCanonAbsorbed once sat behind a headline of 3. A difference dropped by filter is a step further down — it is not even a kind.

The practical consequence is that the incident this fixes becomes unobservable. A run where all 100 correlations are silently tolerating hash-ordered header maps looks identical to a run where the echo is byte-stable. The thing worth knowing — that ordering tolerance is load-bearing for this suite — leaves no trace.

There is already a counter for exactly this, and it has never fired

// divergence/mod.rs:4414
let order_nondeterminism_warnings = 0u64;

Summary.order_nondeterminism_warnings is declared, has its folds(...) entry expecting kind OrderNondeterministicWarning, and is reported on every scorecard. It is hardcoded to zero and never incremented. OrderNondeterministicWarning appears exactly once in the tree — in that folds declaration. It is emitted nowhere.

I checked the effect on real data: across 136 runs in the API window, order_nondeterminism_warnings is 0 in every one, alongside inconclusive_races and undeclared_concurrency_warnings. The field reads as "no ordering nondeterminism has ever been observed", when what it means is "nothing has ever written here".

So the slot this change wants already exists, was designed for this, and is empty.

Revised recommendation

Two things before merge, not one:

1. The erase direction from my earlier comment — reconciliation must not empty a non-empty kernel set.
2. Name what is tolerated. Emit OrderNondeterministicWarning (or route through canon_absorbed with a source, matching ReplyCanonAbsorbed) for each row dropped because the kernel judged its region the same. The counter, the fold and the reporting are already in place; what is missing is the increment.

(2) does not change what blocks — the verdict stays exactly as the table describes. It changes whether a reader can tell the difference between "the bodies agreed" and "the bodies disagreed and we decided it did not matter", which is the whole reason the counter was added.

If the intent was global order tolerance with a warning, (2) is the warning half and it is currently absent.

[iemyashasvi]

Both taken, in 8044dae.

Erase direction. Added the guard as suggested: a reconciliation that would empty a non-empty kernel set returns None and the kernel's rows stand. Pinned by a_recompute_that_accounts_for_none_of_the_kernels_findings_yields_to_them. The doc comment no longer rests safety on the superset claim, and now states the root-path and pairwise notes.

Naming. Each set-aside path notes OrderNondeterministicWarning via note_kind (no divergence counted, as ReplyCanonAbsorbed), increments order_nondeterminism_warnings, and pushes a warning naming the path and why it did not count. Pinned at scorecard level by the_scorecard_names_what_the_kernel_judged_the_same, including counter_disagreements().is_empty().

One correction to the history of that field: it was not designed for this. Its doc described retired UPDATE inherited-row ordering noise, always zero on new scorecards. Repurposed, with the doc stating what it meant before and that no non-zero value has ever carried that meaning.

just verify green.

[maverox]

Re-review at 8044dae

Both findings from the last round are addressed, and I checked them by breaking the new guards rather than by reading them. just verify green: 952 passing, 0 failed (was 950 — the two new tests are the delta).

Each new guard is killed by exactly one test, and it is the right one:

mutation	result
if false && reconciled.rows.is_empty() && ...	a_recompute_that_accounts_for_none_of_the_kernels_findings_yields_to_them FAILED — 449 passed, 1 failed
drop note_kind + the increment	the_scorecard_names_what_the_kernel_judged_the_same FAILED — 449 passed, 1 failed

One open item, inline at the guard: the invariant is stated in both directions but enforced all-or-nothing. It is your call whether the symmetry is worth the test churn — everything else here is verified and I have no objection to it landing.

[maverox]

Reusing the retired counter — I can corroborate this from the other side. I measured order_nondeterminism_warnings across 136 runs earlier in this work and found zero on every one (bounded by the API's 200-row window), so the claim holds where I can see it.

Two things that make the reuse safer than the comment claims, worth having on the record:

• the old emitter is gone — OrderNondeterministicWarning now has exactly one emit site in the tree (5110);
• the counter feeds no verdict reason any more. The old "N order-nondeterminism warning(s) (non-blocking)" line is not in the code, so verdict.pass holding is structural, not a property of the new test's fixture.

Nit, outside the diff: docs/design/HANDOFF-confirm-9of9.md:44 still records a run with order_nondeterminism_warnings:1 under the retired meaning (payment_attempt div 2 demoted). Your wording is correctly scoped to scorecards written since the retirement, but a reader grepping the repo for this field lands on a doc that maps a non-zero value to the DB case. One line there would close it.

[maverox]

This closes the erase direction I raised, and the mutation confirms it is load-bearing. One thing it does not cover: the guard fires only when the reconciliation is completely empty, so a kernel row that nothing backs is still dropped silently whenever at least one other kernel row is backed.

Probe — kernel reports $.alpha and $.beta, stored bodies differ only at $.alpha:

PROBE reconciled = Some(["$.alpha"])
PROBE blocking_leaf_count = 1   (kernel reported 2)

$.beta is neither counted nor named — kernel_equivalent_paths collects unbacked recompute rows, so an unbacked kernel row leaves no trace at all.

The precondition, since it decides whether this is worth fixing: reaching it needs the bodies stored beside the diff to disagree with what the kernel actually compared, because this engine recurses the same structure and a consistent artifact puts a recompute row over every kernel region. That is the same precondition the total-erase case needs. So the argument is symmetry, not a new failure mode — adding the guard concedes the precondition is worth defending, and if it is, it is worth defending exactly.

The exact form drops the special case; keep any kernel row nothing speaks about:

let mut reconciled = reconciled_with_kernel(rows, &diff.body_diff);
// A kernel row that no surviving recompute row speaks about was not
// re-shaped, it was dropped; it stands as the kernel wrote it.
for found in &diff.body_diff {
    if !reconciled
        .rows
        .iter()
        .any(|row| paths_overlap(&row.json_path, &found.json_path))
    {
        reconciled.rows.push(found.clone());
    }
}
Some(reconciled)

I ran it: the probe above goes to blocking_leaf_count = 2, and 449 of 450 tests pass unchanged. See the note on the one that moves.

[maverox]

This is the part that answers the earlier objection — the newly-tolerated cases are now named rather than silenced. Confirmed killed by the_scorecard_names_what_the_kernel_judged_the_same when both lines are dropped.

Worth noting what that test covers beyond the counter itself: it asserts counter_disagreements().is_empty(), so the summary field is a fold of the kind rather than a number maintained beside it — which is the failure this file documents at kind_total.

Volume is fine too: kernel_equivalent_paths_seen is keyed by path, so the 100-correlation prism case emits one warning line with a count, not 100 lines. And the existing assert_eq!(card.summary.order_nondeterminism_warnings, 0) tests at 9897/10472 are a useful vacuity guard that the increment does not fire everywhere.

[maverox]

This is the one assertion that moves under the exact form suggested at the guard above, and it is worth being clear that only the mechanism changes, not the outcome.

is_none() asserts the refusal. Appending every kernel row to an empty reconciliation is the same thing as yielding to them, so the outcome this test goes on to assert — blocking_leaf_count == 1 — is identical under both forms.

The kernel_equivalent_paths.is_empty() assertion below would also need restating: $.alpha then gets named as a path the kernel judged the same, which is accurate for this fixture and arguably the more useful reading of it.

[iemyashasvi]

Exact form adopted, in ba0a2f1.

Guard. Row by row now: every kernel row no surviving recompute row speaks about is carried through as the kernel wrote it, and the None path means only what it did before this series. One refinement to the snippet: the unspoken-for rows are judged against the recompute's survivors alone rather than the growing list, so two unbacked positional rows in one collection cannot speak for each other and lose one. Your probe goes to blocking_leaf_count = 2.

Tests. The erase test is restated as a_kernel_row_nothing_speaks_about_stands_as_the_kernel_wrote_it: same outcome, the kernel's row blocks, and $.alpha is named as you read it. a_backed_kernel_row_does_not_decide_the_fate_of_an_unbacked_one pins the partial case with $.beta[0] and $.beta[1] both standing beside the backed $.alpha.

Docs. One historical note under the cycle entry in HANDOFF-confirm-9of9.md, saying the counter no longer carries that meaning and feeds no verdict reason.

Your two facts on the counter, single emit site and no verdict line, both hold in the tree. just verify green.