Skip to content

Canonicalise capture by type, so array order means something - #108

Merged
maverox merged 2 commits into
mainfrom
work/normalize-at-record
Sep 8, 2026
Merged

maverox merged 2 commits into
mainfrom
work/normalize-at-record

Conversation

@maverox

@maverox maverox commented Sep 4, 2026 •

Copy link
Copy Markdown
Collaborator

Read these two facts before the diff

This does not close the reference run's 25 ordering differences, and it is not
meant to.
Both structural, both checkable:

  1. No Rust type survives to the ingress seam. The reply crosses it on both
    sides as bytes — RecordingBody<B: MessageBody> streaming chunks into
    capture_chunk(&[u8]) — because the order was serialised into them one frame
    upstream (services/api.rs:466, serde_json::to_string(&response)). A
    type-directed canon has nothing to read there, and the only thing that can be
    done to bytes is sort every array, which this change refuses to do (below).
  2. The producers flatten before the boundary. card_network_pms_for_client
    (hyperswitch cards.rs:4155) does
    card_network_hashmap.keys().cloned().collect() into a Vec and pushes into
    a Vec; get_eligible_connectors' HashSet is collected into a
    serde_json::Value::Array one line before the call
    (superposition_sdk_config.rs:392). Both types say ORDERED at the boundary,
    and this change correctly leaves both alone.

What this covers, now that the hash seed is becoming a seam

Since this was opened, the plan for the class of nondeterminism behind those 25
has changed: the service will draw its hash seed once per correlation through a
recorded, substituted seam, so every collection a request builds iterates the
same way on replay as it did in the recording. That is the right fix for
reproducibility, and it takes reproducibility out of this change's job. This
change was first described as "canonicalise so the tape is reproducible"; that
sentence is no longer what it does. What it still does, in the order a reviewer
should weigh it:

It covers collections the seam structurally cannot reach. The seam is the
service's own hasher, installed through a centralised collection type and a
clippy disallowed-types gate — and both reach the service's workspace crates
only. A std::collections::HashSet inside a dependency's type — a generated
message type, a third-party client's response — that reaches a capture still
constructs with std::hash::RandomState, and nothing the service ships touches
it. The type_name check here is the only thing that sorts those, because it
reads the static type at every nested position regardless of which crate
declared it. The same holds for a dependency's HashMap under the router's
serde_json preserve_order build (see the second commit): only the map-key
sort here makes it canonical.

It represents a set as a set. A HashSet has no order by type. With a
reproducible hasher, its iteration order becomes a function of its insertion
sequence — so a refactor that builds the same set from a different source
order, a semantically empty change for a set, changes the tape and a strict
comparator flags an ordering difference on a type that has none. The canonical
sort here is what confines "an order difference means someone changed it" to
the collections whose order carries meaning. The seam makes order reproducible;
this makes it meaningful only where it is.

It makes tapes canonical across the transition. Recordings made before the
seam deploys carry sets in a canonical order already, and the comparator does
not have to know which side of the deployment a tape came from.

What changed

A collection unordered where it is built is serialised into a JSON array that
carries no record of whether its order ever meant anything. Rust seeds its hasher
per collection, so the same members come out in a different order between two
runs of the same image. Our answer so far (#99 → #102 → #104) was to stop
trusting array order in the comparator: it took the reference run from 26
blocking body differences to 1, and made an INTENDED ordering change invisible
in the same stroke.

Canonicalise at capture instead, while the type is still in hand.

Serde's data model does not carry the set/sequence distinction — HashSet and
Vec both call serialize_seq, which unlike serialize_struct has no type
name. But its collection calls are generic over the concrete type at each
position (serialize_field<T>, serialize_element<T>, serialize_value<T>,
serialize_some<T>), and std::any::type_name::<T>() needs no bounds. So a
Serializer producing serde_json::Value can read the static type of every
nested value at arbitrary depth, inside a #[derive(Serialize)] impl this crate
never sees.

Second commit — a map's keys are emitted in key order whatever
serde_json::Map does.
The first commit inserted map entries straight into
serde_json::Map, which is sorted only when serde_json is built without
preserve_order. The router is not: josekit, thirtyfour and
ucs_common_utils enable serde_json/preserve_order transitively (verified
with cargo tree --no-default-features --features release --features v1 -e features -i serde_json at the workspace root), so in the binary this serializer
exists for, Map is an IndexMap that keeps insertion order — for a HashMap,
that process's hash order, written onto the tape as JSON key order. A map's
entries are now collected and emitted in key order; a struct's are not, because
field order is declaration order and deterministic on every run.

It is deliberately not "sort every array"

Sorting an array whose order carries meaning destroys that meaning rather than
preserving it: a removed sort on a ranked list sorts back to the same array —
invisible in the verdict AND unrecoverable from the tape, which is strictly worse
than the tolerance it replaces. The type check is what bounds the loss to
collections that had no order to lose, which is also what makes the sort
round-trip safe: such a collection deserialises the same from any order.

That refusal is enforced by the suite, not asserted in prose. Mutating the type
check to "always true" fails four tests.

Review notes

  • Where it lives. deja-runtime, not deja: the recordable macro emits
    only ::deja_runtime:: paths and its integration test is in a crate that
    cannot see deja. deja re-exports it, so the vendor-facing surface is
    unchanged.
  • A panic removed. recordable emitted json!({ "arg": &arg }), which
    expands to to_value(&arg).unwrap() — a serialisation failure in a delegated
    trait method took down the request. The recorder is invisible instrumentation
    and must never do that.
  • Schema version 8 → 9. No field added or removed; what args and result
    HOLD changes, which is why it takes a version rather than a comment — Carry a correlation's recording decision on its span, and number fork buckets per callsite #100
    changed how fork ids were composed without touching a field and old and new
    tapes silently disagreed. A pre-v9 tape's arrays are in whatever order the
    recording process's hash seed produced. canonical_args_hash puts array
    elements in the key positionally, so a v9 candidate misses at every address
    rank against one (the scorer's args-free twin pairing still repairs it, but it
    is a miss). A pre-v9 recording is re-recorded rather than replayed — the
    same call made for Carry a correlation's recording decision on its span, and number fork buckets per callsite #100.
  • Any strict array comparison must gate on the recording's
    event_schema_version >= 9.
    Not in this PR. The gate already has a source
    and no reader: the compactor collates it into
    SessionManifest.event_schema_versions (deja-compactor/src/lib.rs:282, via
    :1616 → :1739 → :1498) and git grep event_schema_versions crates/deja-orchestrator/src returns nothing.
  • LookupKey::args_hash's doc said "order-independent" — true of object
    keys, false of array elements, on a field that sits on the key rather than
    inside Address and is therefore part of every rank. Corrected to say which
    half is which.

Verification

just verify exit 0, 811 passed (11 new across the two commits). cargo +1.85.0 check exit 0 on all five runtime crates. Every new test mutation-verified, and
the matrix checked for off-diagonal kills rather than assumed:

mutation tests killed
type check → always false (normalisation off) the 4 canonicalisation tests, and nothing else
type check → always true (blanket-sort) a_sequence_keeps_its_order, ..._byte_identical, ..._round_trips_to_the_same_value, ..._canonicalised_at_its_own_depth
sort → sort + dedup members_that_serialize_alike_are_both_kept, and nothing else
map-key sort → no-op map_entries_are_emitted_in_key_order, and nothing else

members_that_serialize_alike_are_both_kept correctly survives the first
mutation — a two-member set of identical renderings reads the same sorted or not
— and falls only to the third. map_entries_are_emitted_in_key_order asserts on
the entry list the builder sorts, not on the finished Map, because on this
crate's BTreeMap build the finished map is sorted whether or not anyone sorted
it.

two_equal_sets_now_hash_to_one_lookup_key also asserts that the two
UNCANONICALISED captures moved the key, so a run whose hash seed happened to
produce the same order for both sets cannot pass while claiming the fix worked.

The fallback in to_value (a shape this serialiser cannot express degrades to
serde_json::to_value rather than to null) has no test that reaches it: no
shape serde_json accepts and this refuses could be constructed, because the key
rules were written to match. It is a safety net for a divergence not anticipated
here; a reachable one would be a bug to fix rather than to fall back from.

…ething

A collection that is unordered where it is built is serialised into a JSON
array that carries no record of whether its order ever meant anything. Rust
seeds its hasher per process, so the same members come out in a different
order between two runs of the same image, and every consumer downstream of
the bytes is left working around a fact that was lost before it saw them.
The only fix available to a consumer is to stop trusting array order
everywhere, which is what we shipped in #99 -> #102 -> #104: it took the
reference run from 26 blocking body differences to 1, and it made an
INTENDED ordering change invisible in the same stroke.

Canonicalise at capture instead, where the fact is still available. The Rust
type knows what JSON forgets: a HashSet is unordered by type, a Vec is
ordered by type, and once the value is a serde_json::Value the distinction
is gone and no comparator can recover it.

Serde's data model does not carry the distinction either — HashSet and Vec
both call serialize_seq, which unlike serialize_struct has no type name. But
its collection calls are generic over the concrete type at each position:
serialize_field<T>, serialize_element<T>, serialize_value<T> and
serialize_some<T>, with std::any::type_name::<T>() requiring no bounds. So a
Serializer producing serde_json::Value can read the static type of every
nested value at arbitrary depth, inside a #[derive(Serialize)] impl this
crate never sees. That is the whole mechanism, and it needs nobody to
declare anything.

It is deliberately NOT "sort every array". Sorting an array whose order
carries meaning destroys that meaning rather than preserving it: a removed
sort on a ranked list would sort back to the same array, invisible in the
verdict and unrecoverable from the tape — strictly worse than the tolerance
it replaces. The type check is what bounds the loss to collections that had
no order to lose, which is also what makes the sort round-trip safe, since
such a collection deserialises the same from any order. The mutation that
turns the type check into "always true" fails four tests, on purpose.

The canonical order is by serialised form, the same rule and the same sort
key the scorer's sort_as_bag already uses, so a tape canonicalised here is
already in the order bag_canon would have put it in and no third notion of
canonical enters the system. A sort, never a dedup.

Wired at the eight capture sites the whole tape funnels through — the
autoref capture! arm, value::serialize, the three result_serialize forms,
and both ReplayCodec::capture impls — plus the two the recordable macro
emits. That last one also removes a panic: json!({ "arg": &arg }) expands to
to_value(&arg).unwrap(), so a serialisation failure in a delegated trait
method took down the request, and the recorder is invisible instrumentation
that must never do that. A shape this serialiser cannot express falls back
to serde_json::to_value, so a capture is never worse than before.

What it does not reach, recorded in the module so nobody re-derives it: a
producer that iterates a HashMap into a Vec before returning has already
lost the fact, and a Vec<Row> from a SELECT with no ORDER BY is unordered in
the database's contract while the Rust type says otherwise. Both are
producer-side and this correctly leaves both alone.

Schema version 9. No field is added or removed; what args and result HOLD
changes, which is why it takes a version rather than a comment — #100
changed how fork ids were composed without touching a field and old and new
tapes silently disagreed. Stated on the constant: a pre-v9 tape's arrays are
in whatever order the recording process's hash seed produced, canonical_args
_hash puts array elements in the key positionally so a v9 candidate misses
at every address rank against one, and a pre-v9 recording is therefore
re-recorded rather than replayed — the same call made for #100. Any strict
array comparison must gate on the recording's event_schema_version >= 9.

LookupKey::args_hash documented itself as "order-independent", which is true
of object keys and false of array elements, and is what let me believe for a
while that permuted args were already safe. Corrected to say which half is
which, because the next reader needs the distinction rather than silence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtydCoMHX5v44cdUN3ytek
@maverox maverox self-assigned this Sep 4, 2026
… does

The canonical serializer inserted a map's entries straight into
serde_json::Map, which is sorted only when serde_json is built without
preserve_order. The router is not: josekit, thirtyfour and ucs_common_utils
enable serde_json/preserve_order transitively, so in the one binary this
serializer exists for, Map is an IndexMap that keeps insertion order — and
for a HashMap that order is this process's hash order, written onto the
tape as JSON object key order. The note and the PR body said a HashMap was
already canonical because serde_json sorts keys; that was true of deja's
own build and false of the router's, and it was reasoned from
hyperswitch's manifests rather than observed. Verified now with
`cargo tree --no-default-features --features release --features v1
-e features -i serde_json` at the workspace root.

A map's entries are now collected and emitted in key order regardless of
the Map implementation. A struct's are not: field names are static and
arrive in declaration order, deterministic on every run of a build, so the
struct builder inserts straight into the map and pays nothing. The cost of
the sort lands only on maps.

The comparator was never at risk. The orchestrator's serde_json has no
preserve_order, so every tape it parses lands in BTreeMap-backed maps and
its equality, its args hash (which sorts keys anyway) and its bag sort key
are all key-order-insensitive there.

The test asserts on the entry list the builder sorts, not on the finished
Map: on this crate's BTreeMap build the finished map is sorted whether or
not anybody sorted it, and a test on it would pass for that reason alone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtydCoMHX5v44cdUN3ytek
@maverox
maverox merged commit 39b1dfe into main Sep 8, 2026
10 checks passed
maverox added a commit that referenced this pull request Sep 8, 2026
… mark

A boundary whose CONTRACT says its reply carries no order — the rows of a
SELECT with no ORDER BY, the keys of a redis SCAN — returns a Vec, and a
type-directed canon (#108) correctly leaves a Vec alone. The fact about
order lives in the protocol here, not in the type, and the only party
holding both the reply and the protocol's evidence at capture time is the
codec that captured it: the DB codec has the statement, the redis kit has
the command the site recorded in its args.

So the codec MARKS the reply, and never sorts it. The rows are recorded in
the order they arrived: a service that (legally, wrongly) takes .first() of
an unordered result must get on replay exactly what it got in the
recording, and the wire rows the DB capture pairs to its serde rows are
paired by index. Reordering at capture would break both. The mark is a
reply canon — `bag:$.value[]`, naming the ResultCodec envelope's rows —
derived per event and appended as a clause to the site's static
declaration, so the comparator reads one declaration and nobody declares a
path by hand. An undeclared site with nothing derived still records
`declaration: None`, byte-identical to before.

The DB side derives it from the statement in args.sql: a multi-row Ok whose
SQL has no top-level ORDER BY. Nesting depth is tracked so a subquery's
ORDER BY does not count, string literals are skipped, diesel's `-- binds:`
tail is cut off, and anything the scan cannot follow answers "ordered",
which leaves the result exactly as it was recorded before this existed.
The redis side derives it from args.command against the protocol's own
list of set-returning and scan commands; list, stream and sorted-set reads
are deliberately absent because their order is the value. Only the redis
kit pays the args clone this needs; every other preset's extractor is
token-identical to before.

The comparator change lands in this same commit on purpose. The
args/result path resolved a declaration through a single-preset parser
whose fallthrough is parse_project_canon, so a `bag:` clause — let alone
one appended after a static clause with `;` — resolved to None and was
never consulted. Routing it through canon_clauses is one line, and on its
own that line would be actively harmful: Canon::equivalent returns false
for a per-path bag clause by design ("consulted per difference, not
here"), so the clause would resolve, be refused unconditionally, and a
correctly stamped recorder declaration would start manufacturing
divergences that look exactly like real ones. The per-path answer is
therefore here too: the named collections are sorted on both sides —
non-recursively, the members of the bag and not the arrays inside them —
and the whole values compared. Whole-value equality is what refuses a
change of shape: an array facing a missing key or a scalar cannot be made
equal by sorting. A rule comparing how many collections each side reached
was tried and could not be killed by any mutation, so it is not here; the
counts remain for the tests, which use them to prove a path was reached and
a sort did work before trusting an equivalence.

pure permutation is absorbed with or without the mark. What the mark adds
now is attribution; what it adds when that default is removed is that
these results stay green for a stated reason instead of going red.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtydCoMHX5v44cdUN3ytek
@maverox
maverox deleted the work/normalize-at-record branch September 24, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant