Canonicalise capture by type, so array order means something - #108
Merged
Merged
Conversation
…ething A collection that is unordered where it is built is serialised into a JSON array that carries no record of whether its order ever meant anything. Rust seeds its hasher per process, so the same members come out in a different order between two runs of the same image, and every consumer downstream of the bytes is left working around a fact that was lost before it saw them. The only fix available to a consumer is to stop trusting array order everywhere, which is what we shipped in #99 -> #102 -> #104: it took the reference run from 26 blocking body differences to 1, and it made an INTENDED ordering change invisible in the same stroke. Canonicalise at capture instead, where the fact is still available. The Rust type knows what JSON forgets: a HashSet is unordered by type, a Vec is ordered by type, and once the value is a serde_json::Value the distinction is gone and no comparator can recover it. Serde's data model does not carry the distinction either — HashSet and Vec both call serialize_seq, which unlike serialize_struct has no type name. But its collection calls are generic over the concrete type at each position: serialize_field<T>, serialize_element<T>, serialize_value<T> and serialize_some<T>, with std::any::type_name::<T>() requiring no bounds. So a Serializer producing serde_json::Value can read the static type of every nested value at arbitrary depth, inside a #[derive(Serialize)] impl this crate never sees. That is the whole mechanism, and it needs nobody to declare anything. It is deliberately NOT "sort every array". Sorting an array whose order carries meaning destroys that meaning rather than preserving it: a removed sort on a ranked list would sort back to the same array, invisible in the verdict and unrecoverable from the tape — strictly worse than the tolerance it replaces. The type check is what bounds the loss to collections that had no order to lose, which is also what makes the sort round-trip safe, since such a collection deserialises the same from any order. The mutation that turns the type check into "always true" fails four tests, on purpose. The canonical order is by serialised form, the same rule and the same sort key the scorer's sort_as_bag already uses, so a tape canonicalised here is already in the order bag_canon would have put it in and no third notion of canonical enters the system. A sort, never a dedup. Wired at the eight capture sites the whole tape funnels through — the autoref capture! arm, value::serialize, the three result_serialize forms, and both ReplayCodec::capture impls — plus the two the recordable macro emits. That last one also removes a panic: json!({ "arg": &arg }) expands to to_value(&arg).unwrap(), so a serialisation failure in a delegated trait method took down the request, and the recorder is invisible instrumentation that must never do that. A shape this serialiser cannot express falls back to serde_json::to_value, so a capture is never worse than before. What it does not reach, recorded in the module so nobody re-derives it: a producer that iterates a HashMap into a Vec before returning has already lost the fact, and a Vec<Row> from a SELECT with no ORDER BY is unordered in the database's contract while the Rust type says otherwise. Both are producer-side and this correctly leaves both alone. Schema version 9. No field is added or removed; what args and result HOLD changes, which is why it takes a version rather than a comment — #100 changed how fork ids were composed without touching a field and old and new tapes silently disagreed. Stated on the constant: a pre-v9 tape's arrays are in whatever order the recording process's hash seed produced, canonical_args _hash puts array elements in the key positionally so a v9 candidate misses at every address rank against one, and a pre-v9 recording is therefore re-recorded rather than replayed — the same call made for #100. Any strict array comparison must gate on the recording's event_schema_version >= 9. LookupKey::args_hash documented itself as "order-independent", which is true of object keys and false of array elements, and is what let me believe for a while that permuted args were already safe. Corrected to say which half is which, because the next reader needs the distinction rather than silence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtydCoMHX5v44cdUN3ytek
… does The canonical serializer inserted a map's entries straight into serde_json::Map, which is sorted only when serde_json is built without preserve_order. The router is not: josekit, thirtyfour and ucs_common_utils enable serde_json/preserve_order transitively, so in the one binary this serializer exists for, Map is an IndexMap that keeps insertion order — and for a HashMap that order is this process's hash order, written onto the tape as JSON object key order. The note and the PR body said a HashMap was already canonical because serde_json sorts keys; that was true of deja's own build and false of the router's, and it was reasoned from hyperswitch's manifests rather than observed. Verified now with `cargo tree --no-default-features --features release --features v1 -e features -i serde_json` at the workspace root. A map's entries are now collected and emitted in key order regardless of the Map implementation. A struct's are not: field names are static and arrive in declaration order, deterministic on every run of a build, so the struct builder inserts straight into the map and pays nothing. The cost of the sort lands only on maps. The comparator was never at risk. The orchestrator's serde_json has no preserve_order, so every tape it parses lands in BTreeMap-backed maps and its equality, its args hash (which sorts keys anyway) and its bag sort key are all key-order-insensitive there. The test asserts on the entry list the builder sorts, not on the finished Map: on this crate's BTreeMap build the finished map is sorted whether or not anybody sorted it, and a test on it would pass for that reason alone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtydCoMHX5v44cdUN3ytek
maverox
added a commit
that referenced
this pull request
Sep 8, 2026
… mark A boundary whose CONTRACT says its reply carries no order — the rows of a SELECT with no ORDER BY, the keys of a redis SCAN — returns a Vec, and a type-directed canon (#108) correctly leaves a Vec alone. The fact about order lives in the protocol here, not in the type, and the only party holding both the reply and the protocol's evidence at capture time is the codec that captured it: the DB codec has the statement, the redis kit has the command the site recorded in its args. So the codec MARKS the reply, and never sorts it. The rows are recorded in the order they arrived: a service that (legally, wrongly) takes .first() of an unordered result must get on replay exactly what it got in the recording, and the wire rows the DB capture pairs to its serde rows are paired by index. Reordering at capture would break both. The mark is a reply canon — `bag:$.value[]`, naming the ResultCodec envelope's rows — derived per event and appended as a clause to the site's static declaration, so the comparator reads one declaration and nobody declares a path by hand. An undeclared site with nothing derived still records `declaration: None`, byte-identical to before. The DB side derives it from the statement in args.sql: a multi-row Ok whose SQL has no top-level ORDER BY. Nesting depth is tracked so a subquery's ORDER BY does not count, string literals are skipped, diesel's `-- binds:` tail is cut off, and anything the scan cannot follow answers "ordered", which leaves the result exactly as it was recorded before this existed. The redis side derives it from args.command against the protocol's own list of set-returning and scan commands; list, stream and sorted-set reads are deliberately absent because their order is the value. Only the redis kit pays the args clone this needs; every other preset's extractor is token-identical to before. The comparator change lands in this same commit on purpose. The args/result path resolved a declaration through a single-preset parser whose fallthrough is parse_project_canon, so a `bag:` clause — let alone one appended after a static clause with `;` — resolved to None and was never consulted. Routing it through canon_clauses is one line, and on its own that line would be actively harmful: Canon::equivalent returns false for a per-path bag clause by design ("consulted per difference, not here"), so the clause would resolve, be refused unconditionally, and a correctly stamped recorder declaration would start manufacturing divergences that look exactly like real ones. The per-path answer is therefore here too: the named collections are sorted on both sides — non-recursively, the members of the bag and not the arrays inside them — and the whole values compared. Whole-value equality is what refuses a change of shape: an array facing a missing key or a scalar cannot be made equal by sorting. A rule comparing how many collections each side reached was tried and could not be killed by any mutation, so it is not here; the counts remain for the tests, which use them to prove a path was reached and a sort did work before trusting an equivalence. pure permutation is absorbed with or without the mark. What the mark adds now is attribution; what it adds when that default is removed is that these results stay green for a stated reason instead of going red. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtydCoMHX5v44cdUN3ytek
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Read these two facts before the diff
This does not close the reference run's 25 ordering differences, and it is not
meant to. Both structural, both checkable:
sides as bytes —
RecordingBody<B: MessageBody>streaming chunks intocapture_chunk(&[u8])— because the order was serialised into them one frameupstream (
services/api.rs:466,serde_json::to_string(&response)). Atype-directed canon has nothing to read there, and the only thing that can be
done to bytes is sort every array, which this change refuses to do (below).
card_network_pms_for_client(hyperswitch
cards.rs:4155) doescard_network_hashmap.keys().cloned().collect()into aVecand pushes intoa
Vec;get_eligible_connectors'HashSetis collected into aserde_json::Value::Arrayone line before the call(
superposition_sdk_config.rs:392). Both types say ORDERED at the boundary,and this change correctly leaves both alone.
What this covers, now that the hash seed is becoming a seam
Since this was opened, the plan for the class of nondeterminism behind those 25
has changed: the service will draw its hash seed once per correlation through a
recorded, substituted seam, so every collection a request builds iterates the
same way on replay as it did in the recording. That is the right fix for
reproducibility, and it takes reproducibility out of this change's job. This
change was first described as "canonicalise so the tape is reproducible"; that
sentence is no longer what it does. What it still does, in the order a reviewer
should weigh it:
It covers collections the seam structurally cannot reach. The seam is the
service's own hasher, installed through a centralised collection type and a
clippy
disallowed-typesgate — and both reach the service's workspace cratesonly. A
std::collections::HashSetinside a dependency's type — a generatedmessage type, a third-party client's response — that reaches a capture still
constructs with
std::hash::RandomState, and nothing the service ships touchesit. The
type_namecheck here is the only thing that sorts those, because itreads the static type at every nested position regardless of which crate
declared it. The same holds for a dependency's
HashMapunder the router'sserde_jsonpreserve_orderbuild (see the second commit): only the map-keysort here makes it canonical.
It represents a set as a set. A
HashSethas no order by type. With areproducible hasher, its iteration order becomes a function of its insertion
sequence — so a refactor that builds the same set from a different source
order, a semantically empty change for a set, changes the tape and a strict
comparator flags an ordering difference on a type that has none. The canonical
sort here is what confines "an order difference means someone changed it" to
the collections whose order carries meaning. The seam makes order reproducible;
this makes it meaningful only where it is.
It makes tapes canonical across the transition. Recordings made before the
seam deploys carry sets in a canonical order already, and the comparator does
not have to know which side of the deployment a tape came from.
What changed
A collection unordered where it is built is serialised into a JSON array that
carries no record of whether its order ever meant anything. Rust seeds its hasher
per collection, so the same members come out in a different order between two
runs of the same image. Our answer so far (#99 → #102 → #104) was to stop
trusting array order in the comparator: it took the reference run from 26
blocking body differences to 1, and made an INTENDED ordering change invisible
in the same stroke.
Canonicalise at capture instead, while the type is still in hand.
Serde's data model does not carry the set/sequence distinction —
HashSetandVecboth callserialize_seq, which unlikeserialize_structhas no typename. But its collection calls are generic over the concrete type at each
position (
serialize_field<T>,serialize_element<T>,serialize_value<T>,serialize_some<T>), andstd::any::type_name::<T>()needs no bounds. So aSerializerproducingserde_json::Valuecan read the static type of everynested value at arbitrary depth, inside a
#[derive(Serialize)]impl this cratenever sees.
Second commit — a map's keys are emitted in key order whatever
serde_json::Mapdoes. The first commit inserted map entries straight intoserde_json::Map, which is sorted only whenserde_jsonis built withoutpreserve_order. The router is not:josekit,thirtyfouranducs_common_utilsenableserde_json/preserve_ordertransitively (verifiedwith
cargo tree --no-default-features --features release --features v1 -e features -i serde_jsonat the workspace root), so in the binary this serializerexists for,
Mapis anIndexMapthat keeps insertion order — for aHashMap,that process's hash order, written onto the tape as JSON key order. A map's
entries are now collected and emitted in key order; a struct's are not, because
field order is declaration order and deterministic on every run.
It is deliberately not "sort every array"
Sorting an array whose order carries meaning destroys that meaning rather than
preserving it: a removed sort on a ranked list sorts back to the same array —
invisible in the verdict AND unrecoverable from the tape, which is strictly worse
than the tolerance it replaces. The type check is what bounds the loss to
collections that had no order to lose, which is also what makes the sort
round-trip safe: such a collection deserialises the same from any order.
That refusal is enforced by the suite, not asserted in prose. Mutating the type
check to "always true" fails four tests.
Review notes
deja-runtime, notdeja: therecordablemacro emitsonly
::deja_runtime::paths and its integration test is in a crate thatcannot see
deja.dejare-exports it, so the vendor-facing surface isunchanged.
recordableemittedjson!({ "arg": &arg }), whichexpands to
to_value(&arg).unwrap()— a serialisation failure in a delegatedtrait method took down the request. The recorder is invisible instrumentation
and must never do that.
argsandresultHOLD changes, which is why it takes a version rather than a comment — Carry a correlation's recording decision on its span, and number fork buckets per callsite #100
changed how fork ids were composed without touching a field and old and new
tapes silently disagreed. A pre-v9 tape's arrays are in whatever order the
recording process's hash seed produced.
canonical_args_hashputs arrayelements in the key positionally, so a v9 candidate misses at every address
rank against one (the scorer's args-free twin pairing still repairs it, but it
is a miss). A pre-v9 recording is re-recorded rather than replayed — the
same call made for Carry a correlation's recording decision on its span, and number fork buckets per callsite #100.
event_schema_version >= 9. Not in this PR. The gate already has a sourceand no reader: the compactor collates it into
SessionManifest.event_schema_versions(deja-compactor/src/lib.rs:282, via:1616→:1739→:1498) andgit grep event_schema_versions crates/deja-orchestrator/srcreturns nothing.LookupKey::args_hash's doc said "order-independent" — true of objectkeys, false of array elements, on a field that sits on the key rather than
inside
Addressand is therefore part of every rank. Corrected to say whichhalf is which.
Verification
just verifyexit 0, 811 passed (11 new across the two commits).cargo +1.85.0 checkexit 0 on all five runtime crates. Every new test mutation-verified, andthe matrix checked for off-diagonal kills rather than assumed:
false(normalisation off)true(blanket-sort)a_sequence_keeps_its_order,..._byte_identical,..._round_trips_to_the_same_value,..._canonicalised_at_its_own_depthmembers_that_serialize_alike_are_both_kept, and nothing elsemap_entries_are_emitted_in_key_order, and nothing elsemembers_that_serialize_alike_are_both_keptcorrectly survives the firstmutation — a two-member set of identical renderings reads the same sorted or not
— and falls only to the third.
map_entries_are_emitted_in_key_orderasserts onthe entry list the builder sorts, not on the finished
Map, because on thiscrate's
BTreeMapbuild the finished map is sorted whether or not anyone sortedit.
two_equal_sets_now_hash_to_one_lookup_keyalso asserts that the twoUNCANONICALISED captures moved the key, so a run whose hash seed happened to
produce the same order for both sets cannot pass while claiming the fix worked.
The fallback in
to_value(a shape this serialiser cannot express degrades toserde_json::to_valuerather than tonull) has no test that reaches it: noshape
serde_jsonaccepts and this refuses could be constructed, because the keyrules were written to match. It is a safety net for a divergence not anticipated
here; a reachable one would be a bug to fix rather than to fall back from.