Record the hash keys a correlation's collections iterate by

[maverox]

Collections iterate in an order derived from the BuildHasher's keys, and std draws a fresh pair per collection. In an instrumented service that order reaches the wire, so a response assembled by iterating a map comes out differently on every run and replay diverges on the body. Boundary substitution cannot reach it, because the divergence is not at a boundary.

std offers one handle: k0/k1 are private and RandomState::new() is their only constructor, so the type has to be owned. DejaRandomState draws one key pair per correlation, records it through a Substitute boundary, and serves the recorded pair on replay. Outside a correlation it is std verbatim, per-draw increment included, so an uninstrumented process is unchanged.

Decisions worth reviewing

One pair per correlation, not per collection. A counter (seed+N) is not replayable: join_all polls N connector futures in a single task, so a counter advances in network-completion order — record k0+3, replay k0+7. Sharing a pair across one request gives up what rust#36481 buys within that request. It is emphatically not a constant seed: pairs are drawn per correlation from the entropy std seeds itself with, and a test asserts two correlations differ.

SipHash-1-3 from siphasher (new dependency), not DefaultHasher. DefaultHasher::new() is hardcoded to new_with_keys(0, 0) and std exposes no keyed constructor. Prefixing a DefaultHasher with the keys would compile, but std explicitly declines to guarantee its algorithm across releases — and record and replay are different processes, routinely built from different toolchains. An unstable algorithm would change iteration order while the recorded keys still matched: the same bug wearing its own fix. siphasher exists to provide the stable implementation std will not promise, and SipHash-1-3 is std's own algorithm, so the only thing that changes versus stock RandomState is where the keys come from.

A miss fail-stops, deliberately. Absorbing it would draw a fresh pair, every collection in the correlation would iterate on keys the recording never held, and every downstream body diff would read as a real divergence rather than the artifact it is. Execute is not the remedy either — it re-runs the draw, which is the nondeterminism being removed. The boundary comment says both, because this is the kind of fail-stop a later reader tries to be helpful about. Note the generic miss message still suggests replay_strategy = Execute; that advice is wrong here, and a dedicated message is a reasonable follow-up.

Boundary tag is hash_seed, not rng/id. Those are in the orchestrator's Pure tier, where a miss scores DeterministicMiss and does not block. A missing hash-key event must block. It is also not spelled bare "seed": that word already means materializing recorded state into containers (SeedEntry, InconclusiveSeedGap, the seeder crates).

Two hazards on the construction path

Default::default() runs on every HashMap::new, every with_capacity, and every serde-deserialised map, on 100% of traffic, sampled or not.

• It allocates nothing on either common answer. The correlation id is read by borrow, not cloned — current_correlation_id() is not on this path. Only the first collection in a correlation allocates, and only to own the map key. An uncontended lock, not std's ~1.9ns; the allocation is what mattered.
• It cannot panic. The context cell holds a String, so it has a destructor, and LocalKey::with panics once that has run. A map built during thread teardown would panic inside Drop — a double panic and an abort, which no guard contains. Both the TLS access and the borrow are fallible and fall back to std. This is the hardest rule deja has: recording never fails the service.

The memo is evicted when the correlation's span closes, on the same clock and for the same reason as the fork counters, or it grows by one entry per request forever.

Verification

just verify clean — exit 0, 57 suites, no clippy warnings. 12 tests across the six required properties, each mutation-verified against the whole suite:

Mutation	Tests that fell
Constant seed	two_correlations_get_different_keys (only)
Memo removed	5, incl. the one-event tape assertion and the tokio::spawn test
Always seeded	outside_a_correlation_std_runs_verbatim (only)
replay = Execute	a_hash_key_miss_fail_stops... (only)
try_with → with	whole binary SIGABRT, "thread local panicked on drop"

The off-diagonal is clean: properties 1 and 3 fall to opposite mutations, which is what proves the branch in Default::default() actually branches rather than passing vacuously. The last row is the teardown hazard reproduced rather than argued.

Open question, narrowed

The memo is a process-global map keyed by correlation, evicted at span close. The concern raised in review was that a post-response task still carrying the correlation after that span closed would re-draw, producing a second event and breaking the exactly-once property.

On the inventoried population this cannot happen, and the reason is the design's existing basis. Detached work on the request path is spawned .in_current_span(), which holds a cloned Span — a reference-counted handle — so on_close cannot fire until that task finishes. correlation_layer.rs:472-478 states it directly: "A span closes when the last handle to it is dropped, which is after every task that carried it has finished." The fork counters have relied on this since #105; if it were false they would already be wrong, independently of this PR.

The shape that would still bite is narrower: a task holding the correlation in deja-context without a span handle — an explicit enter/scope_snapshot on a future that was not .in_current_span()-wrapped. Whether any such call site exists on the request path is a vendor-side question, being checked against detached_spawn_inventory.rs's ALLOWED_BARE list. If that set is empty, this closes as "cannot happen today, guarded by the spawn-inventory test."

Moving the memo onto ContextSnapshot was proposed and is not recommended on current evidence: the snapshot has no equivalent of the span's last-handle-drop clock, and a snapshot built fresh rather than cloned from a parent gets its own cell — the same second draw, different trigger. It would trade a real guarantee for a weaker one.

Not covered

There is no full record→replay round trip through a populated LookupTable — building a correct entry by hand needs the Address stamping machinery. The two ends this PR owns are tested (the tape carries exactly one event with the real pair; a chosen recorded image reconstructs into the pair it holds, and a malformed one fails rather than re-drawing), and deja's table matching is left to deja's own tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_019FmXkygUmueraF9oR4oqwS

[maverox]

Superseded by #135, which is stacked on #133 (the synthesis seam) and #134 (deja::synth).

What #135 keeps from here, because it was right and well-argued:

• SipHash-1-3 from siphasher and the reason — DefaultHasher is hardcoded to keys (0, 0), and std does not guarantee its algorithm across releases while record and replay can run different toolchains.
• HashKeys with a masked Debug, and the point that the masking stops at Debug because a tape holding *** substitutes nothing.
• draw_keys deriving from RandomState as a PRF over std's OS-seeded keys, rather than adding a second, weaker entropy source.
• Forwarding every write_* rather than leaning on the trait defaults.

What it drops, and why: the Default impl. Every cost this branch carried flowed from that one decision — Default on the hot path needed a per-correlation memo, the memo needed a correlation-keyed registry, and reading ambient state from inside HashMap::new() put a thread-local borrow on a path that also runs during thread teardown, where it aborts rather than panics. None of it is essential to recording a seed; it is the cost of not being asked. #135 asks: HashMap::with_hasher(deja::hash_seed("name")), no Default, no machinery.

The trade is honest rather than free — explicit touches the same number of sites, just more invasively, and needs zero machinery in exchange.

One thing that did not need to wait: the with_current_cursor fallibility fix has already landed on its own as #132.