Keep the hash-key cell on the span, and seed every correlation

[maverox]

Review status: reviewed at 1c2a4c65 — mutations re-run, no defect found, no open asks. The head has since moved by one documentation-only commit (18e7811, a paragraph in the hash_seed module doc; no code); the reviewed code is 1c2a4c65. Stacked on #111; retarget to main when #111 merges.

Read this first

Three commits on top of #111, each standing alone:

commit	what	why it is its own commit
c679d8d	the span-cursor read survives thread teardown	a latent abort in the existing callers; the seam only made it live; cherry-pickable if the seam is ever reverted
d8c75ba	the seam's per-correlation memo moves from a correlation-keyed registry onto SpanContext	the relocation; per-correlation state has one home
1c2a4c6	every correlation seeds; the recorder's verdict is the one gate on the event	a behaviour decision distinct from the relocation, and one revert away

What production says this is for

Same-image self-replay of 2026.09.07.0 against its own 42 sealed tapes
(evidence with verbatim rows:
deja-handovers/results/replay-2026-09-07/divergence-rows.md): 353
correlations driven, 296 matched cleanly, 12,725 side-effect calls matched.

The control first. The id boundary matched 777 times and diverged 0;
grpc 23/0; imc 2720/2; superposition 1636/2. Seamed generation is exact
where it exists, so what remains is not a scoring artefact: the entropy
divergences that are left are coverage (unseamed generators), and the ordering
class below is real nondeterminism.

The largest divergence class is one shape, 21 identity skews (17 redis, 4
db). In one correlation the request considers the same four (connector,
wallet) pairs in both runs and visits them rotated:

recorded   paypal/paypal   klarna/klarna   cybersource/google_pay   stripe/apple_pay
observed   cybersource/google_pay   stripe/apple_pay   paypal/paypal   klarna/klarna

[A, B, C, D] recorded, [C, D, A, B] observed — same set, same members,
nothing added or dropped. So at the same call site, same rank and same
position the rollout-config key that is looked up names a different pair —

recorded  ucs_rollout_config_org_…_merchant_…_paypal_wallet_paypal_Session
observed  ucs_rollout_config_org_…_merchant_…_cybersource_wallet_google_pay_Session

— and everything downstream follows: the redis reply that was a value is Null,
the miss falls to the DB, and calls that happened in one run do not happen in
the other. It presents as identity skew rather than a value divergence because
each call is individually correct; they are just not the same calls.

What it is not: the rollout precedence walk (build_rollout_keys_by_precedence,
unified_connector_service.rs:1247, returns a fixed four-element Vec); a
Superposition filter (that path reads the DB configs table and makes no
Superposition call); an FxHashMap (constant seed); a membership test. The
divergence is upstream of the rollout lookup, in whatever orders the pairs that
populate router_data.connector, and that site is still being run down. A set
of members visited in a different order with nothing added or dropped is what
a collection walked under different hash keys looks like, and that is the class
#111 exists for: the seam makes the order reproducible, this stack makes it
hold for the whole life of a request, and the same-image replay after it
deploys is the test — this class should go to zero, or the mechanism is
something else and the rows will say so.

c679d8d — the cursor read cannot panic

ENTERED_SPANS is a RefCell<Vec<_>>, so the thread-local has a destructor,
and LocalKey::with panics once it has run. A read reached from another
thread-local's destructor during teardown — tokio, tracing and metrics layers
all tear down that way — is a panic inside a destructor: a double panic, and an
abort of the process. The two existing readers never ran inside a Drop, so
the defect was latent; the seam puts a reader on Default::default() for every
hash collection, including the ones built while a worker thread tears down.

Both accesses are now fallible and both failures answer "no span entered",
which every caller already handles. The door list in
tests/span_cursor_invariant.rs is unchanged.

The proof is not a red test. Revert try_with to with and the new test
does not fail — the test binary aborts (fatal runtime error: thread local panicked on drop, aborting, signal 6). The test's doc says so, because a green
suite cannot tell the two apart.

d8c75ba — the cell rides on the span

#111 memoized a correlation's key pair in a process-wide
HashMap<String, Arc<OnceLock<HashKeys>>>, filled on first use and evicted from
on_close. That is the shape #100 removed for the fork counters, and it is
wrong here for the same reason: per-correlation state with a home of its own
needs its own eviction clock, and every clock on offer is wrong for some path
that carries a correlation past the response.

Now SpanContext carries hash_keys: Option<Arc<OnceLock<HashKeys>>>, set in
on_new_span by a three-arm match: a span whose request_id matches its
parent's correlation inherits the parent's handle (one request, not two); a span
with its own request_id otherwise mints a fresh cell; any other span inherits.
Every span beneath the owner holds the same handle, so a spawn_fork tail
running under its fork_span() child reads the cell its request did, and the
cell is dropped with the last span that holds it. Nothing to evict, no
registry, no lock on the hot path: the read is the fallible cursor read plus an
OnceLock::get, and the only allocation is the cell itself, made once per
request when its span is created.

A correlation entered into deja-context without a span (deja_context::enter
alone) is deliberately not seeded — there is no span to hold a cell, and giving
it one would be the second home again. Hyperswitch has no installer of a
correlation outside the span layer, so this costs nothing real; the negative
test is the statement of the choice.

1c2a4c6 — one hashing regime, sampled or not

Both #111 and the first cut of this stack left a request the ingress sampled
out on the std hasher: #111 read the engaged deja-context id, which on_enter
leaves empty under a Skip decision, and the span move carried the same value.
That inverted a requirement written before either: recorded and unrecorded
traffic run one hashing regime, and only the event is gated on the decision.

The requirement is the replay premise. A recording stands for the production
request it was taken from; if the sampled few percent hash under one regime
(one pair per request) and the rest under another (one pair per collection), an
order-dependent behaviour anywhere in the service shows on one side and not the
other, and the tape is evidence about a regime only recorded requests see.
Recording is meant to be invisible; a regime split is the recorder changing
what it observes.

Cost, stated: std's per-collection key independence (rust#36481) is given
up inside every request rather than inside the sampled ones. The pair is still
drawn fresh per request from std's own entropy and is not observable before the
request runs, so there is nothing to precompute collisions against; the
residual is a request that leaks an order and then accepts more input, which is
not a shape this service has. This is a judgement, isolated in one commit so it
can be reversed in one.

The seam has no gate of its own. Every correlation draws through the same
hash_seed boundary, and the recorder's capture verdict — the one every other
boundary already goes through — answers no-op for a skipped correlation before
any sequence or occurrence is allocated, so a sampled-out request's draw leaves
nothing on the tape and nothing in the hook. An earlier cut bypassed dispatch
for that case on the belief that the hook would otherwise allocate an
occurrence; it would not, and a second gate whose only effect is unobservable
is the kind of guarantee that later reads as tested when it never was.

Verification

just verify exit 0, 818 passed. cargo +1.85.0 check on deja-runtime,
deja-context, deja, deja-derive: clean. Merges clean onto main de0a42a.

Every seam test now runs under a real DejaCorrelationLayer span, because that
is the only way a correlation gets a cell; the multi-thread test enters the span
for the first time on a worker thread and asserts the thread ids differ, which
is where a per-thread or per-enter cell would show.

Mutation matrix, read across the rows:

mutation	tests killed
one static OnceLock cloned into every owner	two_correlations_get_different_keys, plus three off-diagonal (a shared cell is already filled when the next test writes or draws)
cell minted per enter	the await, spawn, descendants and fork-tail tests
children do not inherit	descendants, fork-tail
same request_id re-stamped mints instead of inheriting	descendant_spans_share_the_owners_cell, and nothing else
different request_id inherits instead of minting	a_nested_span_with_a_different_request_id_mints_its_own_cell, and nothing else
cursor carries the engaged correlation (sampled-out → std)	both sampled-out tests
memo removed	the six sharing tests and the one-event integration test
constant seed	two_correlations_get_different_keys, nested-different
always seeded	outside_a_correlation_std_runs_verbatim, a_correlation_entered_without_a_span_is_not_seeded
draw under the cursor borrow	nothing — vacuous by construction: shared RefCell borrows are reentrant, so no behaviour differs; the draw-outside-the-borrow shape is defensive against a hook that enters a span, and is documented as that, not claimed as tested

Commit 1's mutation (try_with → with) is the SIGABRT above.

Review notes

• The sampled-out integration test (a_sampled_out_request_is_seeded_and_records_nothing)
  asserts the seeded arm first, so its "nothing on the tape" cannot pass
  vacuously; its killer is the recorder's gate, not code in this seam.
• deja_context::with_current_correlation_id (from Record the hash keys a correlation's collections iterate by #111) stays;
  current_correlation_id goes through it and a read that cannot panic during
  thread teardown is right for that caller on its own. Its doc no longer cites
  the seam.
• Not in this stack: a pre-seam-tape precheck (zero hash_seed events on a
  recording → refuse the run up front and say "re-record"), and the
  span_cursor_invariant-style gate against a correlation-keyed static in
  deja-runtime. Both are follow-ups.

[maverox]

Superseded by #135, which is stacked on #133 (the synthesis seam) and #134 (deja::synth).

What #135 keeps from here, because it was right and well-argued:

• SipHash-1-3 from siphasher and the reason — DefaultHasher is hardcoded to keys (0, 0), and std does not guarantee its algorithm across releases while record and replay can run different toolchains.
• HashKeys with a masked Debug, and the point that the masking stops at Debug because a tape holding *** substitutes nothing.
• draw_keys deriving from RandomState as a PRF over std's OS-seeded keys, rather than adding a second, weaker entropy source.
• Forwarding every write_* rather than leaning on the trait defaults.

What it drops, and why: the Default impl. Every cost this branch carried flowed from that one decision — Default on the hot path needed a per-correlation memo, the memo needed a correlation-keyed registry, and reading ambient state from inside HashMap::new() put a thread-local borrow on a path that also runs during thread teardown, where it aborts rather than panics. None of it is essential to recording a seed; it is the cost of not being asked. #135 asks: HashMap::with_hasher(deja::hash_seed("name")), no Default, no machinery.

The trade is honest rather than free — explicit touches the same number of sites, just more invasively, and needs zero machinery in exchange.

One thing that did not need to wait: the with_current_cursor fallibility fix has already landed on its own as #132.