[core] Report the replay position on every event write - #3479
Conversation
🦋 Changeset detectedLatest commit: 11e7eed The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (1 failed)nextjs-turbopack-node (1 failed):
💻 Local Development (1 failed)nextjs-webpack-stable-node (1 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
❌ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 170730ms → this run 190606ms (Δ +19876ms, +12%) 📜 Previous results (2)64002feWed, 12 Aug 2026 17:26:48 GMT · run logs
2d1c9aeTue, 11 Aug 2026 23:11:38 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
VaguelySerious
changed the title
shalabhc
left a comment
shalabhc
left a comment
There was a problem hiding this comment.
Some non blocking comments below. Overall looks good.
-
My clanker kept reporting issues with correctness but I think they are solved by the new principle that if a prefix P produces an event E then any prefix P+Q cannot 'retract' E. Is this written down somewhere - it should be (at least my llm didn't see it).
-
Maybe can move 'hasMore' into the merge call?
Details
One merge primitive, four policies.appendUniqueEventsis the only thing that actually puts events in a log, and the two wrappers over it are justified — a delta extends the tail and carries a cursor, a report is a window strictly below the write and carries none, so one must not re-sort and the other must. What is duplicated is the policy:hasMore !== trueis decided independently atruntime.ts:2702,suspension-handler.ts:375(same three lines) andruntime.ts:950(plus cursor-unchanged), each re-deriving the reasoning in its own docblock, whilestep-executor.tsobserveSlotdiscards the report instead. Worth collapsing the three into oneabsorbReport(log, result)returning the added count.observeSlotcould not call it — it holds no log to merge into — which is the asymmetry worth a line of its own, since as written it reads as an oversight rather than a decision.
- out of date comments in a few places (possibly)
Details
**The id-scheme docblocks have not caught up to slots.** For the record, since the test files ask it too: world-local mints *slots* for every new run and keeps ULIDs only for runs that already had them (`drawEventSlot` — "a run whose visible events are ULIDs stays on ULIDs for life"), and slot ids are deliberately ULID-shaped so existing validators still accept them. Two comments predate that:appendUniqueEvents(:495) says "world-localorders by(createdAt, eventId)and deliberately re-mints keys so that the two diverge".eventSortKeyreturns the eventId itself for a slot run, so that is now true only of pre-slot runs. Keeping receipt order is still correct, just not for the reason given.insertEventByEventIdjustifies id-ordering with "Event IDs are ULIDs, so lexicographic string order matches commit order", and applies it with no slot gate — wheremergeReportedEventsis careful to checkmaxEventSlot !== undefinedfirst. On a slot run the conclusion happens to hold (fixed-width decimal sorts lexicographically), so this is wording rather than a bug, and only pre-slot runs are exposed.
- may be
SimStorecan mint slots now to validate in the slot scheme.
| ### Optional: Rejecting a Stale Write | ||
|
|
||
| ### Optional: The Event Creation Precondition Guard | ||
| A World that allocates positions never needs this: a write that named a stale position still commits, at the next free position, and the skipped events go back on the success response. A World whose event IDs are not positions has no such mechanism, and may fence instead. |
There was a problem hiding this comment.
A world can't "fence instead" IIUC - since the field to fence on is gone.
| * How much of its run's log a replay-context event creation had loaded when it | ||
| * decided to write, as the highest slot that log occupies. | ||
| * | ||
| * One integer says it because slots are dense: a writer that names slot N is |
There was a problem hiding this comment.
"Slots are dense" is true so I think holes are not possible but isSlotGapCheckEnabled (:717): says "A hole is a position claimed by a write that then failed, so most of them stand for an event that never happened."
I think it's talking about something else (a partially loaded log?) so should probably not call it a hole.
| slotSnapshot: { eventCount: scheduledAt }, | ||
| }); | ||
|
|
||
| // world-local mints ULIDs, so nothing the executor commits reads back as a |
There was a problem hiding this comment.
world-local mints ULIDs
still true?
or does this mean it mints slots (that look like ULIDs)
|
@shalabhc Fixed outdated comments |
…ONDITION_GUARD Replay-context `events.create` calls now carry `eventCount` (the number of events the replay had loaded) on every write derived from a loaded log, including the terminal writes that previously sent no snapshot at all: `step_completed`, `step_failed`, the suspension writes, and the replay-timeout `run_failed`. The ULID-era snapshot triple (`stateUpdatedAt` / `stateEventCount` / `stateCursor`) and its `WORKFLOW_PRECONDITION_GUARD` kill switch are gone. A watermark plus a count of events at or below it cannot distinguish a complete-but-stale view from a current one, and specVersion 6 replaces it with a dense position the World allocates at commit time.
…` references a function deleted by this PR, leaving a dangling doc reference.
This commit fixes the issue reported at packages/core/src/runtime/helpers.ts:498
## Bug
The JSDoc for `appendUniqueEvents` in `packages/core/src/runtime/helpers.ts` (line 498) contains:
```ts
* the newest event — see {@link latestEventStateUpdatedAt}.
```
This PR removes the `latestEventStateUpdatedAt` function. A repo-wide grep for `latestEventStateUpdatedAt` returns exactly one match — this `{@link}` — meaning the symbol no longer exists anywhere in the codebase. TSDoc/TypeDoc `{@link}` targets must resolve to a declared symbol, so this is a broken/dangling reference.
## Fix
The reasoning the prose points to ("nothing downstream may assume the tail is the newest event — it's a maximum, not the tail/count") now lives in `maxEventSlot`, which is exported from the same file and confirmed to exist (used in `step-executor.ts` and `suspension-handler.test.ts`). I retargeted the link to `{@link maxEventSlot}` so it resolves again.
Documentation-only change; no runtime impact.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>
Collapse the two skipped-slot merge call sites onto a shared absorbSkippedSlotReport, state the no-retraction principle the merge guards rest on, correct the stale hole/density docblock, and fix the stale-write docs section that told a World with no positions to fence.
|
No backport to This is an architectural change to the stale-write mechanism, not a defect fix: it makes every replay-context write report a slot position and deletes the To override, re-run the Backport to stable workflow manually via |
2 participants
Two changes that only make sense together: every replay-context write now reports the log position it was computed against, and the ULID-era snapshot triple that used to carry that information (plus its
WORKFLOW_PRECONDITION_GUARDkill switch) is deleted.1.
eventCountis sent whenever there is a position to reporteventCount(how many events the writer had loaded, i.e. the position it expects to land on minus one) previously reached the World from a handful of call sites. The hottest writes sent nothing at all, so the World had no anchor for the write it was most likely to be racing.Every write from the replay loop now goes through one seam in
runtime.tsthat stampsslotSnapshot()onto the params, so it is attached by construction rather than per call site:step_completed/step_failedand the rest of the loop's writesstep-executor.ts), whereknownSlot/observeSlotadvance the position across the step's own writes rather than reusing the stale entry valuesuspension-handler.ts)run_failed(replay-budget.ts), which is a terminal write made on a particular view of the log and now says which viewWrites with no loaded log to be stale against still send nothing, and a World must accept those:
start(), queued step bodies, hook resume,runs.*, out-of-band writers.maxEventSlot()also returnsundefinedunless every loaded event ID parses as a slot, so a run on the old ID scheme sends no position either.2.
WORKFLOW_PRECONDITION_GUARDand the ULID triple are removedstateUpdatedAt/stateEventCount/stateCursorare gone fromCreateEventParams, along withisPreconditionGuardEnabled(),latestEventStateUpdatedAt(), and the env var.PreconditionSnapshotParamsbecomesSlotSnapshotParams;preconditionSnapshotParams(events, cursor)becomesslotSnapshotParams(events).The triple approximated a position with a timestamp watermark plus a count of events at or below it. A complete-but-stale snapshot passes that check by construction: every event the writer holds is at or below its own watermark, so the count matches and no fence fires. The write that corrupts a log is exactly the one that view cannot see. A dense slot has no such blind spot, and specVersion 6 assigns it at commit time.
The 412 reception path is untouched:
PreconditionFailedError, the in-process replay restarts, the re-invocation fallback, andWORKFLOW_PRECONDITION_MAX_*all stay. This PR removes the client's ability to ask for the old fence, not its ability to survive one.Docs: the
WORKFLOW_PRECONDITION_GUARDsection in Runtime Tuning becomes "Stale-write rejection" (a World capability rather than a variable),PreconditionFailedErroris reworded, and the World-author guide replaces the triple's contract with theeventCountone.Problem:
step_startedis not duplicate-inertstep_startedis deliberately excluded fromONCE_PER_ENTITY_CLASSESin #3381, because a secondstep_startedfor a correlation ID is how a retry is recorded. That exclusion is load-bearing for retries and wrong for races.A losing lazy
step_startedthat lands anyway is therefore consumed as a real attempt: it incrementsattempt, and the retry budget for that step shrinks by one for a write that represented no execution. On a step whosemaxAttemptsis small, a duplicate storm can exhaust the budget without the body ever having failed. Nothing here fixes that. It needs a discriminator on the event that separates "this is attempt N+1" from "this is a duplicate claim for attempt N", which is a wire-format change and belongs in its own PR.Problem:
capabilities.preconditionGuardno longer means what its consumers read it to meanThe capability is documented as "this World rejects a stale create with 412". Against a v6 run that is now unreachable: the Vercel World evaluates its fence only for a create carrying an old-scheme snapshot, and this SDK sends none. A run created by a deployment carrying this SDK is v6, and skew protection keeps pre-v6 runs replaying on the deployments that created them, so no write from this SDK can be 412'd on the snapshot path.
preconditionGuard: trueis nonetheless kept on the Vercel World, because three runtime behaviors are keyed on it and all three are the conservative choice:requestInlineDeltawith an open hookevents.list)suppressOptimisticStartstep_startedclaim is awaited before user code runsresilientDispatchEligible(node VM)step_createdpublish is sequenced after the createRead as "a write can be refused" they are all still correct, and the slot path carries the load the fence used to. Read as "the fence is armed" the first row is now unpaid-for: it keeps an optimization enabled on the strength of a check that cannot fire. I did not change it here because doing so is a behavior change with its own blast radius, but it should be revisited, most likely by splitting the capability into "can refuse a write" and "fences on
eventCount".world-sim
The sim mints ULIDs, so a client on slot IDs sends it no position and its fence had nothing to read. The facade now reconstructs the snapshot from the pages the writer actually read. Two rules turned out to be load-bearing, both learned by getting them wrong:
run_started) gets no snapshot. Crediting it with its own write hands the fence a position below what the log holds and rejects the next concurrent write on a claim nobody made.AsyncLocalStorage. A facade-lifetime map hands a cold-starting replay the previous delivery's view and fences it forever for a log it never loaded, which is whatin-flight-after-decisiondid until this was fixed.Both books match the pre-change baseline exactly: mint-ordered 35 passed / 6 failed / 6 violations with the identical failing set, append-only 41 passed / 0 failed / 0 violations.
Validation
npx turbo run build45/45,pnpm typecheck43/43,pnpm test48/48 tasks (core 2048 passed + 3 expected-fail, world-sim 72/72, world-postgres 179/179).pnpm lintreports 4 errors, all pre-existing onmainand outside the changed files.No World other than the Vercel one ever read the triple, so nothing else in the repo changes behavior.