[core] Drop pre-slot event ID support and preconditionGuard capability - #3519
Conversation
… resilient dispatch by default Three removals that stand on their own, split out of the v5 API switch. **Pre-slot event ids.** `maxEventSlot`, `findEventSlotGap`, and the step executor's slot observer went through a lenient decoder that answered "no position" for an id that is not a slot. That leniency *was* the pre-slot support, and it is the wrong shape now: a write whose `eventCount` is absent is indistinguishable, to a World, from one that honestly loaded nothing. They call `requireEventSlot` and throw. Skew protection is what makes it safe on Vercel — a run executes on the deployment that created it, so a build carrying this never replays a run created before slot ids. **The `preconditionGuard` capability.** Every World is now assumed to be able to refuse a stale replay-context write, so the three behaviors that keyed on the flag apply unconditionally: the per-step inline event-log delta stays enabled while the run has an open hook, an inline step's `step_started` claim is awaited before the body runs, and resilient dispatch stops consulting it. Before this only world-vercel declared it, so for world-local and world-postgres this is a behavior change rather than a no-op — they now pay the await-then-run claim while a hook is open, and get the inline delta in that same case. **`WORKFLOW_RESILIENT_STEP_DISPATCH`.** Off by default, `=1` to opt in. The publish races the create's verdict: a World that refuses the `step_created` sends the runtime back to replay while the payload-carrying message is already out, and nothing orders the refusal before the consumer's redelivery re-ensure. It was gated on the capability that no longer exists, and was already off for world-vercel, which declared it. `@workflow/world-sim` moves to slot ids with the runtime, since it drives the real one. The book's mint-ordered count goes 35/6/6 to 38/3/3 and append-only stays 41/0/0: four of the six reds staged a read missing an event the log already held, which under slots is a gap the runtime re-reads past. Also deletes `.changeset/windows-preload-timeout.md`, committed here by mistake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 1ca4d70 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
📊 Workflow Benchmarks❌ The benchmark run for Partial results from the failed run: commit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 228513ms → this run 190800ms (Δ -37713ms, -17%) 📜 Previous results (2)88553f8Thu, 13 Aug 2026 21:32:46 GMT · run logs
392953fThu, 13 Aug 2026 02:02:54 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 3 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
VaguelySerious
added
the
event-log-race-repro
Event Log Race Repro1 of 14 latest repro runs hit event-log regressions. Run History
Latest Scenario Breakdown
Latest Non-Completed Runs
|
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Reviewed the full diff, verified the sim-book baselines locally (mint-ordered 38/3/3, append-only 41/0/0, --no-fence 5), and re-ran the affected suites on the current head — all green. 88553f8 addressed every should-fix from the first review round (the #3404 doc revert, the 412-fence safety-argument framing, the latent null-runId mint in the sim facade, the stale scenario descriptions, and the world-vercel changeset). Two non-blocking threads remain open: the building-a-world capabilities paragraph should state slot allocation as a hard requirement of the v5 World contract (and the v5 GA docs PR should call out the breaking change for custom-world maintainers), and the pre-slot-run story for self-hosted worlds deserves a line in the release notes. The new world-local/world-postgres repro lanes added in 7d214db will give this PR's delta-with-open-hooks change its first storm datapoints on the worlds it actually affects.
Review feedback: `slotEventIds` could no longer read as an optional extra. The runtime reads a position out of every event id it loads and fails the run when it cannot, so a World that does not allocate slots cannot replay a single workflow — the flag was the only optional part, not the allocation. So the flag is gone, the same way `preconditionGuard` went: nothing consumed it (three Worlds declared it and two doc comments mentioned it), and a flag describing something mandatory only invites a World to leave it off and wonder why replay fails. The substance moves to `Storage.events`, which is what a World implements, and says which two properties the runtime actually relies on: density, and bump-and-report. It also now states the part that was implicit — allocate the position *at the commit*, because that is what makes a reader's log a prefix rather than a prefix with a hole in it. The guide says it as a requirement in both places it comes up: the capabilities paragraph, which now points out what is deliberately not a capability, and the Event ID Allocation section, which opens with a warning callout naming the failure a World author would otherwise meet at replay time. And `@workflow/world-testing` asserts it, which is the reviewer's point that a World written against earlier guidance passes conformance today and fails every replay later. The new case checks that every id decodes to a slot, that the run is dense from 1, and that ids are in canonical form — a World padding to a different width sorts its own log wrongly past ten events. Verified by breaking the expectation on purpose: red on world-local, then green again, and green on world-postgres. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Event Log Race Repro (world-local)6 of 14 latest repro runs hit event-log regressions. Run History
Latest Scenario Breakdown
Latest Non-Completed Runs
|
|
No backport to This is a deliberate contract/API change rather than a stability fix: it removes the To override, re-run the Backport to stable workflow manually via |
2 participants
Three removals split out of #3512, which needed all of them but is about something else (the v5 event API). Nothing here depends on v5.
slotEventIdsis removed along with the dead code. Nothing consumed it, and a flag describing something mandatory only invites a World to leave it off and then fail every replay. The substance moves toStorage.events: density, bump-and-report, and allocate the position at the commit, which is what makes a reader's log a prefix rather than a prefix with a hole in it@workflow/world-testingasserts it:numbers events by positionchecks every id decodes to a slot, that the run is dense from 1, and that ids are canonical (a World padding to a different width sorts its own log wrongly past ten events). Verified it can fail, not just passEvent id is not slot-numbered. It now fails in the suite instead. The GA docs need an explicit migration note; that is not in this PRmaxEventSlot,findEventSlotGapand the step executor's slot observer went through a lenient decoder that answered "no position" for an id that is not a slot. That leniency was the pre-slot support: a write whoseeventCountis absent is indistinguishable, to a World, from one that honestly loaded nothingrequireEventSlot(new, in@workflow/world), which throwspackages/coremove from hand-written ULIDs toslotToEventId(n)preconditionGuardcapability removedstep_startedclaim is awaited before the body runs (optimistic start suppressed on stale-sensitive batches)WORKFLOW_RESILIENT_STEP_DISPATCHoff by default=1to opt in, where it used to be=0to opt outcapabilities.preconditionGuard, which no longer exists, and already off for world-vercel, which declared itstep_createdsends the runtime back to replay while the payload-carrying message is already out, and nothing orders the refusal before the consumer's redelivery re-ensure.changeset/windows-preload-timeout.md(committed here by mistake)@workflow/world-simmoves to slot ids alongside the runtime, since it drives the real one. Its book goes 35/6/6 to 38/3/3 mint-ordered and stays 41/0/0 append-only: four of the six reds staged a read missing an event the log already held, which under ULIDs was indistinguishable from a complete read and under slots is a gap the runtime re-reads past. Baselines updated in the workflow header, DESIGN.md and the workbench README.#3512 is now stacked on this.