[world-vercel] Call the v5 event API, drop pre-slot event-id support - #3512
Draft
VaguelySerious wants to merge 6 commits into
Draft
[world-vercel] Call the v5 event API, drop pre-slot event-id support#3512VaguelySerious wants to merge 6 commits into
VaguelySerious wants to merge 6 commits into
Conversation
🦋 Changeset detectedLatest commit: 9af6f63 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (161 failed)astro-node (7 failed):
astro-quickjs (6 failed):
example-node (7 failed):
example-quickjs (6 failed):
express-node (7 failed):
express-quickjs (6 failed):
fastify-node (6 failed):
fastify-quickjs (6 failed):
hono-node (7 failed):
hono-quickjs (6 failed):
nest-node (6 failed):
nest-quickjs (6 failed):
nextjs-turbopack-node (7 failed):
nextjs-webpack-node (7 failed):
nextjs-webpack-quickjs (6 failed):
nitro-node (7 failed):
nitro-quickjs (6 failed):
nuxt-node (7 failed):
nuxt-quickjs (6 failed):
sveltekit-node (6 failed):
sveltekit-quickjs (6 failed):
tanstack-start-node (8 failed):
tanstack-start-quickjs (6 failed):
vite-node (7 failed):
vite-quickjs (6 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
Contributor
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 207603ms → this run 179271ms (Δ -28332ms, -14%) 📜 Previous results (6)6a1fe97Thu, 13 Aug 2026 01:39:05 GMT · run logs
43270e6Thu, 13 Aug 2026 01:10:01 GMT · run logs
6566c54Thu, 13 Aug 2026 00:13:00 GMT · run logs
621253bWed, 12 Aug 2026 23:43:54 GMT · run logs
f6b469dWed, 12 Aug 2026 23:24:04 GMT · run logs
cc47927Wed, 12 Aug 2026 22:04:37 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Contributor
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 3 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Contributor
Event Log Race Repro12 of 14 latest repro runs hit event-log regressions. Run History
Latest Scenario Breakdown
Latest Non-Completed Runs
|
… resilient dispatch by default Three removals that stand on their own, split out of the v5 API switch. **Pre-slot event ids.** `maxEventSlot`, `findEventSlotGap`, and the step executor's slot observer went through a lenient decoder that answered "no position" for an id that is not a slot. That leniency *was* the pre-slot support, and it is the wrong shape now: a write whose `eventCount` is absent is indistinguishable, to a World, from one that honestly loaded nothing. They call `requireEventSlot` and throw. Skew protection is what makes it safe on Vercel — a run executes on the deployment that created it, so a build carrying this never replays a run created before slot ids. **The `preconditionGuard` capability.** Every World is now assumed to be able to refuse a stale replay-context write, so the three behaviors that keyed on the flag apply unconditionally: the per-step inline event-log delta stays enabled while the run has an open hook, an inline step's `step_started` claim is awaited before the body runs, and resilient dispatch stops consulting it. Before this only world-vercel declared it, so for world-local and world-postgres this is a behavior change rather than a no-op — they now pay the await-then-run claim while a hook is open, and get the inline delta in that same case. **`WORKFLOW_RESILIENT_STEP_DISPATCH`.** Off by default, `=1` to opt in. The publish races the create's verdict: a World that refuses the `step_created` sends the runtime back to replay while the payload-carrying message is already out, and nothing orders the refusal before the consumer's redelivery re-ensure. It was gated on the capability that no longer exists, and was already off for world-vercel, which declared it. `@workflow/world-sim` moves to slot ids with the runtime, since it drives the real one. The book's mint-ordered count goes 35/6/6 to 38/3/3 and append-only stays 41/0/0: four of the six reds staged a read missing an event the log already held, which under slots is a gap the runtime re-reads past. Also deletes `.changeset/windows-preload-timeout.md`, committed here by mistake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every `events` read and write goes to `/api/v5/...`. The wire frame is unchanged, so the file keeps its name and its schemas: what the route selects is server behavior, not a format. On v5 the backend no longer writes the Step and Wait rows on the event path, so a v5 run reads back with no steps through `steps.list` until the read model is rebuilt from events. `runs.cancel` stays on `/v4/runs/cancel`: there is no v5 mirror of it. `WORKFLOW_EVENTS_TRANSPORT=ws` reaches v5 too, decided on the backend side. Its endpoint stays versioned on its own, which is about the frame envelope rather than which write path a frame reaches. `WORKFLOW_SERVER_URL_OVERRIDE` is pinned to the branch serving /v5 for the life of this PR and must be reverted to '' before merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…now live The retry ceiling is decided against a step's attempt number, and that number came back from the World: the step row carried a counter its start patch incremented. A World that stores only the log has no such counter, and the attempt is on no event unless the writer puts it there — so every attempt reported 1, no step ever exhausted its budget, and any failing step retried forever. The writer already knows the number (it is what the pre-body ceiling is enforced against), so it states it. Omitted when the caller supplied none, so a World that counts for itself is not overwritten by a guess. The e2e step assertions move off the storage listing. `--withData` was forcing it, for two reasons that both changed: `attempt` is now on the analytics listing, and a step's error is a payload no metadata listing ever carries — that one reads the `step_failed` event, which holds it on every World. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t corrupted Two invocations can run one step: a queue-dispatched step has no claim to lose, its message can be delivered twice, and both deliveries start a step that already exists. When the loser's body then fails behind the winner's result, the `step_retrying` it records repeats no class, so no consumer wants it and the replay was calling it divergence — failing a run that had finished. Measured rather than argued: with the backend's inline claim fenced and this reverted, `event-log-race-repro` corrupts 12 of 12 storm runs; with it, 0. It changes one deliberate expectation — a start behind a completion that repeats no class used to be divergence. The harness now prints a failing attempt's event-log tail. It reported `CORRUPTED_EVENT_LOG` and nothing else: the message the runtime failed on does not survive into the run row, and the observability tables are not populated for a preview backend, so which two writes collided had to be inferred from the shape of the code. Inferring it is how two diagnoses went wrong, and the tail is what showed the remaining failure is an infinite retry loop rather than the corruption it was assumed to be. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`writeDiagnosticsSidecar` writes it into the repo root on every e2e run, and the gitignore rule added alongside its sibling does not apply to a file already tracked. Committed here by mistake, from running the suite locally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Step row answered three questions at execution time that nothing answers on a World that materializes no step rows (the v5 event path): 1. **The input.** v4's step_started PATCH returned the row, whose inputRef was stored at step_created; the executor hydrates the body from that response. A bare start against the no-row path synthesizes its response from the event just committed, which carries no payload — so every queue-dispatched step hydrated `undefined` and failed with devalue's "Invalid input" before user code ran, retried, and failed again on the next delivery, forever. 2. **The attempt.** The row's counter was the retry-ceiling authority. With it gone the only cheap signal was the queue delivery count, which resets to 1 on every fresh publish — and every retry after step_retrying IS a fresh publish, so the ceiling's fast gate never opened, the writer stamped attempt 1 on every start, and a failing step never exhausted its budget. This is what turned the race-repro storms into 12/12 stuck-at-running. 3. **The start fence.** The no-row write path reads a write's refusal out of the slots it skipped, which requires the writer to name its log position — and the queued executor had none to name, so a start behind the step's recorded outcome was told it won and re-ran the body. The dispatcher — a replay that holds the loaded log — knows all three, so the step dispatch message now carries them: `stepInput` (the same serialized bytes step_created stored, when binary, within the resilient size bound, and on a CBOR transport), and a new `stepContext` with the 1-based attempt this dispatch asks for plus the dispatcher's slot snapshot. The background executor takes max(delivery count, declared attempt) as its ceiling input (still verified against the recorded start count before failing the step), seeds its writes with the snapshot, and hydrates from the response, then the message, then a step_created read-back — and refuses to run a body with no input at all rather than handing undefined to the deserializer. Every field is optional and advisory: old messages and old consumers behave exactly as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
steps.list,wf inspect, and the dashboard until the backend's read model is rebuilt from events.runs.cancelstays on/v4/runs/cancelWORKFLOW_EVENTS_TRANSPORT=ws(opt-in, off by default) forwards into v5 too, on the backend side: the socket hands each frame to the same events route the HTTP branch posts to, so a run gets the same write semantics over either transport. The WS endpoint stays versioned on its own (/websockets/v1), which is about the frame envelope rather than which write path a frame reachesWORKFLOW_SERVER_URL_OVERRIDEis pinned to the branch serving/v5for the life of this PR. Must be reverted to''before merge.Then three defects the first Vercel Prod run exposed. All 24 prod lanes failed the same ~14 tests while every Local Prod and Local Postgres lane was green on the same commit, so only the v5 route pin was implicated. Fixes:
attemptonstep_startedadd({attempt: 1})on the step row and the runtime sized its retry budget from theattemptthe response returned. There is no row on v5, so the World readseventData.attempt, which nothing sentattemptadded to the analytics steps listing server-side (startedCount AS attempt, the same number the row's counter held by construction), so the two retry tests no longer need--withDatato force the storage pathstep_failedevent, which holds it on every Worldnextjs-turbopackwf inspect stepsreport an empty step list rather than "cannot answer", because the fallback it drops to no longer holds stepsstep_retryingit records repeats no class, so no consumer wants it and the replay was failing a finished run withCORRUPTED_EVENT_LOGevent-log-race-reprocorrupts 12 of 12 storm runs; with it, 0stepCreatedis the SDK's exactly-once inline-ownership signal, and without a step row nothing held the claim, so every racing writer was told it created the stepCORRUPTED_EVENT_LOGand nothing else. The message the runtime failed on does not survive into the run row, and the observability tables are not populated for a preview backend, so which two writes collided had to be inferred from the shape of the code — which is how two diagnoses here went wrongKnown red
event-log-race-reprostill fails, and not on corruption. All 12 storm runs sit atrunningfor the full 240s with 4300-4600 events whose tail is nothing butstep_started/step_retryingcycling over ~15 correlation ids: an infinite retry loop where the ceiling never fires. Two of the twelve already did this before any fix.It is v5-only for a structural reason: there the attempt is whatever the client puts on the event, while every world that materializes a step row computes it from its own counter and ignores the client — so world-postgres is 14/14 clean on identical code and cannot exercise this at all. Not diagnosed further.