Skip to content

core: QuickJS engine divergence-detection and write-fencing parity with node:vm - #3453

Open
TooTallNate wants to merge 2 commits into
mainfrom
quickjs-divergence-parity
Open

core: QuickJS engine divergence-detection and write-fencing parity with node:vm#3453
TooTallNate wants to merge 2 commits into
mainfrom
quickjs-divergence-parity

Conversation

@TooTallNate

@TooTallNate TooTallNate commented Aug 11, 2026

Copy link
Copy Markdown
Member

Why

The QuickJS dry-run #3253 showed that QuickJS's clean CORRUPTED_EVENT_LOG record in the event-log-race repro is largely a classification artifact: the engine had no divergence machinery at all, so the same racing-writer corruption the node engine reports as CORRUPTED_EVENT_LOG surfaced on QuickJS as USER_ERROR (a self-HookConflictError — both hook-storm failures on #3253 decrypt to conflictingRunId == <own runId>), or worse, as a silently wrong completion: with no stepName validation, a step_completed for a different step's ordinal resolved the wrong call with the wrong payload, and run_completed was written over a log the replay demonstrably did not reproduce.

This PR gives the two engines equivalent functionality on both axes, so repro results (and production failures) are directly comparable across engines.

What

1. Replay-divergence arbitration (quickjs-divergence.ts + fixed-point sweeps)

At every replay fixed point (initial drain loop convergence, and each continueWithEvents burst), the host dumps the VM's draws (__pending + hook/abort registries) and arbitrates the full observed event log against them:

  • Orphaned events — a correlation id the replay never drew → ReplayDivergenceError (mirrors the node engine's unconsumed-event check).
  • Family mismatches — a step_* event for an id drawn as a wait, etc.
  • Identity mismatchesstepName (step.ts parity), hook token (hook.ts parity), wait_completed resumeAt (sleep.ts parity).

The error throws out of the entrypoint into runtime.ts's existing replay-loop catch, engaging the same recovery machinery as the node engine: bounded recovery replays (WORKFLOW_REPLAY_DIVERGENCE_MAX_RETRIES), then terminal CorruptedEventLogError. A workflow that already failed with a genuine user error is recorded as such — arbitration only applies to logs the replay claims to have reproduced.

Because the QuickJS host drains the VM's microtask queue synchronously, the fixed point is exact — none of the grace-window / delivery-idle heuristics the node engine needs under the shared event loop.

2. Precondition guard (closes the documented KNOWN GAP)

Every replay-context event write in the entrypoint now carries the optimistic-concurrency view snapshot (stateUpdatedAt / stateEventCount / stateCursor), maintained incrementally from every event the invocation observes:

  • dispatchPendingOps writes: step_created, hook_created, hook_received (aborts), hook_disposed, attr_set, wait_created
  • both elapsed-wait wait_completed passes
  • run_completed
  • inline lazy step claims (executeStep preconditionSnapshot), with suppressOptimisticStart while a hook is open on guard-enforcing Worlds — so a 412-fenced step never executes user code (node parity)
  • run_failed deliberately unfenced, matching the node engine's terminal-failure asymmetry

A 412 propagates to runtime.ts, whose existing restartReplayInProcess recovery invalidates the cached log and re-dispatches the QuickJS replay over the corrected view.

Validation

  • New unit suites: quickjs-divergence.test.ts (pure arbitration semantics), quickjs-entrypoint.fencing.test.ts (snapshot on fenced writes, run_failed unfenced, guard kill-switch), plus VM-level integration tests in quickjs-runtime.test.ts (orphan rejection, wrong-stepName rejection, healthy replay unaffected, user-failure priority, live-continuation divergence).
  • Full packages/core suite: 95 files, 2052 tests passing. Build + typecheck clean.
  • The CI quickjs e2e lanes exercise the full replay paths; combining this with the event-log-race-repro label on a quickjs-default branch (à la [DO NOT MERGE] CI dry-run: QuickJS as the default workflow VM engine #3253) should now show the same CORRUPTED_EVENT_LOG classification as the node engine for the racing-writer class.

@TooTallNate
TooTallNate requested a review from a team as a code owner August 11, 2026 07:57
Copilot AI lite review requested due to automatic review settings August 11, 2026 07:57
@changeset-bot

changeset-bot Bot commented Aug 11, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 5bed21a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
Name Type
@workflow/core Patch
@workflow/builders Patch
@workflow/cli Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Patch
@workflow/world-testing Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview Aug 11, 2026 8:43am
example-nextjs-workflow-webpack Ready Ready Preview Aug 11, 2026 8:43am
example-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-astro-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-express-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-fastify-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-hono-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-nestjs-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-nitro-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-nuxt-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-python-workflow Error Error Aug 11, 2026 8:43am
workbench-sveltekit-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-tanstack-start-workflow Ready Ready Preview Aug 11, 2026 8:43am
workbench-vite-workflow Ready Ready Preview Aug 11, 2026 8:43am
workflow-docs Ready Ready Preview, v0 Aug 11, 2026 8:43am
workflow-swc-playground Ready Ready Preview Aug 11, 2026 8:43am
workflow-tarballs Ready Ready Preview Aug 11, 2026 8:43am
workflow-web Ready Ready Preview Aug 11, 2026 8:43am

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (15 failed)

astro-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

example-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

express-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

fastify-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

hono-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

nest-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

nextjs-turbopack-quickjs (2 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability
  • pages router sleepingWorkflow via pages router

nextjs-webpack-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

nitro-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

nuxt-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

sveltekit-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

tanstack-start-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

vite-node (1 failed):

vite-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R | 🔍 observability

💻 Local Development (9 failed)

astro-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nest-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-turbopack-canary-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-turbopack-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-webpack-canary-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nuxt-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

sveltekit-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

tanstack-start-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

vite-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R
📦 Local Production (14 failed)

astro-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

express-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

fastify-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

hono-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nest-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-turbopack-canary-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-turbopack-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-webpack-canary-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-webpack-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nitro-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nuxt-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

sveltekit-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

tanstack-start-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

vite-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R
🐘 Local Postgres (14 failed)

astro-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

express-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

fastify-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

hono-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nest-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-turbopack-canary-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-turbopack-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-webpack-canary-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nextjs-webpack-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nitro-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

nuxt-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

sveltekit-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

tanstack-start-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

vite-stable-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

🪟 Windows (1 failed)

nextjs-turbopack-quickjs (1 failed):

  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41KZR0CZ3Y0GG9BVW3SJ56XG8R

E2E Test Summary

Summary
Passed Failed Skipped Total
❌ ▲ Vercel Production 3451 15 590 4056
❌ 💻 Local Development 3639 9 558 4206
❌ 📦 Local Production 3796 14 558 4368
❌ 🐘 Local Postgres 3796 14 558 4368
❌ 🪟 Windows 311 1 0 312
✅ vercel-multi-region 27 0 0 27
Total 15020 53 2264 17337
Details by Category

❌ ▲ Vercel Production

App Passed Failed Skipped
✅ astro-node 128 0 28
❌ astro-quickjs 127 1 28
✅ example-node 128 0 28
❌ example-quickjs 127 1 28
✅ express-node 128 0 28
❌ express-quickjs 127 1 28
✅ fastify-node 128 0 28
❌ fastify-quickjs 127 1 28
✅ hono-node 128 0 28
❌ hono-quickjs 127 1 28
✅ nest-node 128 0 28
❌ nest-quickjs 127 1 28
✅ nextjs-turbopack-node 153 0 3
❌ nextjs-turbopack-quickjs 151 2 3
✅ nextjs-webpack-node 153 0 3
❌ nextjs-webpack-quickjs 152 1 3
✅ nitro-node 128 0 28
❌ nitro-quickjs 127 1 28
✅ nuxt-node 128 0 28
❌ nuxt-quickjs 127 1 28
✅ sveltekit-node 147 0 9
❌ sveltekit-quickjs 146 1 9
✅ tanstack-start-node 128 0 28
❌ tanstack-start-quickjs 127 1 28
❌ vite-node 127 1 28
❌ vite-quickjs 127 1 28

❌ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
❌ astro-stable-quickjs 129 1 26
✅ express-stable-node 130 0 26
✅ express-stable-quickjs 129 0 26
✅ fastify-stable-node 130 0 26
✅ fastify-stable-quickjs 129 0 26
✅ hono-stable-node 130 0 26
✅ hono-stable-quickjs 128 0 26
✅ nest-stable-node 130 0 26
❌ nest-stable-quickjs 129 1 26
✅ nextjs-turbopack-canary-node 137 0 19
❌ nextjs-turbopack-canary-quickjs 136 1 19
✅ nextjs-turbopack-stable-node 156 0 0
❌ nextjs-turbopack-stable-quickjs 155 1 0
✅ nextjs-webpack-canary-node 137 0 19
❌ nextjs-webpack-canary-quickjs 136 1 19
✅ nextjs-webpack-stable-node 156 0 0
✅ nitro-stable-node 130 0 26
✅ nitro-stable-quickjs 128 0 26
✅ nuxt-stable-node 130 0 26
❌ nuxt-stable-quickjs 129 1 26
✅ sveltekit-stable-node 149 0 7
❌ sveltekit-stable-quickjs 148 1 7
✅ tanstack-start-node 130 0 26
❌ tanstack-start-quickjs 129 1 26
✅ vite-stable-node 130 0 26
❌ vite-stable-quickjs 129 1 26

❌ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
❌ astro-stable-quickjs 129 1 26
✅ express-stable-node 130 0 26
❌ express-stable-quickjs 129 1 26
✅ fastify-stable-node 130 0 26
❌ fastify-stable-quickjs 129 1 26
✅ hono-stable-node 130 0 26
❌ hono-stable-quickjs 129 1 26
✅ nest-stable-node 130 0 26
❌ nest-stable-quickjs 129 1 26
✅ nextjs-turbopack-canary-node 137 0 19
❌ nextjs-turbopack-canary-quickjs 136 1 19
✅ nextjs-turbopack-stable-node 156 0 0
❌ nextjs-turbopack-stable-quickjs 155 1 0
✅ nextjs-webpack-canary-node 137 0 19
❌ nextjs-webpack-canary-quickjs 136 1 19
✅ nextjs-webpack-stable-node 156 0 0
❌ nextjs-webpack-stable-quickjs 155 1 0
✅ nitro-stable-node 130 0 26
❌ nitro-stable-quickjs 129 1 26
✅ nuxt-stable-node 130 0 26
❌ nuxt-stable-quickjs 129 1 26
✅ sveltekit-stable-node 149 0 7
❌ sveltekit-stable-quickjs 148 1 7
✅ tanstack-start-node 130 0 26
❌ tanstack-start-quickjs 129 1 26
✅ vite-stable-node 130 0 26
❌ vite-stable-quickjs 129 1 26

❌ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
❌ astro-stable-quickjs 129 1 26
✅ express-stable-node 130 0 26
❌ express-stable-quickjs 129 1 26
✅ fastify-stable-node 130 0 26
❌ fastify-stable-quickjs 129 1 26
✅ hono-stable-node 130 0 26
❌ hono-stable-quickjs 129 1 26
✅ nest-stable-node 130 0 26
❌ nest-stable-quickjs 129 1 26
✅ nextjs-turbopack-canary-node 137 0 19
❌ nextjs-turbopack-canary-quickjs 136 1 19
✅ nextjs-turbopack-stable-node 156 0 0
❌ nextjs-turbopack-stable-quickjs 155 1 0
✅ nextjs-webpack-canary-node 137 0 19
❌ nextjs-webpack-canary-quickjs 136 1 19
✅ nextjs-webpack-stable-node 156 0 0
❌ nextjs-webpack-stable-quickjs 155 1 0
✅ nitro-stable-node 130 0 26
❌ nitro-stable-quickjs 129 1 26
✅ nuxt-stable-node 130 0 26
❌ nuxt-stable-quickjs 129 1 26
✅ sveltekit-stable-node 149 0 7
❌ sveltekit-stable-quickjs 148 1 7
✅ tanstack-start-node 130 0 26
❌ tanstack-start-quickjs 129 1 26
✅ vite-stable-node 130 0 26
❌ vite-stable-quickjs 129 1 26

❌ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 156 0 0
❌ nextjs-turbopack-quickjs 155 1 0

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

📋 View full workflow run

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 5bed21a · Tue, 11 Aug 2026 08:57:01 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1154 (+9.2%) 1299 🔴 (+10%) 1365 🔴 (+13%) 1550 🔴 (-9.3%) 30
TTFS stream 1149 (+10%) 1206 🔴 (+5.5%) 1229 🔴 (+5.4%) 1293 🔴 (+6.9%) 30
TTFS hook + stream 1385 (+11%) 1468 🔴 (+7.3%) 1554 🔴 (+11%) 1645 🔴 (+9.1%) 30
STSO 1020 steps (inline) 96 (±0%) 123 (-25%) 💚 143 (-30%) 💚 270 (-65%) 💚 1019
WO 1020 steps 126086 (-26%) 💚 126086 (-26%) 💚 126086 (-26%) 💚 126086 (-26%) 💚 1
SL stream latency 81 (-11%) 100 🔴 (-35%) 💚 109 🔴 (-37%) 💚 232 🔴 (-58%) 💚 30
SO stream overhead (text) 95 (-28%) 💚 135 (-51%) 💚 150 (-55%) 💚 229 (-71%) 💚 30
SO stream overhead (structured) 94 (-23%) 💚 134 (-45%) 💚 176 (-44%) 💚 196 (-56%) 💚 30
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 170410ms → this run 124907ms (Δ -45503ms, -27%)

   50-100 ms  ┃                         main   1  this  18   +17
  100-150 ms  ████████████████░░░░░░░┃  main 627  this 919  +292
  150-200 ms  ┃██████                   main 279  this  57  -222
  200-250 ms  ┃█                        main  59  this  11   -48
  250-300 ms  ┃                         main  17  this   6   -11
  300-350 ms  ┃                         main   8  this   0    -8
  350-400 ms  ┃                         main   3  this   1    -2
  400-450 ms  ┃                         main   2  this   2    +0
  450-500 ms  ┃                         main   1  this   0    -1
  500-550 ms  ┃                         main   2  this   4    +2
  600-650 ms  ┃                         main   2  this   0    -2
  650-700 ms  ┃                         main   4  this   0    -4
  700-750 ms  ┃                         main   3  this   0    -3
  750-800 ms  ┃                         main   2  this   0    -2
  800-850 ms  ┃                         main   3  this   1    -2
  850-900 ms  ┃                         main   1  this   0    -1
 950-1000 ms  ┃                         main   1  this   0    -1
1000-1050 ms  ┃                         main   1  this   0    -1
1250-1300 ms  ┃                         main   1  this   0    -1
1300-1350 ms  ┃                         main   1  this   0    -1
1650-1700 ms  ┃                         main   1  this   0    -1
📜 Previous results (1)

a8bff02

Tue, 11 Aug 2026 08:15:18 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 244 (-77%) 💚 1444 🔴 (+23%) 🔻 1532 🔴 (+27%) 🔻 1582 🔴 (-7.4%) 30
TTFS stream 258 (-75%) 💚 1434 🔴 (+25%) 🔻 1498 🔴 (+28%) 🔻 1587 🔴 (+31%) 🔻 30
TTFS hook + stream 333 (-73%) 💚 1697 🔴 (+24%) 🔻 1756 🔴 (+26%) 🔻 1996 🔴 (+32%) 🔻 30
STSO 1020 steps (inline) 93 (-3.1%) 151 (-7.9%) 169 (-17%) 💚 327 (-58%) 💚 1019
WO 1020 steps 148479 (-13%) 148479 (-13%) 148479 (-13%) 148479 (-13%) 1
SL stream latency 103 (+13%) 141 🔴 (-8.4%) 212 🔴 (+23%) 🔻 458 🔴 (-18%) 💚 30
SO stream overhead (text) 126 (-4.5%) 206 (-25%) 💚 252 (-24%) 💚 698 (-11%) 30
SO stream overhead (structured) 121 (-0.8%) 308 🔴 (+26%) 🔻 801 🔴 (+157%) 🔻 963 (+114%) 🔻 30
ℹ️ Metric definitions & methodology

The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: is main, marks where this run lands, bridges the gap when this run has more samples in a bucket.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR brings the QuickJS workflow VM engine up to parity with the node:vm engine for (1) replay divergence detection and (2) optimistic-concurrency write fencing, so corrupted/diverged event logs are consistently classified and recovered across engines.

Changes:

  • Add fixed-point replay-divergence arbitration for QuickJS (orphaned events, family mismatches, and identity mismatches like stepName / hook token / wait resumeAt), escalating via existing runtime recovery.
  • Thread an optimistic-concurrency precondition snapshot through replay-context event writes in the QuickJS entrypoint (leaving run_failed intentionally unfenced for parity).
  • Add unit + integration coverage for divergence arbitration and precondition fencing behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/core/src/runtime/quickjs-runtime.ts Runs fixed-point replay-divergence sweeps during initial replay and each continueWithEvents burst.
packages/core/src/runtime/quickjs-runtime.test.ts Adds VM-level integration tests for replay divergence behavior (orphan, wrong stepName, live continuation, etc.).
packages/core/src/runtime/quickjs-entrypoint.ts Implements incremental precondition snapshot tracking and applies it to replay-context writes (plus inline-step claim fencing behavior).
packages/core/src/runtime/quickjs-entrypoint.fencing.test.ts Verifies fenced vs. unfenced writes and guard kill-switch behavior with a mocked VM.
packages/core/src/runtime/quickjs-divergence.ts Introduces the pure arbitration logic used by the fixed-point sweep.
packages/core/src/runtime/quickjs-divergence.test.ts Unit-tests arbitration semantics (structural events, orphan/family/identity mismatches).
.changeset/quickjs-divergence-parity.md Changeset entry documenting the parity improvements in @workflow/core.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/core/src/runtime/quickjs-divergence.ts
Comment thread packages/core/src/runtime/quickjs-entrypoint.ts
TooTallNate added a commit that referenced this pull request Aug 11, 2026
Combines the QuickJS-default dry-run with divergence-detection and
write-fencing parity (#3453) so the event-log-race repro measures the
same corruption classes on both engines. Conflict resolutions:

- makeLiveSession takes both the snapshot-state accessor and the
  observed-events ledger; the snapshot-restore path runs the divergence
  sweep against its delta view (a throw there falls back to a fresh
  full replay, whose sweep is authoritative).
- The precondition view tracker seeds its event count from the restored
  snapshot's metadata.eventCount (delta ids sort above the snapshot
  cursor, so watermark+count still describe the full log); snapshots
  without an eventCount fail open, and the restore-failure fallback
  rebuilds the view from the refetched full log.
@TooTallNate TooTallNate added the event-log-race-repro Run the event log race reproduction job label Aug 11, 2026
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Event Log Race Repro

1 of 14 latest repro runs hit event-log regressions.

Run History

Metric 2026-08-11 08:38 UTC #1
logs / deploy
2026-08-11 08:45 UTC #1
logs / deploy
Result 4/14 regressions 1/14 regressions
Total 14 14
completed 10 13
CORRUPTED_EVENT_LOG 4 1
USER_ERROR 0 0
RUNTIME_ERROR 0 0
stuck 0 0
other 0 0
infra 0 0
Config 14 runs / step-storm 6, hook-storm 6, hook-sleep 2 / c8 / 6x8 14 runs / step-storm 6, hook-storm 6, hook-sleep 2 / c8 / 6x8
Timing watchdog 2500ms / step 2200±250ms / stagger 400ms / poke 750ms / timeout 240000ms watchdog 2500ms / step 2200±250ms / stagger 400ms / poke 750ms / timeout 240000ms

Latest Scenario Breakdown

Scenario Total completed CORRUPTED_EVENT_LOG USER_ERROR RUNTIME_ERROR stuck other infra
step-storm 6 6 0 0 0 0 0 0
hook-storm 6 5 1 0 0 0 0 0
hook-sleep 2 2 0 0 0 0 0 0

Latest Non-Completed Runs

Scenario Attempt Outcome Status Error code Run
hook-storm 5 CORRUPTED_EVENT_LOG failed CORRUPTED_EVENT_LOG wrun_41KZQZW4H20GMCB9240WQ0ZPA1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

event-log-race-repro Run the event log race reproduction job

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants