core: QuickJS engine divergence-detection and write-fencing parity with node:vm - #3453
core: QuickJS engine divergence-detection and write-fencing parity with node:vm#3453TooTallNate wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 5bed21a The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (15 failed)astro-quickjs (1 failed):
example-quickjs (1 failed):
express-quickjs (1 failed):
fastify-quickjs (1 failed):
hono-quickjs (1 failed):
nest-quickjs (1 failed):
nextjs-turbopack-quickjs (2 failed):
nextjs-webpack-quickjs (1 failed):
nitro-quickjs (1 failed):
nuxt-quickjs (1 failed):
sveltekit-quickjs (1 failed):
tanstack-start-quickjs (1 failed):
vite-node (1 failed):
vite-quickjs (1 failed):
💻 Local Development (9 failed)astro-stable-quickjs (1 failed):
nest-stable-quickjs (1 failed):
nextjs-turbopack-canary-quickjs (1 failed):
nextjs-turbopack-stable-quickjs (1 failed):
nextjs-webpack-canary-quickjs (1 failed):
nuxt-stable-quickjs (1 failed):
sveltekit-stable-quickjs (1 failed):
tanstack-start-quickjs (1 failed):
vite-stable-quickjs (1 failed):
📦 Local Production (14 failed)astro-stable-quickjs (1 failed):
express-stable-quickjs (1 failed):
fastify-stable-quickjs (1 failed):
hono-stable-quickjs (1 failed):
nest-stable-quickjs (1 failed):
nextjs-turbopack-canary-quickjs (1 failed):
nextjs-turbopack-stable-quickjs (1 failed):
nextjs-webpack-canary-quickjs (1 failed):
nextjs-webpack-stable-quickjs (1 failed):
nitro-stable-quickjs (1 failed):
nuxt-stable-quickjs (1 failed):
sveltekit-stable-quickjs (1 failed):
tanstack-start-quickjs (1 failed):
vite-stable-quickjs (1 failed):
🐘 Local Postgres (14 failed)astro-stable-quickjs (1 failed):
express-stable-quickjs (1 failed):
fastify-stable-quickjs (1 failed):
hono-stable-quickjs (1 failed):
nest-stable-quickjs (1 failed):
nextjs-turbopack-canary-quickjs (1 failed):
nextjs-turbopack-stable-quickjs (1 failed):
nextjs-webpack-canary-quickjs (1 failed):
nextjs-webpack-stable-quickjs (1 failed):
nitro-stable-quickjs (1 failed):
nuxt-stable-quickjs (1 failed):
sveltekit-stable-quickjs (1 failed):
tanstack-start-quickjs (1 failed):
vite-stable-quickjs (1 failed):
🪟 Windows (1 failed)nextjs-turbopack-quickjs (1 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
❌ 💻 Local Development
❌ 📦 Local Production
❌ 🐘 Local Postgres
❌ 🪟 Windows
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 170410ms → this run 124907ms (Δ -45503ms, -27%) 📜 Previous results (1)a8bff02Tue, 11 Aug 2026 08:15:18 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
There was a problem hiding this comment.
Pull request overview
This PR brings the QuickJS workflow VM engine up to parity with the node:vm engine for (1) replay divergence detection and (2) optimistic-concurrency write fencing, so corrupted/diverged event logs are consistently classified and recovered across engines.
Changes:
- Add fixed-point replay-divergence arbitration for QuickJS (orphaned events, family mismatches, and identity mismatches like
stepName/ hooktoken/ waitresumeAt), escalating via existing runtime recovery. - Thread an optimistic-concurrency precondition snapshot through replay-context event writes in the QuickJS entrypoint (leaving
run_failedintentionally unfenced for parity). - Add unit + integration coverage for divergence arbitration and precondition fencing behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/runtime/quickjs-runtime.ts | Runs fixed-point replay-divergence sweeps during initial replay and each continueWithEvents burst. |
| packages/core/src/runtime/quickjs-runtime.test.ts | Adds VM-level integration tests for replay divergence behavior (orphan, wrong stepName, live continuation, etc.). |
| packages/core/src/runtime/quickjs-entrypoint.ts | Implements incremental precondition snapshot tracking and applies it to replay-context writes (plus inline-step claim fencing behavior). |
| packages/core/src/runtime/quickjs-entrypoint.fencing.test.ts | Verifies fenced vs. unfenced writes and guard kill-switch behavior with a mocked VM. |
| packages/core/src/runtime/quickjs-divergence.ts | Introduces the pure arbitration logic used by the fixed-point sweep. |
| packages/core/src/runtime/quickjs-divergence.test.ts | Unit-tests arbitration semantics (structural events, orphan/family/identity mismatches). |
| .changeset/quickjs-divergence-parity.md | Changeset entry documenting the parity improvements in @workflow/core. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Combines the QuickJS-default dry-run with divergence-detection and write-fencing parity (#3453) so the event-log-race repro measures the same corruption classes on both engines. Conflict resolutions: - makeLiveSession takes both the snapshot-state accessor and the observed-events ledger; the snapshot-restore path runs the divergence sweep against its delta view (a throw there falls back to a fresh full replay, whose sweep is authoritative). - The precondition view tracker seeds its event count from the restored snapshot's metadata.eventCount (delta ids sort above the snapshot cursor, so watermark+count still describe the full log); snapshots without an eventCount fail open, and the restore-failure fallback rebuilds the view from the refetched full log.
Event Log Race Repro1 of 14 latest repro runs hit event-log regressions. Run History
Latest Scenario Breakdown
Latest Non-Completed Runs
|
…quickjs wait_completed
Why
The QuickJS dry-run #3253 showed that QuickJS's clean
CORRUPTED_EVENT_LOGrecord in the event-log-race repro is largely a classification artifact: the engine had no divergence machinery at all, so the same racing-writer corruption the node engine reports asCORRUPTED_EVENT_LOGsurfaced on QuickJS asUSER_ERROR(a self-HookConflictError— both hook-storm failures on #3253 decrypt toconflictingRunId == <own runId>), or worse, as a silently wrong completion: with nostepNamevalidation, astep_completedfor a different step's ordinal resolved the wrong call with the wrong payload, andrun_completedwas written over a log the replay demonstrably did not reproduce.This PR gives the two engines equivalent functionality on both axes, so repro results (and production failures) are directly comparable across engines.
What
1. Replay-divergence arbitration (
quickjs-divergence.ts+ fixed-point sweeps)At every replay fixed point (initial drain loop convergence, and each
continueWithEventsburst), the host dumps the VM's draws (__pending+ hook/abort registries) and arbitrates the full observed event log against them:ReplayDivergenceError(mirrors the node engine's unconsumed-event check).step_*event for an id drawn as a wait, etc.stepName(step.ts parity), hooktoken(hook.ts parity),wait_completedresumeAt(sleep.ts parity).The error throws out of the entrypoint into
runtime.ts's existing replay-loop catch, engaging the same recovery machinery as the node engine: bounded recovery replays (WORKFLOW_REPLAY_DIVERGENCE_MAX_RETRIES), then terminalCorruptedEventLogError. A workflow that already failed with a genuine user error is recorded as such — arbitration only applies to logs the replay claims to have reproduced.Because the QuickJS host drains the VM's microtask queue synchronously, the fixed point is exact — none of the grace-window / delivery-idle heuristics the node engine needs under the shared event loop.
2. Precondition guard (closes the documented KNOWN GAP)
Every replay-context event write in the entrypoint now carries the optimistic-concurrency view snapshot (
stateUpdatedAt/stateEventCount/stateCursor), maintained incrementally from every event the invocation observes:dispatchPendingOpswrites:step_created,hook_created,hook_received(aborts),hook_disposed,attr_set,wait_createdwait_completedpassesrun_completedexecuteSteppreconditionSnapshot), withsuppressOptimisticStartwhile a hook is open on guard-enforcing Worlds — so a 412-fenced step never executes user code (node parity)run_faileddeliberately unfenced, matching the node engine's terminal-failure asymmetryA 412 propagates to
runtime.ts, whose existingrestartReplayInProcessrecovery invalidates the cached log and re-dispatches the QuickJS replay over the corrected view.Validation
quickjs-divergence.test.ts(pure arbitration semantics),quickjs-entrypoint.fencing.test.ts(snapshot on fenced writes,run_failedunfenced, guard kill-switch), plus VM-level integration tests inquickjs-runtime.test.ts(orphan rejection, wrong-stepName rejection, healthy replay unaffected, user-failure priority, live-continuation divergence).packages/coresuite: 95 files, 2052 tests passing. Build + typecheck clean.event-log-race-reprolabel on a quickjs-default branch (à la [DO NOT MERGE] CI dry-run: QuickJS as the default workflow VM engine #3253) should now show the sameCORRUPTED_EVENT_LOGclassification as the node engine for the racing-writer class.