QuickJS engine: threshold-based VM-memory snapshotting (WORKFLOW_SNAPSHOT_THRESHOLD) - #3251
QuickJS engine: threshold-based VM-memory snapshotting (WORKFLOW_SNAPSHOT_THRESHOLD)#3251TooTallNate wants to merge 6 commits into
Conversation
🦋 Changeset detectedLatest commit: eee5695 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests📦 Local Production (1 failed)vite-stable-node (1 failed):
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
❌ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: blocking issues found
00d4d3e to
7f4d517
Compare
…KFLOW_SNAPSHOT_THRESHOLD)
…based PRNG fast-forward, unified host-callback list, lifecycle hardening - SnapshotMetadata gains eventCount, rngDraws and formatVersion. The max-events guard now compares restored total + delta (both at entry and per loop turn) — previously a run that kept snapshotting could never accumulate enough delta to trip the ceiling it exists for. - Correlation-id generation is position-based across snapshots: the runtime seeds from the BASE seed and fast-forwards the persisted draw count instead of mixing the snapshot cursor into the seed. Ids are now identical across snapshot generations AND identical to a no-snapshot run, so overlapping invocations straddling a snapshot save still collide on the world's dedup (new test pins restored ids == full-replay ids). Snapshots without a draw count fall back to full replay. - Host callbacks are declared in ONE list that drives both the fresh-boot install and the restore re-registration, so adding a callback can't silently skip the restore path. - Preloaded events are used again with snapshotting enabled (the first qualifying suspension skips its save — no cursor yet); short runs keep the zero-round-trip fast path. - Snapshot persist runs off the response path (waitUntil), with a 32MB plaintext size ceiling (skip + warn). Loads that fail format/shape checks warn instead of silently miming a miss. Terminal deletes are gated on a snapshot actually existing and now also fire on the runGone path; a server-side TTL remains the backstop for unobserved cancellations.
81784aa to
c139c2a
Compare
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 158558ms → this run 132599ms (Δ -25959ms, -16%) 📜 Previous results (3)a5ee711Mon, 10 Aug 2026 19:43:13 GMT · run logs
e294ed1Tue, 04 Aug 2026 00:47:10 GMT · run logs
c139c2aFri, 31 Jul 2026 23:29:13 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Reviewed the incremental diff (12 files, +926) and ran the stack top locally. Not merge-ready — headline finding inline at the load gate: snapshots never restore on world-postgres or world-vercel, and nothing in CI can tell, because a snapshot that never restores falls back to full replay, which is correct behavior. The quickjs-snapshot CI legs are currently red only on the inherited #3049 overflow bug; once that's fixed they'd go green while snapshots remain pure cost on both production worlds.
Local validation of what does work (world-local): snapshot lifecycle is real — 101 snapshot_saved / 283 snapshot_load diag checkpoints across a full e2e leg, 4 MB-ish .bin/.json pairs created and deleted on completion, restored: true resumptions observed. The position-based PRNG fast-forward design is correct: the restored-run-produces-identical-correlationIds property is the right one to pin, and I could not construct a divergence; the per-(runId, correlationId) dedup backstop can't wedge. Host-callback re-registration via the unified list is complete (no other newFunction sites). The eventsCursor frontier is sound across all three worlds (strictly exclusive cursors, no off-by-one). Compression is already workerd-safe.
One n=1 observation from a full snapshot-leg run worth your eyes: parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race failed once with a correctness assertion (harness posted body-<tokenA> for a token it listed from storage, workflow expected body-<tokenB>), i.e. a post-restore disagreement about the run's own webhook token. 10/10 passes in isolation afterwards and absent from a second full leg — but a restore-path token/PRNG-position determinism bug is what it would smell like if it recurs.
Non-blocking: first-invocation captures with no cursor are taken then dropped; the restore drain loop lacks the boot path's iteration-bound warning; deleteSnapshotIfAny early-returns when the threshold is 0, so flipping the env off strands stored snapshots (and a waitUntil save landing after a terminal delete re-creates one); __hookPayloadBuffer.__processedEventIds grows monotonically in-heap (snapshot size for long-lived reusable hooks only increases); __wdk_env isn't refreshed after restore; the docs say invalid threshold values "throw at startup" but getSnapshotThresholdFromEnv throws on first invocation. Changeset omits @workflow/world (this PR modifies packages/world/src/snapshots.ts).
Stack coordination: #3263 rebases under this per its own description — do that first. Three silent semantic breaks to check on the rebase: the inline __generateUlid registration won't be re-registered on restore (the branch's own comment says host callbacks must never be inline); the restore path's "serde survives in the heap" comment becomes false (serde is host-side after #3263 — createQuickJSSerde must run on restore); and the ULID monotonic factory's state moves host-side, so it's no longer captured by the snapshot and the rngDraws fast-forward desynchronizes.
| const version = loaded.metadata.formatVersion; | ||
| if ( | ||
| (version !== undefined && version !== SNAPSHOT_FORMAT_VERSION) || | ||
| loaded.metadata.rngDraws === undefined |
There was a problem hiding this comment.
Blocking: this gate rejects every snapshot on world-postgres and world-vercel, so the feature never restores on either production world. Neither world persists the new metadata fields: world-postgres writes/reads only eventsCursor + createdAt (migration 0018 has no columns for eventCount/rngDraws/formatVersion), and world-vercel sends/parses only the two original headers (carrying the new fields also needs a workflow-server change). Only world-local round-trips them, which is why the local tests pass.
Net effect on prod worlds: every qualifying suspension pays session.snapshot() (two full heap copies) + compression + encryption + a 5–15 MB PUT, and every resume throws it away and full-replays — strictly worse than WORKFLOW_SNAPSHOT_THRESHOLD=0. Persisting the fields in both worlds is the right fix; tolerating missing rngDraws is not safe (a restored heap with rngDraws: 0 re-draws and collides correlationIds).
There was a problem hiding this comment.
Fixed in eee5695 (+ 374dbaa on the base PR). Rather than adding postgres columns and a workflow-server change, the base PR now packs the FULL metadata object into the stored blob itself (snapshot envelope: metadata + bytes in one self-describing unit, schema-validated on decode with passthrough for forward compat). postgres stores the envelope in the existing data column; vercel sends it as the PUT/GET body the server already stores opaquely — so every metadata field this PR adds (eventCount, rngDraws, formatVersion, and the newer serdeRootPtr/lastUlid/clockMs/engineVersion) round-trips on BOTH production worlds today, no server change needed, and future fields need no storage work at all. The gate still refuses to tolerate missing rngDraws (agreed that direction is unsafe); it just no longer has a reason to fire on prod worlds. Also: world.snapshots went optional on the base PR, so the entrypoint now feature-detects and forces the threshold to 0 when absent.
| // its next qualifying suspension instead.) | ||
| const snapshot = capturedSnapshot; | ||
| const totalEventCount = restoredEventCount + seenEventIds.size; | ||
| safeWaitUntil( |
There was a problem hiding this comment.
"Off the response path" is false on the default codec: this IIFE runs synchronously up to its first real await, and compress with the default write codec is zlib.zstdCompressSync — synchronous compression of a multi-MB heap image blocks the event loop before the response flushes. The gzip fallback (CompressionStream) is genuinely async, so this only bites the default path. Defer past the current tick (Promise.resolve().then(...)) or use the async codec for snapshots.
There was a problem hiding this comment.
Fixed in eee5695, both halves: (1) the waitUntil IIFE now yields past the current tick (setImmediate) before touching the bytes, so the response flushes first; (2) compress() gained a preferAsync option — zstd via node:zlib's callback API (libuv threadpool, identical output bytes; sync fallback where unavailable) — and the snapshot pipeline uses it, so the compression itself no longer blocks the loop regardless of when it starts. Event-payload compression keeps the sync path (small payloads, unchanged behavior).
| // (pre-snapshot count persisted in the metadata + delta) — otherwise a | ||
| // run that keeps snapshotting would never accumulate enough delta to | ||
| // trip the ceiling it exists to enforce. | ||
| const restoredEventCount = existingSnapshot?.metadata.eventCount ?? 0; |
There was a problem hiding this comment.
restoredEventCount double-counts after a restore failure. The restore-failure catch resets existingSnapshot/lastEventsCursor to fall back to full replay but can't reset this const; the ceiling check then adds the stale count to a seenEventIds set that now holds the entire log, so MaxEventsExceededError can fire well below the real limit — and the next save stamps the inflated eventCount, compounding. Currently masked by the load-gate finding (no restores on prod worlds ⇒ no failures to fall back from), which is exactly the latent-bug shape that surfaces the day that's fixed.
There was a problem hiding this comment.
Fixed in eee5695: restoredEventCount is a let and the restore-failure fallback resets it to 0 alongside existingSnapshot/lastEventsCursor — from that point events/seenEventIds cover the whole run, so the ceiling compares the true total and the next save stamps an accurate eventCount.
| ) { | ||
| try { | ||
| capturedSnapshot = session.snapshot(); | ||
| if (capturedSnapshot.data.byteLength > MAX_SNAPSHOT_PLAINTEXT_BYTES) { |
There was a problem hiding this comment.
Ceiling enforced after the expensive work: session.snapshot() has already made two full copies of the WASM heap by this check, and since WASM linear memory never shrinks, a run that once crossed 32 MB re-pays both copies at every subsequent suspension and discards the result every time. Gate on VM memory size before snapshotting, or latch a per-run "too big" flag on first rejection.
There was a problem hiding this comment.
Fixed in eee5695 with the latch you suggested: a process-local bounded set of runIds whose heap exceeded the ceiling. Since WASM linear memory never shrinks, once-oversized is always-oversized — later suspensions of that run now skip BEFORE session.snapshot() (the two heap copies), not after. Process-local is the right scope: the warm instance replaying the same run repeatedly is where the repeated cost lived; a cold instance pays one probe and re-latches.
| * Current snapshot format version, bumped when the heap layout or the | ||
| * metadata contract changes incompatibly. | ||
| */ | ||
| export const SNAPSHOT_FORMAT_VERSION = 1; |
There was a problem hiding this comment.
formatVersion covers the SDK's metadata envelope, not the heap image — the QJSS header that deserializeSnapshot validates is identical across quickjs-wasi builds, so bytes from one build restored by another pass validation and execute as undefined behavior in the interpreter. Real deploy-skew hazard (a quickjs-wasi bump mid-rollout has live snapshots from the old build), and a data-corruption-class failure. Cheap close: the library exposes vm.versions — persist it in metadata and reject a mismatch. Related smaller gap: the deterministic clock's high-water mark (vmNowMs) isn't persisted either, so Date.now() in a restored VM can regress until the first event advances it.
There was a problem hiding this comment.
Fixed in eee5695, both parts. Engine pin: the asset build script now bakes the installed quickjs-wasi version into quickjs-assets.generated.ts (the exact build the embedded WASM came from — more precise than runtime vm.versions probing and available before any VM exists); it's stamped into metadata as engineVersion and the load gate treats any mismatch/absence as a clean miss, so a mid-rollout quickjs-wasi bump degrades to full replay instead of restoring a foreign heap. Clock: clockMs (the deterministic clock's high-water mark at capture) persists in metadata and primes the restored VM via the monotonic advanceClock, so Date.now() no longer regresses to run-creation time until the first delta event.
…-snapshots Carries the #3049 merge down the stack. Resolution notes: - inline loop's post-batch feed keeps BOTH sides: zero-event feeds raise pendingRequeueSignal before breaking (ack-without-requeue fix from #3049) and non-empty feeds advance eventsProcessedSinceSnapshot - quickjs-runtime.test.ts keeps both trailing suites (VM snapshot/restore + hook dispose-then-sleep replay) Validated: core 1899 passed; hook (26/26) and promiseRace e2e green under WORKFLOW_VM=quickjs with WORKFLOW_SNAPSHOT_THRESHOLD=5.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| // (pre-snapshot count persisted in the metadata + delta) — otherwise a | ||
| // run that keeps snapshotting would never accumulate enough delta to | ||
| // trip the ceiling it exists to enforce. | ||
| const restoredEventCount = existingSnapshot?.metadata.eventCount ?? 0; |
There was a problem hiding this comment.
restoredEventCount is captured as a const before snapshot restore and cannot be reset when restore fails and the code falls back to full event-log replay, causing the pre-snapshot count to be added on top of the now-full events/seenEventIds.
| // its next qualifying suspension instead.) | ||
| const snapshot = capturedSnapshot; | ||
| const totalEventCount = restoredEventCount + seenEventIds.size; | ||
| safeWaitUntil( |
There was a problem hiding this comment.
Snapshot compression claimed to run off the response path via safeWaitUntil actually runs synchronously on the response-path call stack for the default zstd codec, blocking the event loop while compressing a multi-MB heap.
Port threshold snapshotting onto the post-#3263/#3342 engine: - Snapshot format v2: per-run snapshot metadata now carries the host-serde capture root's snapshot-portable token (serdeRootPtr) and the monotonic correlation-id ULID factory's last output (lastUlid). v1 snapshots (in-VM serde era) are treated as a miss. - Restore path rebuilds the host-side serde by re-adopting the capture root from the restored memory image (adoptSerdeRoot) — no guest code executes after user code has run, same mechanism as baseline snapshots. - ULID continuation: the factory's monotonic state lives host-side and does not survive into the memory image; restores continue the sequence via incrementUlidRandom (byte-equivalent to the ulid package's same-timestamp increment, asserted by test), so a restored invocation emits the exact ids full replay would — keeping concurrent invocations dedupable via per-(runId, correlationId) uniqueness. - __generateUlid joins the shared host-callback list (host functions are name-referenced from the WASM heap and must be re-registered on restore). - Preloaded event logs (no cursor) are ignored when restoring from a snapshot; the caller-attested complete preload fast path is kept for the no-snapshot paths.
# Conflicts: # packages/world/src/index.ts
…ck restore, off-path compression, ceiling latch, optional snapshots - Blocking round-trip fix: full snapshot metadata now survives postgres and vercel via the snapshot envelope (see the world-side commit on quickjs-vm-snapshots) — the load gate no longer rejects every snapshot on production worlds. - world.snapshots is optional: the entrypoint feature-detects and forces the threshold to 0 when absent (pure full replay, always correct). - Engine-build pin: quickjs-wasi's version is baked into the generated assets module, stamped into snapshot metadata (engineVersion), and compared on load — the QJSS heap header is identical across builds, so a cross-build restore would execute as undefined behavior (live hazard mid-rollout). Mismatch = clean miss. - Deterministic clock high-water (clockMs) persists in metadata and primes the restored VM's clock — Date.now() no longer regresses to run-creation time until the first delta event. - Snapshot compression truly off the response path: the waitUntil IIFE yields past the current tick before touching bytes, and compress() gains preferAsync (zstd via the libuv threadpool with sync fallback; identical output bytes) so a multi-MB heap no longer blocks the loop. - restoredEventCount resets when a restore fails and the fallback refetches the full log — the ceiling no longer double-counts pre-snapshot events, and the next save no longer stamps an inflated eventCount. - Size-ceiling latch: a run whose heap once exceeded MAX_SNAPSHOT_PLAINTEXT_BYTES skips future captures outright (linear memory never shrinks), instead of re-paying two full heap copies per suspension to discard the result.
Note
Supersedes #3053, which GitHub auto-marked as merged (and auto-deleted the head branch) when a restacking mistake briefly force-pushed the head branch to the same commit as its base. Same content, freshly rebased on the stack.
Summary
PR 4 of the QuickJS VM roadmap: threshold-based VM-memory snapshotting — the middle ground that motivated reviving this effort (see #1298 / #1300 discussion). Instead of snapshotting at every suspension (the original branch's model, which cost ~25% on e2e wall clock), snapshots are taken only once
WORKFLOW_SNAPSHOT_THRESHOLDevents have been processed since the last one:How it works
WORKFLOW_SNAPSHOT_THRESHOLDenv var (default0= disabled) or per-runexecutionContext.snapshotThreshold, stamped atstart()for run affinity likeWORKFLOW_VM.session.snapshot()) → compress (zstd/gzip via the shared serialization pipeline; QuickJS heaps compress ~4×, measured 16.5 MB → 3.9 MB) → encrypt with the run's key when configured →world.snapshots.savewith the events cursor at the VM's feed frontier.world.snapshots.load→ decrypt → decompress →QuickJS.restoreover the cached WASM module, re-register host callbacks, fetch events from the snapshot's cursor and feed only the delta. Runs seamlessly through PR 2's inline continuation loop.Determinism model (restore + partial replay)
The threshold model's new mechanism vs. the original branch: a resumption may restore a snapshot older than the log head (suspensions since the snapshot weren't persisted) and must deterministically re-derive everything in between:
eventsCursor: the heap already consumed pre-snapshot draws, so re-seeding from the base would replay the first-N draws and collide with recorded correlationIds. The cursor is identical for every resume from the same snapshot (concurrent resumes still collide ids for the world's dedup) and advances only when a newer snapshot is taken.Validation
WORKFLOW_SNAPSHOT_THRESHOLD=1(maximum churn: snapshot on every qualifying suspension), wall clock within ~10% of the node baselinerestored: trueresumptions, save/restore/delete lifecycle, and threshold gating (threshold=100 short run ⇒ zero snapshots, pure replay)quickjs-snapshotmatrix leg (nextjs-turbopack, threshold=1) across local dev/prod/postgres e2e jobsNotes / follow-ups
quickjs-wasibuild that produced them. Per the deployment contract (runs continue on the version they started on — free on Vercel), this is a non-issue in production; environments without skew protection are covered by the restore-failure fallback to full replay.WORKFLOW_SNAPSHOT_THRESHOLDsection added to v5 Runtime Tuning.