[pull] main from vercel:main - #513
Merged
Merged
Conversation
…ublishes (#3365) * feat(world,world-vercel,core): resilient step dispatch (parallel step_created + queue publish) Newly created steps are handed to the queue in parallel with their step_created event write, with the serialized input carried on the message (stepInput) so the queue consumer can idempotently re-ensure the event when the direct write failed transiently — mirroring resilient start (runInput) and resilient hook resume (hookInput). - @workflow/world: stepInput on WorkflowInvokePayload, CreateEventParams.viaStepDispatch, WorldCapabilities.resilientStepDispatch - core (node:vm): suspension handler publishes eligible steps alongside their create; the dispatch pass skips them (queuedStepCorrelationIds) - core (quickjs): dispatchPendingOps does the same for overflow steps; the ineligible fallback is now published in parallel too (removes the serial per-step enqueue loop) - consumer: on a redelivery, a stepInput-carrying message re-ensures step_created (marked viaStepDispatch) before executing - under an enforced precondition guard the parallel path requires backend cooperation (capabilities.resilientStepDispatch, declared by world-vercel): a 412-rejected step's in-flight dispatch is revoked server-side and its re-ensure refused - step dispatch/retry idempotency keys are step-identity-scoped (cid + hashed step name) so a revoked message for a reassigned correlation id cannot absorb the corrected schedule's dispatch - kill switch: WORKFLOW_RESILIENT_STEP_DISPATCH=0 * Validate stepInput.input as Uint8Array at the schema boundary Review feedback: producers only attach stepInput when the dehydrated input is binary and the queue transport preserves bytes (CBOR), so a non-binary value means the payload was mangled in transit. Enforcing Uint8Array in StepDispatchInputSchema fails the message parse instead of silently writing non-binary data into a step_created, and types the consumer's re-ensure so the unchecked 'as SerializedData' cast goes away. * Keep sequential dispatch under an enforced precondition guard (drop the resilientStepDispatch capability lift) Review feedback (two P1s): backend-side revocation bookkeeping cannot carry the guard's correctness property across the queue side-channel — - nothing orders a slow guarded create's eventual 412 (the moment the backend learns the dispatch is poisoned and records the revocation marker) before the consumer's redelivery re-ensure, so attempt > 1 is a probabilistic mitigation, not a happens-before; and - a best-effort marker that fails open (Redis loss) cannot back a capability the SDK treats as a correctness attestation. Only sequencing the publish after the create gives the message a happens-after edge over the create's guard verdict, so the guard gate is now unconditional: worlds that enforce the precondition guard keep the sequential create-then-publish dispatch. The parallel resilient path remains for unguarded writes (the quickjs engine everywhere, and worlds without the guard). Removes WorldCapabilities.resilientStepDispatch and world-vercel's declaration; the viaStepDispatch flag is kept and re-documented as advisory (server-side defense-in-depth only). This also dissolves the reviewed dedupe hazard on the step-identity- scoped dispatch keys: with no 410-ack path in any real SDK flow, a message for a never-created step keeps redelivering until an entity exists, execution always hydrates input from the committed entity (never the message), and a name-mismatched stale start is skipped by the server's stepName fence. * Correct the MAX_RESILIENT_STEP_INPUT_BYTES rationale: VQS has no hard message-size cap 256 KB is the queue's inline-vs-S3 threshold, not a rejection limit (payloads above it spill to S3-backed storage transparently). The 128 KiB bound is a cost/latency choice — keep step messages on the inline path rather than paying an S3 double-hop for bytes that already live in the event log. * Recover a missing step in-band when a stepInput-carrying delivery beats its create Durabench parallel sweeps (guard-off, node engine) caught ~4-8% of fan-out runs stalling one branch for ~306s on the resilient dispatch path. Root cause: the consumer's step_created re-ensure was gated on metadata.attempt > 1, but world-vercel's failure-retry path re-enqueues a FRESH message whose attempt resets to 1 — so when a delivery beat the producer's parallel step_created write, every fast retry hit the same 'step not found' rejection with attempt 1, and the step only recovered when the ORIGINAL message's ~300s visibility-timeout redelivery finally arrived with attempt 2. The recovery is now in-band and attempt-independent: when a stepInput-carrying execution rejects with the step-missing signature (WorkflowWorldError, 404 or the local worlds' message shape), the consumer materializes the step_created from the message payload and retries the execution once within the same delivery. The eager attempt>1 ensure is kept as a round-trip saver on genuine redeliveries. Sweep effect expected: the 305-306s TTLS outliers disappear while the resilient path keeps its p50 win (1054ms vs 1425ms at 64 branches).
## Summary & Motivation Adds `StreamExpiredError` to `@workflow/errors`, carrying the run, stream, and server-reported expiry timestamp from workflow-server's 410 `stream-expired` envelope. The reconnect loop rethrows it instead of retrying, since retention expiry is terminal and a retry budget would only convert it into a generic exhaustion error. ## Test Plan Unit tests added for the 410 decoding path and the reconnect rethrow; typechecks pass across the touched packages.
## Summary - reject compact sparse arrays above the supported logical length at the main devalue hydration boundary - delegate accepted sparse-array construction to devalue - cover both current binary payloads and legacy flattened payloads ## Why Compact sparse-array encodings can represent a logical length that is disproportionate to the stored payload. Applying one codec-level bound keeps hydration predictable before downstream consumers process the decoded value. ## Impact Compact sparse arrays with logical lengths above 100,000 now fail hydration with a `RangeError`. Other payloads are unchanged. ## Verification - `pnpm --filter @workflow/core test` — 2,023 passed, 3 expected failures - `pnpm --filter @workflow/core typecheck` - `pnpm --filter @workflow/core build` - focused serialization suite — 154 passed - direct root-argument, bound-step, and aggregate-error payload checks - reuse, quality, and efficiency review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )