Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/drop-precondition-guard-capability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@workflow/world': patch
'@workflow/core': patch
---

Remove the `preconditionGuard` World capability. Every World is now assumed to be able to reject a stale replay-context write, so the behaviors that keyed on the flag apply everywhere.
6 changes: 6 additions & 0 deletions .changeset/require-slot-event-ids.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@workflow/world': patch
'@workflow/core': patch
---

Require every event id the runtime reads to be a log position. `requireEventSlot` replaces the lenient decode that returned "no position" for an id that is not a slot.
5 changes: 5 additions & 0 deletions .changeset/resilient-step-dispatch-off.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/core': patch
---

Turn resilient step dispatch off by default. Set `WORKFLOW_RESILIENT_STEP_DISPATCH=1` to opt back in.
6 changes: 6 additions & 0 deletions .changeset/step-dispatch-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@workflow/world': patch
'workflow': patch
---

Carry step input, attempt number, and log position on step dispatch messages so queued steps hydrate their input and stop retrying past maxRetries on Worlds without step rows
5 changes: 5 additions & 0 deletions .changeset/step-started-attempt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/core': patch
---

State the attempt number on `step_started`, so a World that keeps no step row can still report which attempt a step is on.
5 changes: 5 additions & 0 deletions .changeset/straggler-after-step-outcome.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/core': patch
---

Ignore a `step_started` or `step_retrying` written behind a step's recorded result instead of failing the run with `CORRUPTED_EVENT_LOG`. A losing attempt that outlives the winning one produces exactly that log, and neither writer was wrong.
5 changes: 0 additions & 5 deletions .changeset/windows-preload-timeout.md

This file was deleted.

5 changes: 5 additions & 0 deletions .changeset/world-vercel-v5-events.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/world-vercel': patch
---

Send event reads and writes to the Vercel World's v5 event API.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Send event reads and writes to the Vercel World's v5 event API.
Perf: use the v5 event API for event creations

10 changes: 5 additions & 5 deletions .github/workflows/world-sim.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,20 @@ name: World Sim
# Plays the deterministic scenario book (`workbench/sim-world`) against the
# runtime in this commit, once per log world, and publishes the two summaries.
#
# This lane never blocks a merge, by design. Six scenarios in the book fail on
# This lane never blocks a merge, by design. Three scenarios in the book fail on
# purpose: each one is a reproduction of a corruption the runtime can still
# produce, stating the outcome its own durable log implies, and staying red
# until the runtime gets there. A gate that goes red on every PR is a gate
# everyone learns to ignore, so the job publishes numbers instead of verdicts —
# and the number to watch is in the comment, not the check mark.
#
# mint-ordered (production): 35 passed, 6 failed, 6 violations
# mint-ordered (production): 38 passed, 3 failed, 3 violations
# append-only: 41 passed, 0 failed, 0 violations
#
# A seventh red is a regression. Five means something got fixed and a scenario
# A fourth red is a regression. Two means something got fixed and a scenario
# is ready to retire. The append-only column is the measurement the pair exists
# for: it says which of the six would close if event positions were assigned at
# commit instead of at the handler's mint.
# for: it says which of the three would close if event positions were assigned
# at commit instead of at the handler's mint.

on:
push:
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ workbench/nextjs-*/public/.well-known/workflow
workbench/sveltekit/static/.well-known/workflow


# E2E diagnostics sidecar files (written to the repo root by writeDiagnosticsSidecar()
# in packages/core/e2e/utils.ts during e2e runs)
e2e-diagnostics-*.json

# Event log race repro output (written to the repo root by the harness and by
# scripts/event-log-race-repro-local.sh)
event-log-race-repro-results.json
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ related:

`PreconditionFailedError` is thrown by world implementations when an event creation is rejected because the client's event-log snapshot is stale: the log already held more events than the position the creation named. It corresponds to HTTP 412 Precondition Failed semantics.

This only occurs against a world that fences on that position (`capabilities.preconditionGuard` — see [Stale-write rejection](/docs/configuration/runtime-tuning#stale-write-rejection)); event creations that carry no position are never rejected with this error.
This only occurs against a world that fences on that position (see [Stale-write rejection](/docs/configuration/runtime-tuning#stale-write-rejection)); event creations that carry no position are never rejected with this error.

A world rejects only on evidence and accepts the creation whenever it cannot decide, so this error always means the snapshot really was stale — but not receiving it does not prove the snapshot was current.

Expand Down
16 changes: 8 additions & 8 deletions docs/content/docs/v5/configuration/runtime-tuning.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,22 +73,22 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL

### `WORKFLOW_RESILIENT_STEP_DISPATCH`

- Default: enabled
- Default: disabled
- When a suspension hands newly created steps to the queue, the runtime publishes each step's execution message in parallel with its `step_created` event write instead of sequencing them, cutting a round trip per dispatched step. The message also carries the serialized step input (`stepInput`), so a transient `step_created` write failure (429 / 5xx / transport) still executes the step — the queue consumer idempotently re-ensures the event before running it, converging with the producer's write on the step's correlation ID. This mirrors resilient start (`runInput`) and the lazy hook resume (`hookInput`).
- The runtime falls back to the sequential create-then-publish dispatch automatically when the step input is too large to inline on the queue message, when the run's queue transport cannot carry binary payloads (pre-CBOR spec versions), or — on the `node` VM engine, whose suspension writes are replay-context writes — when the World can [reject a write as stale](#stale-write-rejection) (`capabilities.preconditionGuard`; the Vercel World declares it): a rejected `step_created` must not be materializable through the queue side-channel, and only sequencing the publish after the create gives the message a happens-after edge over the create's verdict. The `quickjs` engine's suspension writes are not replay-context writes, so it uses resilient dispatch against every World.
- It is off by default because the publish races the create's verdict. A World that [rejects the `step_created` as stale](#stale-write-rejection) sends the runtime back to replay from a corrected log, but the message carrying the payload is already out, so the consumer can materialize a step the World refused. Nothing orders the create's refusal before the consumer's redelivery re-ensure, so the sequential path is the only one that gives the message a happens-after edge over the create's verdict.
- Even when enabled, the runtime falls back to the sequential create-then-publish dispatch when the step input is too large to inline on the queue message, or when the run's queue transport cannot carry binary payloads (pre-CBOR spec versions).
- Producer-side recoveries are reported on the suspension span as `workflow.step.resilient_dispatch_recovered`; a consumer that materialized the event reports `workflow.step.resilient_dispatch_materialized`.
- Set `0` to force the sequential dispatch as a kill switch.
- Set `1` to enable it.

### Stale-write rejection

- Not a variable: this is what a World declaring `capabilities.preconditionGuard` does, and what the runtime does about it.
- Not a variable: this is what a World that fences on the replayed-from position does, and what the runtime does about it. The runtime assumes any World may.
- A replay-context event creation names the position it replayed from (`eventCount`, the number of events the replay had loaded), and a World that fences on it rejects the creation with 412 ([`PreconditionFailedError`](/docs/api-reference/workflow-errors/precondition-failed-error)) when the log already held more than that. The position is derived from the run's event IDs, so it is sent only for a run whose World numbers events by position; a run on the older ID scheme, and any caller with no loaded log to be stale against, sends none and is never rejected.
- On rejection the runtime restarts the replay in the same invocation from a corrected event log, and falls back to a re-invocation with a fresh replay once the restart budget is spent. The rejected write is never retried as-is: a replay working from a corrected log derives different events, so only a fresh replay may write again.
- Against a fencing World the runtime also keeps the per-step event-log delta optimization (consuming the delta returned by a step's terminal write instead of issuing an extra `events.list` per step) active while the run has an open hook. Without a fence, an open hook disables it.
- While a hook is open on a fencing World, inline steps take the await-then-run path even when optimistic inline start is enabled: the step's `step_started` claim is awaited before the body runs, so a claim rejected as stale never executes user code.
- Worlds that do not fence ignore the position and must not declare the capability, so the dependent optimizations stay off against them.
- The runtime keeps the per-step event-log delta optimization (consuming the delta returned by a step's terminal write instead of issuing an extra `events.list` per step) active while the run has an open hook, because a `hook_received` missed by the delta window is what a fence rejects.
- While a hook is open, inline steps take the await-then-run path even when optimistic inline start is enabled: the step's `step_started` claim is awaited before the body runs, so a claim rejected as stale never executes user code.
- A fence only ever rejects on evidence, and fails open in every other case: a World that cannot decide must accept the write. A rejection therefore always means the position really was stale, but the absence of one does not prove it was current.
- The Vercel World declares the capability. It does not fence a run that uses [slot-numbered event IDs](/docs/how-it-works/event-sourcing#event-ids), because such a run has no position to reject: the World assigns each event its slot at commit time and reports back the slots the write skipped over.
- The Vercel World does not fence a run that uses [slot-numbered event IDs](/docs/how-it-works/event-sourcing#event-ids), because such a run has no position to reject: the World assigns each event its slot at commit time and reports back the slots the write skipped over.

### `WORKFLOW_SLOT_GAP_CHECK`

Expand Down
5 changes: 2 additions & 3 deletions docs/content/worlds/v5/building-a-world.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ interface WorldCapabilities {
active: boolean;
};
slotEventIds?: boolean;
preconditionGuard?: boolean;
}

interface World extends Storage, Queue, Streamer {
Expand All @@ -49,7 +48,7 @@ interface World extends Storage, Queue, Streamer {
}
```

The optional `capabilities` object advertises additional behavior. Set `hookRetention.active` to `true` only when the World implements Hook token retention, `slotEventIds` when it allocates [slot-numbered event IDs](#event-id-allocation), and `preconditionGuard` when it can [reject a stale write](#optional-rejecting-a-stale-write). The optional `start()` method initializes background tasks (for example, queue polling). The optional `close()` method releases resources like connection pools and listeners. The optional `getEncryptionKeyForRun()` method returns the AES-256 key used to encrypt data for a run; if it is not implemented, encryption is disabled.
The optional `capabilities` object advertises additional behavior. Set `hookRetention.active` to `true` only when the World implements Hook token retention, and `slotEventIds` when it allocates [slot-numbered event IDs](#event-id-allocation). The optional `start()` method initializes background tasks (for example, queue polling). The optional `close()` method releases resources like connection pools and listeners. The optional `getEncryptionKeyForRun()` method returns the AES-256 key used to encrypt data for a run; if it is not implemented, encryption is disabled.

## The Event Log Model

Expand Down Expand Up @@ -132,7 +131,7 @@ Two rules make this safe:

A rejection may optionally carry the events the caller was missing, as `{ events, cursor }` on the error's `details`. Only include them when you can prove the set is complete — that those events fully account for the discrepancy and are not truncated — and that every one of them belongs to the run being written. The runtime merges them straight into the replay's event log, so anything else there is worse than no delta at all. Otherwise omit them, and the runtime performs a full reload instead.

Declare `capabilities.preconditionGuard` if your World can refuse a write this way. The runtime reads it as "a write can come back refused", not as a promise that any particular one will be, and three behaviors key on it: the per-step event-log delta optimization stays enabled while the run has an open hook, an inline step's `step_started` claim is awaited before the body runs, and a `step_created` publish is sequenced after the create on the `node` VM engine. A World that accepts `eventCount` and ignores it must leave the capability unset — sending a position is not the same as one being enforced.
The runtime assumes any World may refuse a write this way, and shapes three behaviors around it: the per-step event-log delta optimization stays enabled while the run has an open hook, an inline step's `step_started` claim is awaited before the body runs, and a `step_created` publish is sequenced after the create on the `node` VM engine. A World that accepts `eventCount` and ignores it pays those costs without the benefit, which is another reason to prefer reporting the skipped events over rejecting.

## Queue Interface

Expand Down
8 changes: 0 additions & 8 deletions e2e-diagnostics-nextjs-turbopack-vercel.json

This file was deleted.

46 changes: 14 additions & 32 deletions packages/core/e2e/e2e.test.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
import fs from 'node:fs';
import path from 'node:path';
import { setTimeout as sleep } from 'node:timers/promises';
Expand Down Expand Up @@ -45,6 +45,7 @@
isLocalDeployment,
setupRunTracking,
setupWorld,
stepFailedError,
trackRun,
writeDiagnosticsSidecar,
} from './utils';
Expand Down Expand Up @@ -1314,20 +1315,10 @@
expect(result.stack).not.toContain('99_e2e.ts');
}

// Verify step failed via CLI (--withData needed to resolve errorRef)
const { json: steps } = await cliInspectJson(
`steps --runId ${run.runId} --withData`
);
const failedStep = steps.find((s: any) =>
s.stepName.includes('errorStepFn')
);
expect(failedStep.status).toBe('failed');
// The CLI hydrates `step.error` from the serialization pipeline.
// Errors thrown from steps are wrapped in `FatalError` by the
// step executor, which serializes via the Instance reducer
// (`{ classId, data }`); the CLI surfaces unregistered class
// instances as placeholders with the original `data` payload.
const errorData = failedStep.error.data ?? failedStep.error;
// Verify the step failed, and that its error survived the write.
// The error comes from the event log rather than the step listing
// — see `stepFailedError`.
const errorData = await stepFailedError(run.runId, 'errorStepFn');
expect(errorData.message).toContain('Step error message');

// Step error stack should contain the original step function name
Expand Down Expand Up @@ -1376,17 +1367,11 @@
expect(result.stack).not.toContain('helpers.ts');
}

// Verify step failed via CLI - same stack info available there too (--withData needed to resolve errorRef)
const { json: steps } = await cliInspectJson(
`steps --runId ${run.runId} --withData`
);
const failedStep = steps.find((s: any) =>
s.stepName.includes('stepThatThrowsFromHelper')
// Same stack info is available on the durable event too.
const errorData = await stepFailedError(
run.runId,
'stepThatThrowsFromHelper'
);
expect(failedStep.status).toBe('failed');
// See note above: serialized step errors arrive as Instance refs
// when the FatalError class isn't registered in this process.
const errorData = failedStep.error.data ?? failedStep.error;
if (hasNestedStepStackFrames()) {
expect(errorData.stack).toContain('throwErrorFromStep');
}
Expand Down Expand Up @@ -1418,12 +1403,11 @@

expect(result.finalAttempt).toBe(3);

// --withData forces the storage-backed listing: the analytics
// listing may omit the attempt column entirely (it is optional in
// the analytics schema), so only the durable step entity can be
// asserted on. Poll because rows for a just-finished run can lag.
// The analytics listing reports `attempt` as the number of starts
// the log holds, which is the number the step row's counter held.
// Poll because rows for a just-finished run can lag.
const steps = await cliInspectJsonUntil(
`steps --runId ${run.runId} --withData`,
`steps --runId ${run.runId}`,
(json) =>
json.some(
(s: any) =>
Expand Down Expand Up @@ -1455,10 +1439,8 @@
// (which inspect the value inside the SWC-instrumented workflow).
// Here we only assert step lifecycle behavior.

// --withData forces the storage-backed listing — see the
// retry-success test above.
const steps = await cliInspectJsonUntil(
`steps --runId ${run.runId} --withData`,
`steps --runId ${run.runId}`,
(json) =>
json.some(
(s: any) =>
Expand Down
Loading
Loading