Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .changeset/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@
"workflow-sdk-compiler-playground",
"@workflow/docs-typecheck",
"@workflow/example-*",
"@workflow/vitest-workbench"
"@workflow/vitest-workbench",
"@workflow/world-sim",
"@workflow/sim-world-workbench"
]
}
6 changes: 0 additions & 6 deletions .changeset/per-kind-correlation-ids.md

This file was deleted.

9 changes: 9 additions & 0 deletions .changeset/slot-event-ids.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@workflow/world-postgres': patch
'@workflow/world-vercel': patch
'@workflow/world-local': patch
'@workflow/core': patch
'@workflow/world': patch
---

**Breaking**: SpecVersion 6: Event IDs are now a dense per-run slot number, allocated by the world at publish time so a rejected write leaves no gap in the event log. A replay tells the world how many events it had read and gets back the ones it did not see, so an event that arrives from outside the replay and lands ahead of an event the replay wrote no longer fails the run with `CORRUPTED_EVENT_LOG`: it is held for whichever part of the workflow awaits it. A gap in the numbering fails the run instead of being replayed over.
155 changes: 155 additions & 0 deletions .github/workflows/world-sim.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
name: World Sim

# Plays the deterministic scenario book (`workbench/sim-world`) against the
# runtime in this commit, once per log world, and publishes the two summaries.
#
# This lane never blocks a merge, by design. Six scenarios in the book fail on
# purpose: each one is a reproduction of a corruption the runtime can still
# produce, stating the outcome its own durable log implies, and staying red
# until the runtime gets there. A gate that goes red on every PR is a gate
# everyone learns to ignore, so the job publishes numbers instead of verdicts —
# and the number to watch is in the comment, not the check mark.
#
# mint-ordered (production): 35 passed, 6 failed, 6 violations
# append-only: 41 passed, 0 failed, 0 violations
#
# A seventh red is a regression. Five means something got fixed and a scenario
# is ready to retire. The append-only column is the measurement the pair exists
# for: it says which of the six would close if event positions were assigned at
# commit instead of at the handler's mint.

on:
push:
branches:
- main
tags:
- "!*"
pull_request:
types:
- opened
- reopened
- synchronize
workflow_dispatch:

concurrency:
# Unique group for this workflow and branch
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

permissions:
contents: read
pull-requests: write

jobs:
world-sim:
name: World Sim
runs-on: ubuntu-latest
# Non-blocking at the job level, not just on the sim steps: a failed
# checkout, install, or PR comment should not turn this lane into a red X
# on a PR it has nothing to say about. The step summary and the artifact
# still carry whatever did happen.
continue-on-error: true
# The book itself is ~11s per world on a laptop. Everything else here is
# install and build.
timeout-minutes: 15
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}

steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}

# The default build (every package, no workbench) rather than a filter
# narrowed to the sim's own dependencies: those two graphs reach most of
# the repo anyway — through `@workflow/builders`, which is what compiles
# the scenario workflows — and matching what the other lanes build is
# what keeps this one on their turbo cache instead of warming its own.
# The workbench itself has no build; `run.ts` is executed by Node's type
# stripping.
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev

# Both worlds run with their real exit codes: `continue-on-error` records
# the non-zero without ending the job, so the step's own status still
# says whether the book was clean. Paths are absolute because pnpm runs
# the script from the package directory, not the workspace root.
- name: Play the book (mint-ordered log)
id: mint
continue-on-error: true
run: |
pnpm --filter @workflow/sim-world-workbench sim \
--no-color \
--title 'Mint-ordered log' \
--summary-file "${{ github.workspace }}/world-sim-mint.md" \
--detail-file "${{ github.workspace }}/world-sim-mint.txt"

- name: Play the book (append-only log)
id: append-only
continue-on-error: true
run: |
pnpm --filter @workflow/sim-world-workbench sim \
--no-color \
--append-only \
--title 'Append-only log' \
--summary-file "${{ github.workspace }}/world-sim-append-only.md" \
--detail-file "${{ github.workspace }}/world-sim-append-only.txt"

# Four visible lines when collapsed: the heading, one line saying what
# this is, and one per world. Everything else is behind a fold. The
# comment is reposted on every push to the PR, so what it costs when it
# has nothing new to say is the thing to keep small.
- name: Render summary
if: always()
run: |
run_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
{
echo "## Sim World"
echo
echo "Simulated world deterministic testing for races. [Traces]($run_url)"
echo
for world in mint append-only; do
if [ -f "world-sim-$world.md" ]; then
# The summary names its detail file by the path it was given,
# which is absolute so that pnpm's package-directory cwd cannot
# scatter them. Strip the workspace prefix back off, leaving
# the bare name the artifact below actually contains.
sed "s|${{ github.workspace }}/||g" "world-sim-$world.md"
# Blank line between the two folds. Adjacent HTML blocks with
# nothing between them get parsed as one, and the second world
# disappears into the first one's fold.
echo
else
echo "🟠 _The $world run produced no summary — see the job log._"
echo
fi
done
} | tee world-sim-summary.md >> "$GITHUB_STEP_SUMMARY"

# Skipped on forks, where `pull_request` grants read-only permissions and
# the write would fail. The step summary above is the fallback there.
- name: Update PR comment
if: >-
always() && github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository
continue-on-error: true
uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
with:
header: world-sim-results
path: world-sim-summary.md

- name: Upload traces
if: always()
uses: actions/upload-artifact@v4
with:
name: world-sim-traces
path: |
world-sim-summary.md
world-sim-mint.md
world-sim-mint.txt
world-sim-append-only.md
world-sim-append-only.txt
retention-days: 7
if-no-files-found: ignore
19 changes: 8 additions & 11 deletions docs/content/docs/v5/configuration/runtime-tuning.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,14 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
- As a result, 412 volume describes a workload rather than the health of a deployment, and a run that never sees one is not evidence the guard is inactive.
- Set `0` to disable.

### `WORKFLOW_SLOT_GAP_CHECK`

- Default: enabled
- A replay checks that the [event log](/docs/how-it-works/event-sourcing#event-ids) it loaded is dense before it runs, and fails the run with [`CORRUPTED_EVENT_LOG`](/docs/errors/corrupted-event-log) when a position below the log's highest holds no event. A log missing only its first position, meaning a run whose `run_created` is still being written, is left alone.
- A position can be briefly empty while the write that occupies it is still committing, so the check re-reads the log a few times before it decides, and the replay continues from whichever log it settled on.
- The check trades one failure for another. Most holes stand for an event that never happened, and replaying past those is correct. A hole standing for an event that did happen looks identical, and replaying past that one produces a run whose result is silently wrong. Failing is the recoverable side of that trade.
- Set `0` to replay across holes instead.

### `WORKFLOW_PRECONDITION_MAX_INPROCESS_RESTARTS`

- Default: `3`
Expand All @@ -101,17 +109,6 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
- Delay before a re-invocation caused by a rejected event creation.
- Unlike an in-process restart, which re-reads immediately, a re-invocation only happens once the in-process budget failed to catch up — so the delay gives the other writers a moment to quiesce.

### `WORKFLOW_PER_KIND_CORRELATION_IDS`

- Default: disabled
- Experimental. Gives each kind of entity a workflow creates — steps, waits, hooks, attribute writes, abort controllers, stream IDs — its own sequence of correlation IDs.
- With one sequence shared by every kind, an ID is an ordinal over the whole run, so a single extra draw of any kind shifts every ID after it. Two concurrent replays of the same run that disagree about one `sleep()` then assign different IDs to every step that follows, and each writes events the other can neither match nor consume, which fails the run with `CORRUPTED_EVENT_LOG`. Per-kind sequences confine that to the kind that actually differs.
- IDs remain ordered within a kind, so hooks created by your workflow are still listed in creation order. A hook the runtime creates for you, such as the one backing an abort controller, draws from its own kind and so is listed at an arbitrary position relative to your hooks rather than at its creation position.
- A run must replay under the scheme that minted its IDs. A replay that switches schemes mid-run assigns IDs its own earlier events do not carry, so it can consume none of them and the run fails.
- On Vercel, a run keeps replaying on the deployment it started on, so it only ever sees the value baked into that deployment. Changing the setting affects new runs only.
- Elsewhere — `@workflow/world-postgres`, `@workflow/world-local`, any self-hosted process — nothing pins a run to the code that started it. Turn the setting on during a quiet window with no runs in flight, and roll the new value out to your whole fleet at once: a rolling deploy that leaves both values live replays one run under two schemes concurrently, which is the failure the setting exists to reduce.
- Set `1` to enable.

## Inline execution

### `WORKFLOW_V2_TIMEOUT_MS`
Expand Down
11 changes: 7 additions & 4 deletions docs/content/docs/v5/errors/corrupted-event-log.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,18 @@ Workflow replay diverged <divergenceCount> times after <maxRecoveryReplays> reco

## Why This Happens

Workflows persist their progress as an ordered event log. During replay, the runtime processes each event in sequence — every event must be consumed by a matching callback (e.g., a step or sleep waiting for its result). When an event has no matching consumer, the runtime cannot advance past it, which would block all subsequent events and hang the workflow indefinitely.
Workflows persist their progress as an ordered event log. During replay, the runtime processes each event in sequence — every event must be consumed by a matching callback (e.g., a step or sleep waiting for its result). An event no callback ever claims is one the runtime would have to drop to finish the run, so it fails the run instead of returning a result that silently ignored it.

Instead of silently hanging, the runtime retries a divergent replay before failing the workflow and surfacing this terminal error.
A delivery written from outside the replay, such as a hook firing or a step completing on another invocation, can land ahead of the events the replay is writing itself. That is ordinary concurrency rather than corruption, so the runtime holds such an event and offers it to each consumer the replay registers afterwards. The failure comes only when the workflow function returns while an event is still held, at which point no consumer can ever appear. A replay that suspends still holding one reports it on the span (`workflow.events.parked.count`, `.event_id`, `.event_type`) and leaves the decision to the replay that follows.

Before failing, the runtime retries a divergent replay and surfaces this terminal error only if replay still cannot recover.

Common scenarios that produce this error:

1. **Duplicate completion events** — Two `wait_completed` events for a single `wait_created`, or two `step_completed` events for the same step. The first is consumed normally, but the second has no consumer.
2. **Orphaned events** — A `step_completed` or `wait_completed` event whose `correlationId` doesn't match any step or sleep in the workflow code.
1. **Duplicate completion events** — Two `wait_completed` events for a single `wait_created`, or two `step_completed` events for the same step. The first is consumed normally, and the second resolves something already resolved, so no later consumer can claim it.
2. **Orphaned events** — A `step_completed` or `wait_completed` event whose `correlationId` doesn't match any step or sleep in the workflow code, so the replay reaches its end still holding it.
3. **Events after terminal state** — An event that arrives after its corresponding step or wait has already reached a terminal state (e.g., `step_retrying` after `step_completed`).
4. **A hole in the log** — Events are numbered by their position in the run's log, and those positions are dense, so a position below the log's highest that holds no event means the log the replay loaded is incomplete. The runtime cannot tell a position no write ever occupied from one whose event it failed to read, so it refuses to replay rather than produce a result that may be silently wrong. See [`WORKFLOW_SLOT_GAP_CHECK`](/docs/configuration/runtime-tuning#workflow_slot_gap_check).

## What To Do

Expand Down
14 changes: 11 additions & 3 deletions docs/content/docs/v5/how-it-works/event-sourcing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -260,19 +260,27 @@ On Vercel, `requestId` is the platform request ID when available. Other worlds a

## Entity IDs

All entities in the Workflow SDK use a consistent ID format: a 4-character prefix followed by an underscore and a [ULID](https://github.com/ulid/spec) (Universally Unique Lexicographically Sortable Identifier).
All entities in the Workflow SDK use a consistent ID format: a 4-character prefix followed by an underscore and a fixed-width body. For every entity except events, that body is a [ULID](https://github.com/ulid/spec) (Universally Unique Lexicographically Sortable Identifier). An event's body is its slot number, described below.

| Entity | Prefix | Example |
|--------|--------|---------|
| Run | `wrun_` | `wrun_01HXYZ123ABC456DEF789GHJ` |
| Step | `step_` | `step_01HXYZ123ABC456DEF789GHJ` |
| Hook | `hook_` | `hook_01HXYZ123ABC456DEF789GHJ` |
| Wait | `wait_` | `wait_01HXYZ123ABC456DEF789GHJ` |
| Event | `evnt_` | `evnt_01HXYZ123ABC456DEF789GHJ` |
| Event | `evnt_` | `evnt_00000000000000000000000042` (slot 42) |
| Stream | `strm_` | `strm_01HXYZ123ABC456DEF789GHJ` |

**Why this format?**

- **Prefixes enable introspection**: Given any ID, you can immediately identify what type of entity it refers to. This makes debugging, logging, and cross-referencing entities across the system straightforward.

- **ULIDs enable chronological ordering**: Unlike UUIDs, ULIDs encode a timestamp in their first 48 bits, making them lexicographically sortable by creation time. This property is essential for the event log—events are always stored and retrieved in the correct chronological order simply by sorting their IDs.
- **Fixed-width bodies enable ordering**: Unlike UUIDs, these bodies sort lexicographically in creation order, so the event log is stored and retrieved in the correct order by sorting IDs alone. Slot numbers get that from counting at a fixed width, which makes string order the same as numeric order. ULIDs get it from the timestamp in their first 48 bits, which also makes a ULID's creation time recoverable from the ID itself.

### Event IDs

An event ID is a **slot number**: the event's 1-based position in the run's event log, zero-padded to the same width as a ULID. The world assigns it when the event is published, so two writers racing to append never claim the same position and a rejected write leaves no gap behind. Slots are dense, and unique only within a run, so an event ID identifies an event only when paired with its `runId`.

Density is what lets a reader tell a complete log from an incomplete one by its length alone. A replay that loads a log with a position missing below the highest one it can see cannot tell an event that was never written from one it failed to read, so it fails the run with [`CORRUPTED_EVENT_LOG`](/docs/errors/corrupted-event-log) rather than replay across the hole. See [`WORKFLOW_SLOT_GAP_CHECK`](/docs/configuration/runtime-tuning#workflow_slot_gap_check).

A slot ID carries no timestamp. Zero-padded decimal digits are a subset of the ULID alphabet, so a slot ID passes ULID validation and sorts correctly, but decoding its first 48 bits yields the Unix epoch instead of a creation time. Read `createdAt` on the event when you need to know when it was written, and don't decode the time from an `evnt_` ID you get back from an API, a log line, or a cursor.
Loading
Loading