Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/catchable-step-arg-serialization-errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/core': patch
---

Step-argument serialization failures now fail the step with a catchable `SerializationError` (via a `step_failed` event, like a step-body failure) instead of failing the run from outside the workflow, and when uncaught they fail the run immediately as a `USER_ERROR` rather than retrying until max queue deliveries.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

This changes a user-facing guarantee (a step-argument serialization failure is now catchable in workflow code, and classifies as USER_ERROR when uncaught), and docs doesn't say either way today. docs/content/docs/v5/errors/serialization-failed.mdx is the natural place — it lists "Serializing step arguments" as a cause but never says whether the failure is observable from the workflow — and the try/catch shape would fit foundations/errors-and-retries.mdx next to the error-code table.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in 5480141, in both suggested places (v5): errors/serialization-failed.mdx gains a "Where the Error Surfaces" section spelling out per boundary what's observable — start() throws synchronously, step arguments/return values fail the step catchably (no retries, marker placeholder input), workflow return values fail the run — plus the uncaught → immediate USER_ERROR, no-retry semantics. foundations/errors-and-retries.mdx gains a "Serialization Failures" section with the try/catch shape next to the retry docs, cross-linking the error page. v4 docs are intentionally untouched here — they should change with the stable backport, since v4's behavior only changes when that ships.

28 changes: 28 additions & 0 deletions docs/content/docs/v5/errors/serialization-failed.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,34 @@ This error can appear when:
- Serializing step arguments
- Serializing step return values

## Where the Error Surfaces

Where you observe the failure depends on which boundary it crosses:

- **Workflow arguments** — `start()` throws synchronously in your application code.
- **Step arguments and step return values** — the *step* fails with the `SerializationError`, exactly like a step whose body threw a fatal error: no retries (the failure is deterministic), and a `try/catch` around the step call in your workflow code observes it. The step's recorded input shows `[input unavailable: step argument serialization failed]` when the arguments were the unserializable part.
- **Workflow return values** — the workflow body has already returned, so nothing can catch it; the run fails.

```typescript lineNumbers
async function stepWithBadArguments(value: unknown) {
"use step";
return value;
}

export async function processWorkflow(someValue: unknown) {
"use workflow";

try {
await stepWithBadArguments(someValue);
} catch (err) {
// err.name === "SerializationError"
// "Failed to serialize step arguments at path ..."
}
}
```

Uncaught, the error propagates out of the workflow body and the run fails immediately with the error code `USER_ERROR` — it does not retry.

## Why This Happens

Workflows persist their state using an event log. Every value that crosses execution boundaries must be:
Expand Down
25 changes: 25 additions & 0 deletions docs/content/docs/v5/foundations/errors-and-retries.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,31 @@ callApi.maxRetries = 5; // Retry up to 5 times on failure (6 total attempts)
step can run up to 4 times total (1 initial attempt + 3 retries).
</Callout>

## Serialization Failures

A step whose arguments or return value cannot be [serialized](/docs/foundations/serialization) fails like a step whose body threw a `FatalError`: the failure is deterministic, so it skips the retry loop, and a `try/catch` around the step call observes the `SerializationError`:

```typescript lineNumbers
async function someStep(input: unknown) {
"use step";
return input;
}

export async function myWorkflow(input: unknown) {
"use workflow";

try {
await someStep(input);
} catch (err) {
if ((err as Error).name === "SerializationError") {
// e.g. `Failed to serialize step arguments at path "..."`
}
}
}
```

Uncaught, the run fails immediately with the `USER_ERROR` code — without retrying. See [serialization-failed](/docs/errors/serialization-failed) for common causes and fixes.

## Error Codes

When a workflow run fails, the error includes an `errorCode` that classifies the failure, alongside the original thrown value (preserved as `cause`):
Expand Down
114 changes: 114 additions & 0 deletions packages/core/e2e/e2e.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1719,6 +1719,120 @@ describe('e2e', () => {
);
});

describe('serialization failures', () => {
test(
'step-argument serialization failure is catchable in workflow code',

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Blocking

Both step-argument tests fail on the QuickJS engine. On the current run every one of the 37 failing E2E jobs is a quickjs lane and every node lane passes, so this is a signature, not flake (E2E Local Dev/Postgres/Prod + E2E Windows).

From e2e-results-local-dev-astro-stable-quickjs:

FAIL  serialization failures > step-argument serialization failure is catchable in workflow code
Error: Workflow run "wrun_..." failed: Cannot stringify arbitrary non-POJOs
Caused by: DevalueError: Cannot stringify arbitrary non-POJOs
    at eval (packages/core/src/runtime/quickjs-serde.ts:2139:22)
    at Object.serialize (packages/core/src/runtime/quickjs-serde.ts:2137:28)
    at dumpPendingOps (packages/core/src/runtime/quickjs-runtime.ts:2515:23)
    at checkWorkflowState (packages/core/src/runtime/quickjs-runtime.ts:2662:26)

FAIL  serialization failures > uncaught step-argument serialization failure fails the run as USER_ERROR ...
AssertionError: expected 'Workflow run "wrun_..."' to contain 'Failed to serialize step arguments'

On QuickJS the pending op's raw fields are serialized in dumpPendingOps (serde.serialize(valueHandle)) while extracting the VM's suspension state, i.e. before handleSuspension ever sees the queue item. So on that engine:

  • the new catch around dehydrateStepArguments cannot fire, and the failure still leaves the workflow from the outside, unobservable — the exact bug this PR fixes for node:vm;
  • the throw is a bare DevalueError, not a SerializationError, so it also misses the framing/hint (Failed to serialize step arguments at path "...") the uncaught test asserts.

QuickJS is opt-in rather than the default, but it is a supported engine and it is CI-gated, so this has to be resolved either way. The equivalent per-op treatment in dumpPendingOps (reframe as SerializationError and route it to the same finalizeUnserializableStep) would make both engines agree; wrapping only the message would fix the second test while leaving QuickJS users with the unobservable failure.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5480141 — both engines now agree. dumpPendingOps catches the per-op serialize of a step's input, reframes it through formatSerializationError('step arguments', …) into a SerializationError (same framed message + hint as dehydrateStepArguments), and surfaces it on the op (PendingStep.serializationError, input absent) instead of failing the whole collection. Other raw fields (hook metadata, abort payloads) keep the throwing behavior, matching the node:vm engine's scope.

dispatchPendingOps then finalizes such steps as step_created (placeholder input) + step_failed — the QuickJS analog of finalizeUnserializableStep — and the inline loop excludes them from inline claims and overflow publishes, marks them handled, and raises pendingRequeueSignal so the terminal event is observed even when the feed lags and the failed step was the only pending work. The live-VM feed's existing step_failed processing rejects the resolver, so the catchable/uncaught semantics are identical to node:vm (verified: both step-argument e2e tests plus parallel/FatalError/hookWithSleepWorkflow slices green against a local WORKFLOW_VM=quickjs dev server).

{ timeout: 60_000 },
async () => {
// Passing an unserializable value (a class instance with no serde
// model) to a step must fail THAT STEP — step_created +
// step_failed — not the whole run, so a try/catch around the step
// call observes the SerializationError.
const run = await start(
await e2e('serializationErrorStepArgsCaught'),
[]
);
const result = await run.returnValue;

expect(result.caught).toBe(true);
expect(result.name).toBe('SerializationError');
expect(result.messageIncludesStepArguments).toBe(true);

// The workflow completed (the error was caught) …
const { json: runData } = await cliInspectJson(`runs ${run.runId}`);
expect(runData.status).toBe('completed');

// … and the step itself is recorded as failed.
const steps = await cliInspectJsonUntil(
`steps --runId ${run.runId}`,
(json) =>
json.some(
(s: any) =>
s.stepName.includes('acceptAnyValue') && s.status === 'failed'
)
);
const step = steps.find((s: any) =>
s.stepName.includes('acceptAnyValue')
);
expect(step.status).toBe('failed');
}
);

test(
'uncaught step-argument serialization failure fails the run as USER_ERROR without redelivery retries',
{ timeout: 60_000 },
async () => {
// Regression coverage for the production failure mode where a
// step-argument serialization error caused the run to redeliver
// until "exceeded max deliveries (49/48)". The run must fail
// promptly (well within this test's timeout — 48 redeliveries
// with backoff would take many minutes) and classify as
// USER_ERROR, not MAX_DELIVERIES_EXCEEDED.
const run = await start(
await e2e('serializationErrorStepArgsUncaught'),
[]
);
const error = await run.returnValue.catch((e: unknown) => e);

expect(WorkflowRunFailedError.is(error)).toBe(true);
assert(WorkflowRunFailedError.is(error));
expect(error.errorCode).toBe('USER_ERROR');
expect(String(error.message)).toContain(
'Failed to serialize step arguments'
);

const { json: runData } = await cliInspectJson(`runs ${run.runId}`);
expect(runData.status).toBe('failed');
expect(runData.errorCode).toBe('USER_ERROR');
}
);

test(
'step-return-value serialization failure is catchable in workflow code',
{ timeout: 60_000 },
async () => {
// The step executor treats a return-value SerializationError as
// fatal (skipping the retry loop) and writes step_failed, so the
// workflow's try/catch observes it.
const run = await start(
await e2e('serializationErrorStepReturnCaught'),
[]
);
const result = await run.returnValue;

expect(result.caught).toBe(true);
expect(result.name).toBe('SerializationError');
expect(result.messageIncludesReturnValue).toBe(true);

const { json: runData } = await cliInspectJson(`runs ${run.runId}`);
expect(runData.status).toBe('completed');
}
);

test(
'uncaught step-return-value serialization failure fails the run as USER_ERROR',
{ timeout: 60_000 },
async () => {
const run = await start(
await e2e('serializationErrorStepReturnUncaught'),
[]
);
const error = await run.returnValue.catch((e: unknown) => e);

expect(WorkflowRunFailedError.is(error)).toBe(true);
assert(WorkflowRunFailedError.is(error));
expect(error.errorCode).toBe('USER_ERROR');
expect(String(error.message)).toContain(
'Failed to serialize step return value'
);

const { json: runData } = await cliInspectJson(`runs ${run.runId}`);
expect(runData.status).toBe('failed');
expect(runData.errorCode).toBe('USER_ERROR');
}
);
});

describe('not registered', () => {
// JS-only: the workflowId is hand-built in the JS scheme, so on another
// language it names nothing rather than naming something missing.
Expand Down
48 changes: 48 additions & 0 deletions packages/core/src/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3329,6 +3329,54 @@ export function workflowEntrypoint(
continue;
}

// Steps whose arguments failed to serialize were
// finalized by the suspension handler as step_created
// + step_failed (see finalizeUnserializableStep). No
// step-execution message is dispatched for them, so
// when such a step is the only pending work nothing
// would ever re-invoke the run — replay in-process
// over the reloaded log instead. The replay rejects
// the step's promise with the SerializationError,
// which a try/catch around the step call observes;
// uncaught, it propagates out of the workflow body
// and fails the run as a USER_ERROR. Healthy sibling
// steps are not dispatched this pass: the replay
// re-suspends over their already-committed
// step_created events and the next pass dispatches
// them as usual.
if (
suspensionResult.failedStepCorrelationIds.size > 0
) {
// Join the batched fan-out's trailing chunk commits
// and step-message publishes before continuing,
// exactly like the two joins on the dispatch paths
// below: this invocation must not proceed (and
// eventually ack) before every create and publish
// it launched is durable. A rejection propagates
// like theirs — transient world errors rethrow to
// the queue for redelivery.
await suspensionResult.deferredBatchWork;
// Inline steps whose pair-folded step_started this
// pass already committed (`inlineClaims`) are
// deliberately NOT executed on this pass: the
// forced replay below re-suspends over the same
// pending steps, and owned recovery (the claims
// carry this message's ownerMessageId) re-executes
// them there. Its "previous delivery crashed
// mid-body" log is a misnomer on this path —
// nothing crashed, the bodies were never started.
//
// The failed dehydration may have executed
// workflow-owned code (getters/proxies) before
// throwing; demote to a cold replay rather than
// resume a VM that may have diverged. This is a
// rare terminal-error path, so the replay cost is
// irrelevant next to the divergence risk.
retainedSession = null;
eventLog = nextEventLogLoad(eventLog);
continue;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Blocking

This continue is above the only place suspensionResult.deferredBatchWork is joined (runtime.ts:3739 and :4125) and above the only place a committed inlineClaims body is executed. Both can be live when a step fails serialization: the batched fan-out is on by default (WORKFLOW_BATCH_TRANSITIONS unset ⇒ enabled), the caller always passes allowDeferredBatchWork: true and an ownerMessageId, and the bad step neither blocks the fold nor the pair pre-claims.

Reproduced against this branch (34 steps, s1 pair-folded, s5 carrying a non-POJO, only the pair chunk released):

✓ failedStepCorrelationIds -> ['s5']
✓ events.create           -> [step_created s5, step_failed s5]
✓ deferredBatchWork        -> defined, still PENDING at return
✓ inlineClaims.get('s1')   -> { owned: true }   // step_started committed, body not run

and with the trailing publish rejecting:

✓ handleSuspension resolves normally
✓ deferredBatchWork rejects with 'queue publish failed'   // observed by nobody on this path

Two consequences:

  1. A trailing chunk commit or step-message publish that rejects is swallowed, so this invocation can ack having lost a publish. That is the property the joins below exist to protect ("this invocation must not ack before every create and publish is durable"). The next pass usually re-dispatches from pendingSteps, but only if its reload sees those steps as created — if it races the still-in-flight commits it designates them lazy-inline instead, their claims conflict, executeStep returns skipped, and nothing published a message for them.
  2. Inline steps whose step_started this pass already committed are left claimed and unexecuted. They are recovered on the next pass through owned recovery, which logs Re-executing inline steps owned by this queue message — a previous delivery crashed mid-body. That is misleading here (nothing crashed) and it is gated on isInlineOwnershipEnabled().

await suspensionResult.deferredBatchWork before the continue, letting a rejection propagate the way the two joins below do, covers (1) cheaply. (2) is worth at least a comment saying the claims are deliberately handed to owned recovery, if that is the intent.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5480141. The failed-step path now does await suspensionResult.deferredBatchWork before the continue, so a trailing chunk commit or step-message publish rejection propagates exactly like the two joins below (transient world errors rethrow to the queue for redelivery) instead of being swallowed after ack — covering (1).

For (2): the committed inlineClaims hand-off to owned recovery is deliberate — the forced replay re-suspends over the same pending steps and the claims carry this message's ownerMessageId, so owned recovery re-executes them on the next pass. Added a comment at the site saying exactly that, including that the "previous delivery crashed mid-body" log is a misnomer on this path (nothing crashed; the bodies were never started).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: [P1] Await suspensionResult.deferredBatchWork before replaying. handleSuspension is called with allowDeferredBatchWork: true, and a mixed serialization failure plus sufficiently large healthy fan-out can return while trailing batch commits and publishes are still running. This continue drops the only join for that promise, so those writes can race the event-log reload and a deferred rejection can be lost despite the documented must-await-before-ack contract. Please join the deferred work here and add a mixed bad-step and large-fan-out test that exercises a deferred rejection.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The join landed in 5480141: the failed-step path now does await suspensionResult.deferredBatchWork before the continue, with a comment tying it to the must-await-before-ack contract; a rejection propagates exactly like the two joins below (transient world errors rethrow to the queue for redelivery).

The requested mixed test is in 00b30cf (suspension-handler.test.ts, 'mixed bad step + large fan-out: deferred rejection still surfaces through deferredBatchWork'): 34 steps with one carrying unserializable args — asserts the bad step finalizes on the sequential path (step_created + step_failed), the healthy fan-out still returns live (pending) deferredBatchWork on the same result, and a trailing publish rejection surfaces through it. That pins the handler-side contract the runtime join consumes; the join itself is a one-liner sharing the propagation path of the existing joins.

}

const pendingSteps = suspensionResult.pendingSteps;

// Inline execution is gated on ownership. The
Expand Down
Loading
Loading