diff --git a/.changeset/workflow-lifecycle-hooks.md b/.changeset/workflow-lifecycle-hooks.md new file mode 100644 index 0000000000..ddbe454c4c --- /dev/null +++ b/.changeset/workflow-lifecycle-hooks.md @@ -0,0 +1,6 @@ +--- +'@workflow/core': minor +'workflow': minor +--- + +Add `registerLifecycleHooks` (exported from `workflow/api`) for registering global `onRunCompleted`/`onRunFailed` handlers that receive the lazily-hydrated `Run` instance (and, for failures, a `WorkflowRunFailedError` with the hydrated cause and error code), enabling centralized reporting (e.g. to Sentry) from `instrumentation.ts`. diff --git a/docs/content/docs/v5/api-reference/workflow-api/index.mdx b/docs/content/docs/v5/api-reference/workflow-api/index.mdx index 82d80d4bb6..214d249ec5 100644 --- a/docs/content/docs/v5/api-reference/workflow-api/index.mdx +++ b/docs/content/docs/v5/api-reference/workflow-api/index.mdx @@ -27,6 +27,9 @@ The API package is for access and introspection of workflow data to inspect runs Get workflow run status and metadata without waiting for completion. + + Observe run completions and failures with global handlers. + diff --git a/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx b/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx new file mode 100644 index 0000000000..078f6f7d57 --- /dev/null +++ b/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx @@ -0,0 +1,75 @@ +--- +title: registerLifecycleHooks +description: Register global handlers that observe workflow runs completing or failing. +type: reference +summary: Use registerLifecycleHooks to observe run completions and failures from one central place. +prerequisites: + - /docs/foundations/workflows-and-steps +related: + - /docs/observability/lifecycle-hooks + - /docs/api-reference/workflow-api/get-run +--- + +Registers global workflow lifecycle handlers, invoked by the runtime on the compute that records a run's terminal transition. Use it for centralized reporting, such as forwarding every failed run to Sentry, without wrapping each workflow body. + +Register early in the process lifecycle (in Next.js, `instrumentation.ts`) so handlers exist before the first run finishes. See the [lifecycle hooks guide](/docs/observability/lifecycle-hooks) for semantics and a full Sentry example. + +```typescript title="instrumentation.ts" lineNumbers +import { registerLifecycleHooks } from "workflow/api"; + +export function register() { + if (process.env.NEXT_RUNTIME === "nodejs") { + registerLifecycleHooks({ + async onRunCompleted({ run }) { + console.log(`Run ${run.runId} completed`); + }, + async onRunFailed({ run, error }) { + console.error(`Run ${run.runId} failed (${error.errorCode})`, error.cause); + }, + }); + } +} +``` + +## API Signature + +### Parameters + + + +### Returns + +Returns a function that unregisters these hooks. + +## Handlers + +Both handlers receive the run as a lazily-hydrated [`Run`](/docs/api-reference/workflow-api/get-run) instance. Accessors like `run.workflowName` and `run.returnValue` only fetch from the backend when used. + +### `onRunCompleted` + +Invoked when a workflow run completes successfully. + +| Parameter | Type | Description | +| --- | --- | --- | +| `params.run` | `Run` | The completed run. | + +### `onRunFailed` + +Invoked when a workflow run fails terminally (after any retries). + +| Parameter | Type | Description | +| --- | --- | --- | +| `params.run` | `Run` | The failed run. | +| `params.error` | `WorkflowRunFailedError` | The failure, in the same shape `run.returnValue` rejects with: `error.errorCode` carries the classification (e.g. `USER_ERROR`) and `error.cause` is the hydrated thrown value. | + +## Behavior + +- Handlers run on the host (full Node.js), never inside the workflow VM. Calling `registerLifecycleHooks` from workflow code throws. +- Handlers are fire-and-forget: they cannot delay or change the run's outcome, and the runtime logs and swallows a throwing handler. On serverless platforms, the runtime keeps the invocation alive with `waitUntil`. +- Handlers fire only on the invocation that wrote the terminal event. Transitions recorded outside your app's compute (e.g. a run cancelled from the CLI or dashboard) do not fire handlers. +- You can register multiple hook sets, and handlers run in registration order. diff --git a/docs/content/docs/v5/observability/lifecycle-hooks.mdx b/docs/content/docs/v5/observability/lifecycle-hooks.mdx new file mode 100644 index 0000000000..81599157e2 --- /dev/null +++ b/docs/content/docs/v5/observability/lifecycle-hooks.mdx @@ -0,0 +1,82 @@ +--- +title: Lifecycle Hooks +description: Register global handlers that observe workflow runs completing or failing, for centralized reporting to services like Sentry. +type: guide +summary: Observe run completions and failures from a single place with registerLifecycleHooks. +prerequisites: + - /docs/foundations/workflows-and-steps +related: + - /docs/observability + - /docs/observability/tracing + - /docs/errors +--- + +Lifecycle hooks let you register global handlers that the runtime invokes whenever a workflow run completes or fails. They observe even the failures that never reach a `try/catch` in workflow code, such as a replay timing out or a run exhausting its queue deliveries. The most common use is centralized error reporting, such as forwarding every failed run to a service like Sentry without wrapping each workflow body. + +## Registering hooks + +Call `registerLifecycleHooks` from `workflow/api` early in your application's lifecycle, so the handlers exist before the first run finishes. In Next.js, [`instrumentation.ts`](https://nextjs.org/docs/app/building-your-application/optimizing/instrumentation) is the natural place; in any other app, any module that loads at startup works. + +```typescript title="instrumentation.ts" lineNumbers +import { registerLifecycleHooks } from "workflow/api" + +export function register() { + if (process.env.NEXT_RUNTIME === "nodejs") { + registerLifecycleHooks({ + async onRunCompleted({ run }) { + console.log(`Run ${run.runId} completed`) + }, + async onRunFailed({ run, error }) { + console.error( + `Run ${run.runId} failed with ${error.errorCode}:`, + error.cause + ) + }, + }) + } +} +``` + +`registerLifecycleHooks` returns an unregister function. You can register multiple hook sets, and handlers run in registration order. + +## Handler parameters + +Both handlers receive the [`Run`](/docs/api-reference/workflow-api/get-run) instance for the transitioned run. The `Run` instance hydrates lazily, meaning accessors like `run.workflowName` or `run.returnValue` only fetch from the backend when the handler uses them, so a handler that filters on cheap metadata pays nothing for the runs it ignores. + +`onRunFailed` additionally receives the failure as a `WorkflowRunFailedError`, the same shape `run.returnValue` rejects with: + +- `error.errorCode`: the failure classification (`USER_ERROR`, `RUNTIME_ERROR`, `MAX_DELIVERIES_EXCEEDED`, and more). See [error codes](/docs/errors) for the full list. +- `error.cause`: the hydrated thrown value, with Error subclass identity, message, stack, and cause chain preserved. Any JavaScript value can be thrown, so this is typed `unknown`. + +## Reporting failed runs to Sentry + +```typescript title="instrumentation.ts" lineNumbers +import * as Sentry from "@sentry/nextjs" +import { registerLifecycleHooks } from "workflow/api" + +export function register() { + if (process.env.NEXT_RUNTIME === "nodejs") { + Sentry.init({ dsn: process.env.SENTRY_DSN }) + + registerLifecycleHooks({ + async onRunFailed({ run, error }) { + Sentry.captureException(error.cause ?? error, { + tags: { + workflowRunId: run.runId, + workflowName: await run.workflowName, + errorCode: error.errorCode, + }, + }) + await Sentry.flush(2000) + }, + }) + } +} +``` + +## How handlers behave + +- **Host-only.** Handlers run with full Node.js access, never inside the workflow's sandboxed VM. Calling `registerLifecycleHooks` from workflow code throws. +- **Fire-and-forget.** Handlers cannot delay or change the run's outcome. The runtime logs and swallows a throwing handler, and the remaining handlers still run. On serverless platforms, the runtime keeps the invocation alive with `waitUntil` while handlers finish. +- **Fires where the transition is recorded.** Handlers fire on the compute that wrote the terminal event. For a failure, that means after any retries are exhausted, exactly once per run under normal operation. Terminal transitions recorded outside your app's compute do **not** fire handlers. For example, when you cancel a run from the CLI or the Vercel dashboard, the backend writes that transition, so no handler runs. For a complete record of every transition, consume the [event log](/docs/how-it-works/event-sourcing) or set up alerts on the [observability](/docs/observability) surface instead. +- **Register everywhere your workflows run.** The terminal write can happen in any function invocation that processes the run's queue messages, so registration must run at startup in every instance of the app (which `instrumentation.ts` guarantees). diff --git a/docs/content/docs/v5/observability/meta.json b/docs/content/docs/v5/observability/meta.json index 617b3eeff4..c94b9f648d 100644 --- a/docs/content/docs/v5/observability/meta.json +++ b/docs/content/docs/v5/observability/meta.json @@ -1,4 +1,4 @@ { "title": "Observability", - "pages": ["tracing", "attributes"] + "pages": ["tracing", "attributes", "lifecycle-hooks"] } diff --git a/packages/core/e2e/e2e.test.ts b/packages/core/e2e/e2e.test.ts index d8868d2c47..b5b5b03f54 100644 --- a/packages/core/e2e/e2e.test.ts +++ b/packages/core/e2e/e2e.test.ts @@ -3666,6 +3666,78 @@ describe.concurrent('e2e', () => { } ); + // Lifecycle hooks (`registerLifecycleHooks`) are registered in the Next.js + // workbenches' instrumentation.ts (see lifecycle-hooks-e2e.ts there). The + // handlers report each lifecycleHookTarget* run's terminal transition by + // resuming the lifecycleHookObserver workflow's hook, a durable channel + // that works even when the terminal write happens on a different instance + // than the one serving these HTTP requests. + describe.skipIf(!isNextJsApp)('lifecycle hooks', () => { + test( + 'onRunCompleted receives the Run and can read its return value', + { timeout: 90_000 }, + async () => { + const token = `lifecycle-completed-${Math.random().toString(36).slice(2)}`; + + const observer = await start(await e2e('lifecycleHookObserver'), [ + token, + ]); + await waitForHook(token, { runId: observer.runId }); + + const target = await start(await e2e('lifecycleHookTargetCompleted'), [ + token, + ]); + await expect(target.returnValue).resolves.toMatchObject({ + outcome: 'completed', + }); + + // The onRunCompleted handler fetched the target's workflowName and + // returnValue off the lazily-hydrated Run instance, then resumed the + // observer's hook with what it saw. + const payload = await observer.returnValue; + expect(payload).toMatchObject({ + observed: 'completed', + runId: target.runId, + workflowName: expect.stringContaining('lifecycleHookTargetCompleted'), + returnedOutcome: 'completed', + }); + } + ); + + test( + 'onRunFailed receives the hydrated error with errorCode and cause', + { timeout: 90_000 }, + async () => { + const token = `lifecycle-failed-${Math.random().toString(36).slice(2)}`; + + const observer = await start(await e2e('lifecycleHookObserver'), [ + token, + ]); + await waitForHook(token, { runId: observer.runId }); + + const target = await start(await e2e('lifecycleHookTargetFailed'), [ + token, + ]); + const error = await target.returnValue.catch((e: unknown) => e); + expect(WorkflowRunFailedError.is(error)).toBe(true); + + // The onRunFailed handler received a WorkflowRunFailedError whose + // errorCode carries the classification and whose cause is the + // hydrated thrown FatalError (name + message preserved). + const payload = await observer.returnValue; + expect(payload).toMatchObject({ + observed: 'failed', + runId: target.runId, + errorCode: 'USER_ERROR', + causeName: 'FatalError', + causeMessage: expect.stringContaining( + `lifecycle-hook-target-failed:${token}` + ), + }); + } + ); + }); + test( 'hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep', { timeout: 90_000 }, diff --git a/packages/core/package.json b/packages/core/package.json index 964a51ddd4..dc50a89702 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -36,6 +36,10 @@ "types": "./dist/runtime/run.d.ts", "default": "./dist/runtime/run.js" }, + "./runtime/lifecycle-hooks": { + "types": "./dist/runtime/lifecycle-hooks.d.ts", + "default": "./dist/runtime/lifecycle-hooks.js" + }, "./runtime/start": { "types": "./dist/runtime/start.d.ts", "default": "./dist/runtime/start.js" diff --git a/packages/core/src/runtime.ts b/packages/core/src/runtime.ts index 11667984f4..bbbed9613c 100644 --- a/packages/core/src/runtime.ts +++ b/packages/core/src/runtime.ts @@ -87,6 +87,10 @@ import { stepDispatchIdempotencyKey, withHealthCheck, } from './runtime/helpers.js'; +import { + dispatchRunCompletedHooks, + dispatchRunFailedHooks, +} from './runtime/lifecycle-hooks.js'; import { handleReplayBudgetExhausted, ReplayBudget, @@ -429,6 +433,7 @@ async function recordFatalRunError({ } throw failErr; } + dispatchRunFailedHooks(runId, err, errorCode); } function hasRecordedTerminalRunEvent(events: Event[], runId: string): boolean { @@ -761,6 +766,11 @@ export function workflowEntrypoint( }, { requestId } ); + dispatchRunFailedHooks( + runId, + err, + RUN_ERROR_CODES.MAX_DELIVERIES_EXCEEDED + ); } catch (err) { if (EntityConflictError.is(err) || RunExpiredError.is(err)) { // Run already finished, consume the message silently @@ -3025,6 +3035,7 @@ export function workflowEntrypoint( } throw err; } + dispatchRunCompletedHooks(runId); span?.setAttributes({ ...Attribute.WorkflowRunStatus('completed'), @@ -3233,6 +3244,11 @@ export function workflowEntrypoint( } throw failErr; } + dispatchRunFailedHooks( + runId, + suspensionError, + errorCode + ); span?.setAttributes({ ...Attribute.WorkflowRunStatus('failed'), ...Attribute.WorkflowErrorCode(errorCode), @@ -4607,6 +4623,7 @@ export function workflowEntrypoint( } throw failErr; } + dispatchRunFailedHooks(runId, terminalError, errorCode); span?.setAttributes({ ...Attribute.WorkflowRunStatus('failed'), diff --git a/packages/core/src/runtime/deployment-guard.ts b/packages/core/src/runtime/deployment-guard.ts index 99f6c059df..de813d5486 100644 --- a/packages/core/src/runtime/deployment-guard.ts +++ b/packages/core/src/runtime/deployment-guard.ts @@ -14,6 +14,7 @@ import { runtimeLogger } from '../logger.js'; import { dehydrateRunError } from '../serialization.js'; import * as Attribute from '../telemetry/semantic-conventions.js'; import { getDeploymentMismatchMaxRetries } from './constants.js'; +import { dispatchRunFailedHooks } from './lifecycle-hooks.js'; /** Cap on the re-route backoff, in seconds. */ const MAX_REROUTE_DELAY_SECONDS = 8; @@ -212,6 +213,11 @@ export async function guardDeploymentAffinity({ }, { requestId } ); + dispatchRunFailedHooks( + run.runId, + error, + RUN_ERROR_CODES.DEPLOYMENT_MISMATCH + ); } catch (failError) { // Run already reached a terminal state (a concurrent writer failed it, or // it was cancelled/expired) — still stop. Anything else is a transient diff --git a/packages/core/src/runtime/lifecycle-hooks.test.ts b/packages/core/src/runtime/lifecycle-hooks.test.ts new file mode 100644 index 0000000000..21d0e73efa --- /dev/null +++ b/packages/core/src/runtime/lifecycle-hooks.test.ts @@ -0,0 +1,196 @@ +import { WorkflowRunFailedError } from '@workflow/errors'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { + dispatchRunCompletedHooks, + dispatchRunFailedHooks, + registerLifecycleHooks, + type WorkflowLifecycleHooks, +} from './lifecycle-hooks.js'; +import { Run } from './run.js'; + +vi.mock('../version.js', () => ({ version: '0.0.0-test' })); + +// Capture every promise handed to waitUntil so tests can await the +// fire-and-forget dispatch work deterministically. +const waitUntilPromises: Promise[] = []; +vi.mock('@vercel/functions', () => ({ + waitUntil: (promise: Promise) => { + waitUntilPromises.push(promise); + }, +})); + +/** Await everything the dispatcher scheduled through waitUntil. */ +async function flushDispatches(): Promise { + // The dispatcher resolves a dynamic import before handing the promise to + // waitUntil, so yield macrotask (check-phase) turns via setImmediate + // (which drains the intervening microtasks too) until the capture lands. + for (let i = 0; i < 10 && waitUntilPromises.length === 0; i++) { + await new Promise((resolve) => setImmediate(resolve)); + } + await Promise.all(waitUntilPromises); +} + +describe('lifecycle hooks', () => { + const unregisters: Array<() => void> = []; + + const register = (hooks: WorkflowLifecycleHooks) => { + const unregister = registerLifecycleHooks(hooks); + unregisters.push(unregister); + return unregister; + }; + + beforeEach(() => { + waitUntilPromises.length = 0; + }); + + afterEach(() => { + for (const unregister of unregisters) { + unregister(); + } + unregisters.length = 0; + }); + + it('invokes onRunCompleted with a lazily-hydrated Run instance', async () => { + const onRunCompleted = vi.fn(); + register({ onRunCompleted }); + + dispatchRunCompletedHooks('wrun_completed_1'); + await flushDispatches(); + + expect(onRunCompleted).toHaveBeenCalledTimes(1); + const { run } = onRunCompleted.mock.calls[0][0]; + expect(run).toBeInstanceOf(Run); + expect(run.runId).toBe('wrun_completed_1'); + }); + + it('invokes onRunFailed with the Run and a WorkflowRunFailedError carrying errorCode and cause', async () => { + const onRunFailed = vi.fn(); + register({ onRunFailed }); + + const cause = new Error('workflow exploded'); + dispatchRunFailedHooks('wrun_failed_1', cause, 'USER_ERROR'); + await flushDispatches(); + + expect(onRunFailed).toHaveBeenCalledTimes(1); + const { run, error } = onRunFailed.mock.calls[0][0]; + expect(run).toBeInstanceOf(Run); + expect(run.runId).toBe('wrun_failed_1'); + expect(WorkflowRunFailedError.is(error)).toBe(true); + expect(error.runId).toBe('wrun_failed_1'); + expect(error.errorCode).toBe('USER_ERROR'); + // The cause is the round-tripped (dehydrate → hydrate) value, not the + // original reference: handlers always see the host-realm hydrated shape. + expect(error.cause).toBeInstanceOf(Error); + expect((error.cause as Error).message).toBe('workflow exploded'); + expect(error.message).toContain('workflow exploded'); + }); + + it('hydrates a VM-realm thrown error into a host-realm Error for handlers', async () => { + const onRunFailed = vi.fn(); + register({ onRunFailed }); + + // Simulate a workflow-VM thrown error: a real native error from another + // realm, for which host `instanceof Error` is false. + const { runInNewContext } = await import('node:vm'); + const vmError = runInNewContext( + 'const e = new Error("vm exploded"); e.name = "FatalError"; e' + ); + expect(vmError instanceof Error).toBe(false); + + dispatchRunFailedHooks('wrun_vm_realm', vmError, 'USER_ERROR'); + await flushDispatches(); + + const { error } = onRunFailed.mock.calls[0][0]; + expect(error.cause).toBeInstanceOf(Error); + expect((error.cause as Error).name).toBe('FatalError'); + expect((error.cause as Error).message).toBe('vm exploded'); + }); + + it('does not schedule any work when no hooks are registered', async () => { + dispatchRunCompletedHooks('wrun_none'); + dispatchRunFailedHooks('wrun_none', new Error('x'), 'USER_ERROR'); + // Give a potential (buggy) schedule a chance to land. + await new Promise((resolve) => setImmediate(resolve)); + expect(waitUntilPromises).toHaveLength(0); + }); + + it('invokes multiple registrations in registration order', async () => { + const order: string[] = []; + register({ onRunCompleted: () => void order.push('first') }); + register({ + onRunCompleted: async () => { + // Async handler: the next handler must still wait for it. + await new Promise((resolve) => setTimeout(resolve, 5)); + order.push('second'); + }, + }); + register({ onRunCompleted: () => void order.push('third') }); + + dispatchRunCompletedHooks('wrun_order'); + await flushDispatches(); + + expect(order).toEqual(['first', 'second', 'third']); + }); + + it('swallows a throwing handler and still runs later handlers', async () => { + const later = vi.fn(); + register({ + onRunFailed: () => { + throw new Error('sync handler boom'); + }, + }); + register({ + onRunFailed: async () => { + throw new Error('async handler boom'); + }, + }); + register({ onRunFailed: later }); + + dispatchRunFailedHooks('wrun_boom', new Error('cause'), 'USER_ERROR'); + // Must not reject (safeWaitUntil relies on the promise never rejecting). + await expect(Promise.all(waitUntilPromises)).resolves.toBeDefined(); + await flushDispatches(); + + expect(later).toHaveBeenCalledTimes(1); + }); + + it('unregister removes the hooks', async () => { + const onRunCompleted = vi.fn(); + const unregister = registerLifecycleHooks({ onRunCompleted }); + unregister(); + + dispatchRunCompletedHooks('wrun_unregistered'); + await new Promise((resolve) => setImmediate(resolve)); + + expect(onRunCompleted).not.toHaveBeenCalled(); + expect(waitUntilPromises).toHaveLength(0); + }); + + it('shares one registry across module copies via the Symbol.for global', async () => { + const onRunCompleted = vi.fn(); + register({ onRunCompleted }); + + const registry = (globalThis as Record)[ + Symbol.for('@workflow/core//lifecycleHooks') + ] as WorkflowLifecycleHooks[]; + expect(Array.isArray(registry)).toBe(true); + expect(registry.some((h) => h.onRunCompleted === onRunCompleted)).toBe( + true + ); + }); + + it('non-Error thrown values round-trip through WorkflowRunFailedError.cause', async () => { + const onRunFailed = vi.fn(); + register({ onRunFailed }); + + const thrown = { kind: 'business-rule-violation', code: 'LOCKED' }; + dispatchRunFailedHooks('wrun_nonerror', thrown, 'USER_ERROR'); + await flushDispatches(); + + const { error } = onRunFailed.mock.calls[0][0]; + // Structural clone via the serialization round-trip, not coerced to an + // Error. + expect(error.cause).not.toBeInstanceOf(Error); + expect(error.cause).toEqual(thrown); + }); +}); diff --git a/packages/core/src/runtime/lifecycle-hooks.ts b/packages/core/src/runtime/lifecycle-hooks.ts new file mode 100644 index 0000000000..8ac94d66be --- /dev/null +++ b/packages/core/src/runtime/lifecycle-hooks.ts @@ -0,0 +1,218 @@ +import { WorkflowRunFailedError } from '@workflow/errors'; +import { runtimeLogger } from '../logger.js'; +import { dehydrateRunError, hydrateRunError } from '../serialization.js'; +import { Run } from './run.js'; +import { safeWaitUntil } from './wait-until.js'; + +/** + * Parameters passed to an {@link WorkflowLifecycleHooks.onRunCompleted} + * handler. + */ +export interface RunCompletedHookParams { + /** + * The completed run. The instance hydrates lazily, so reading + * `run.returnValue` (or any other accessor) fetches from the backend only + * when the handler actually uses it. + */ + run: Run; +} + +/** + * Parameters passed to an {@link WorkflowLifecycleHooks.onRunFailed} + * handler. + */ +export interface RunFailedHookParams { + /** + * The failed run. The instance hydrates lazily, so accessors fetch from + * the backend only when the handler actually uses them. + */ + run: Run; + /** + * The failure, in the same shape `run.returnValue` rejects with: a + * `WorkflowRunFailedError` whose `errorCode` carries the failure + * classification (e.g. `USER_ERROR`, `RUNTIME_ERROR`) and whose `cause` is + * the hydrated thrown value (original Error subclass identity preserved). + */ + error: WorkflowRunFailedError; +} + +/** + * Global handlers observing workflow run lifecycle transitions. Register via + * {@link registerLifecycleHooks}. + */ +export interface WorkflowLifecycleHooks { + /** Invoked when a workflow run completes successfully. */ + onRunCompleted?: (params: RunCompletedHookParams) => void | Promise; + /** Invoked when a workflow run fails terminally (after any retries). */ + onRunFailed?: (params: RunFailedHookParams) => void | Promise; +} + +/** + * The registry lives on `globalThis` under a `Symbol.for` key so that every + * copy of `@workflow/core` in the process (bundled + unbundled, ESM + CJS) + * shares one list, the same pattern as the cross-realm error-class registry in + * `@workflow/errors` and the World cache in `get-world-lazy.ts`. The property + * is non-writable/non-configurable so accidental clobbering is loud; the + * array's contents stay mutable for register/unregister. + */ +const REGISTRY_KEY = Symbol.for('@workflow/core//lifecycleHooks'); + +function getRegistry(): WorkflowLifecycleHooks[] { + if (!Object.hasOwn(globalThis, REGISTRY_KEY)) { + Object.defineProperty(globalThis, REGISTRY_KEY, { + value: [], + writable: false, + enumerable: false, + configurable: false, + }); + } + return (globalThis as Record)[ + REGISTRY_KEY + ] as WorkflowLifecycleHooks[]; +} + +/** + * Registers global workflow lifecycle handlers, invoked by the runtime on + * the compute that records a run's terminal transition. Useful for + * centralized reporting (e.g. forwarding failed runs to Sentry) without + * wrapping every workflow body. + * + * Register early in the process lifecycle so handlers exist before the first + * run finishes: in Next.js, `instrumentation.ts` is the natural place; in any + * other app, any module that loads at startup works. + * + * Semantics: + * - Handlers run on the host (full Node.js), never inside the workflow VM. + * - Handlers fire only on the invocation that actually wrote the terminal + * event. Transitions recorded elsewhere (e.g. a run cancelled from the + * CLI or dashboard) do not fire handlers in the app. + * - Handlers are fire-and-forget: they cannot delay or change the run's + * outcome, and a throwing handler is logged and swallowed. On serverless + * platforms the invocation is kept alive via `waitUntil`. + * - Multiple registrations are allowed; handlers run in registration order. + * + * @returns A function that unregisters these hooks. + */ +export function registerLifecycleHooks( + hooks: WorkflowLifecycleHooks +): () => void { + const registry = getRegistry(); + registry.push(hooks); + return () => { + const index = registry.indexOf(hooks); + if (index !== -1) { + registry.splice(index, 1); + } + }; +} + +/** + * Runs every registered handler for one lifecycle transition without ever + * throwing into (or blocking) the runtime's terminal-write path: the work is + * scheduled through `safeWaitUntil`, the params are prepared at most once + * per transition, each handler's failure is logged and swallowed + * individually, and handlers run sequentially in registration order. + */ +function dispatch( + runId: string, + event: 'onRunCompleted' | 'onRunFailed', + prepare: () => Promise, + invoke: ( + hooks: WorkflowLifecycleHooks, + params: TParams + ) => void | Promise | undefined +): void { + // Snapshot so an unregister inside a handler cannot skew iteration. + const registered = [...getRegistry()]; + if (registered.length === 0) { + return; + } + safeWaitUntil( + (async () => { + const params = await prepare(); + for (const hooks of registered) { + try { + await invoke(hooks, params); + } catch (err) { + runtimeLogger.error(`Workflow lifecycle ${event} handler threw`, { + workflowRunId: runId, + error: err instanceof Error ? err.message : String(err), + }); + } + } + })(), + // Covers a `prepare()` rejection; handler failures are caught above. + (err) => { + runtimeLogger.error(`Workflow lifecycle ${event} dispatch failed`, { + workflowRunId: runId, + error: err instanceof Error ? err.message : String(err), + }); + } + ); +} + +/** + * Called by the runtime after it successfully wrote a `run_completed` event. + * Never throws. + */ +export function dispatchRunCompletedHooks(runId: string): void { + dispatch( + runId, + 'onRunCompleted', + async () => ({ run: new Run(runId) }), + (hooks, params) => hooks.onRunCompleted?.(params) + ); +} + +/** + * The thrown value a `run_failed` writer holds is often a VM-realm object + * (the workflow runs in a separate realm, so `instanceof Error` on it is + * `false` for handlers) and may carry VM-realm exotics in its cause chain. + * Round-trip it through the run-error serialization pipeline so handlers + * receive the same host-realm hydrated shape `run.returnValue` rejects + * with: real host Error instances with name/message/stack/cause preserved + * and registered classes (FatalError, custom serde classes) revived with + * their class identity. No encryption: the bytes never leave this process. + * + * Falls back to the original value when the round-trip fails, since a + * degraded report beats no report. + */ +async function hydrateForHandlers( + error: unknown, + runId: string +): Promise { + try { + const bytes = await dehydrateRunError(error, runId, undefined); + return await hydrateRunError(bytes, runId, undefined); + } catch { + return error; + } +} + +/** + * Called by the runtime after it successfully wrote a `run_failed` event. + * Never throws. + * + * @param error - The thrown value the terminal write recorded (host-side + * object where available; QuickJS passes its rehydrated reconstruction). + * @param errorCode - The classification written to the event's `errorCode`. + */ +export function dispatchRunFailedHooks( + runId: string, + error: unknown, + errorCode: string +): void { + dispatch( + runId, + 'onRunFailed', + async () => ({ + run: new Run(runId), + error: new WorkflowRunFailedError( + runId, + await hydrateForHandlers(error, runId), + { errorCode } + ), + }), + (hooks, params) => hooks.onRunFailed?.(params) + ); +} diff --git a/packages/core/src/runtime/quickjs-entrypoint.ts b/packages/core/src/runtime/quickjs-entrypoint.ts index 74d0300cb6..b911efcc43 100644 --- a/packages/core/src/runtime/quickjs-entrypoint.ts +++ b/packages/core/src/runtime/quickjs-entrypoint.ts @@ -60,6 +60,10 @@ import { queueMessage, stepDispatchIdempotencyKey, } from './helpers.js'; +import { + dispatchRunCompletedHooks, + dispatchRunFailedHooks, +} from './lifecycle-hooks.js'; import { BASELINE_BUNDLE_FILENAME, type PendingAttribute, @@ -1732,6 +1736,7 @@ export async function runWorkflowWithQuickJS(params: { }, }); wfdiag('exit_completed', { result: 'run_completed_written' }); + dispatchRunCompletedHooks(runId); } catch (err) { if (EntityConflictError.is(err) || RunExpiredError.is(err)) { runtimeLogger.warn( @@ -1970,6 +1975,11 @@ export async function runWorkflowWithQuickJS(params: { // `dehydrateRunError`. Used when valueBytes is absent (e.g. // extractError pseudo-failures from VM bootstrap). let dehydratedError: Uint8Array; + // The most faithful host-side error value available, handed to the + // lifecycle onRunFailed hooks after the terminal write lands: the + // hydrated VM value when the modern path succeeds, otherwise the + // reconstructed host Error. + let lifecycleError: unknown = reconstructed; if (result.failed.valueBytes) { // Hydrate the VM-side bytes, remap the error stack with the // host-side source map (the VM can't do this — it lacks both the @@ -2024,6 +2034,7 @@ export async function runWorkflowWithQuickJS(params: { runId, encryptionKey ); + lifecycleError = hydrated; } catch (rehydrateErr) { // If hydration / re-dehydration fails for any reason, fall // back to passing through the original VM bytes (just apply @@ -2090,6 +2101,7 @@ export async function runWorkflowWithQuickJS(params: { }); throw err; } + dispatchRunFailedHooks(runId, lifecycleError, errorCode); wfdiag('exit_failed', { result: 'run_failed_written' }); } } diff --git a/packages/core/src/runtime/replay-budget.test.ts b/packages/core/src/runtime/replay-budget.test.ts index ee0acf44f3..4bc33fa9a5 100644 --- a/packages/core/src/runtime/replay-budget.test.ts +++ b/packages/core/src/runtime/replay-budget.test.ts @@ -1,6 +1,7 @@ import type { World } from '@workflow/world'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { runtimeLogger } from '../logger.js'; +import { registerLifecycleHooks } from './lifecycle-hooks.js'; import { handleReplayBudgetExhausted, ReplayBudget, @@ -12,7 +13,11 @@ vi.mock('./world.js', () => ({ getWorld: vi.fn(), })); -vi.mock('../serialization.js', () => ({ +// Partial mock: the lifecycle-hook registry pulls in `run.ts` (for the Run +// instance handed to handlers), whose import chain needs the real module's +// other exports (e.g. SerializationFormat). +vi.mock(import('../serialization.js'), async (importOriginal) => ({ + ...(await importOriginal()), dehydrateRunError: vi.fn(async () => new Uint8Array([1, 2, 3])), })); @@ -20,6 +25,27 @@ vi.mock('./helpers.js', () => ({ memoizeEncryptionKey: () => async () => undefined, })); +// Capture lifecycle-hook dispatch work (scheduled via waitUntil) so tests +// can await it deterministically. +const waitUntilPromises: Promise[] = []; +vi.mock('@vercel/functions', () => ({ + waitUntil: (promise: Promise) => { + waitUntilPromises.push(promise); + }, +})); + +/** + * Await everything the lifecycle dispatcher scheduled through waitUntil. + * The dispatcher resolves a dynamic import before handing the promise to + * waitUntil, so yield to the macrotask queue until the capture lands. + */ +async function flushLifecycleDispatches(): Promise { + for (let i = 0; i < 10 && waitUntilPromises.length === 0; i++) { + await new Promise((resolve) => setImmediate(resolve)); + } + await Promise.all(waitUntilPromises); +} + describe('ReplayBudget', () => { beforeEach(() => { vi.useFakeTimers(); @@ -238,4 +264,49 @@ describe('handleReplayBudgetExhausted', () => { expect(exitSpy).not.toHaveBeenCalled(); }); + + it('fires onRunFailed lifecycle hooks after the terminal write lands, and not on write failure', async () => { + const onRunFailed = vi.fn(); + const unregister = registerLifecycleHooks({ onRunFailed }); + try { + // Write failure: no dispatch. + mockEventsCreate.mockRejectedValueOnce(new Error('storage unavailable')); + vi.mocked(getWorld).mockResolvedValue(makeMockWorld()); + await expect( + handleReplayBudgetExhausted({ + runId: 'wrun_test', + workflowName: 'wf', + requestId: 'req_test', + attempt: 4, + limitMs: 240_000, + }) + ).rejects.toThrow('storage unavailable'); + // Give a (buggy) schedule a chance to land before asserting none did. + await new Promise((resolve) => setImmediate(resolve)); + expect(waitUntilPromises).toHaveLength(0); + expect(onRunFailed).not.toHaveBeenCalled(); + + // Successful write: dispatch with the Run and classified error. + await handleReplayBudgetExhausted({ + runId: 'wrun_test', + workflowName: 'wf', + requestId: 'req_test', + attempt: 4, + limitMs: 240_000, + }); + await flushLifecycleDispatches(); + + expect(onRunFailed).toHaveBeenCalledTimes(1); + const { run, error } = onRunFailed.mock.calls[0][0]; + expect(run.runId).toBe('wrun_test'); + expect(error.errorCode).toBe('REPLAY_TIMEOUT'); + expect(error.cause).toBeInstanceOf(Error); + expect((error.cause as Error).message).toContain( + 'exceeded maximum duration' + ); + } finally { + unregister(); + waitUntilPromises.length = 0; + } + }); }); diff --git a/packages/core/src/runtime/replay-budget.ts b/packages/core/src/runtime/replay-budget.ts index 53f70f3b1f..60c58e9222 100644 --- a/packages/core/src/runtime/replay-budget.ts +++ b/packages/core/src/runtime/replay-budget.ts @@ -5,6 +5,7 @@ import { runtimeLogger } from '../logger.js'; import { dehydrateRunError } from '../serialization.js'; import { getReplayTimeoutMaxRetries, getReplayTimeoutMs } from './constants.js'; import { memoizeEncryptionKey, type SlotSnapshotParams } from './helpers.js'; +import { dispatchRunFailedHooks } from './lifecycle-hooks.js'; import { getWorld } from './world.js'; /** @@ -180,4 +181,5 @@ export async function handleReplayBudgetExhausted(args: { }, { requestId, ...slotSnapshot } ); + dispatchRunFailedHooks(runId, timeoutErr, RUN_ERROR_CODES.REPLAY_TIMEOUT); } diff --git a/packages/workflow/src/api-workflow.ts b/packages/workflow/src/api-workflow.ts index a489fd929d..a1eec8a7b0 100644 --- a/packages/workflow/src/api-workflow.ts +++ b/packages/workflow/src/api-workflow.ts @@ -22,3 +22,10 @@ export const getHookByToken = () => workflowStub('getHookByToken'); export const resumeHook = () => workflowStub('resumeHook'); export const resumeWebhook = () => workflowStub('resumeWebhook'); export const runStep = () => workflowStub('runStep'); +export const registerLifecycleHooks = () => + workflowStub('registerLifecycleHooks'); +export type { + RunCompletedHookParams, + RunFailedHookParams, + WorkflowLifecycleHooks, +} from '@workflow/core/runtime/lifecycle-hooks'; diff --git a/packages/workflow/src/api.ts b/packages/workflow/src/api.ts index 31ca6ed146..12ab004256 100644 --- a/packages/workflow/src/api.ts +++ b/packages/workflow/src/api.ts @@ -14,6 +14,12 @@ export type { StopSleepResult, WorkflowRun, } from '@workflow/core/runtime'; +export { + type RunCompletedHookParams, + type RunFailedHookParams, + registerLifecycleHooks, + type WorkflowLifecycleHooks, +} from '@workflow/core/runtime/lifecycle-hooks'; export { getHookByToken, type ResumedHook, diff --git a/workbench/example/workflows/99_e2e.ts b/workbench/example/workflows/99_e2e.ts index 587c200215..22ca31b1e2 100644 --- a/workbench/example/workflows/99_e2e.ts +++ b/workbench/example/workflows/99_e2e.ts @@ -3894,3 +3894,44 @@ export async function crossRegionStreamWorkflow(chunkCount: number) { await closeCrossRegionStream(writable); return 'done'; } + +// ============================================================ +// LIFECYCLE HOOK TESTS +// Exercised only by the Next.js workbenches, whose +// instrumentation.ts registers `registerLifecycleHooks` handlers +// (see workbench/nextjs-*/instrumentation.ts). The handlers +// observe these target runs' terminal transitions and report +// them by resuming the observer workflow's hook, a durable +// channel that works across serverless instances. +// ============================================================ + +/** + * Target: completes immediately. The `onRunCompleted` handler reads this + * run's return value (exercising the Run instance's lazy hydration) to + * discover the observer's hook token. + */ +export async function lifecycleHookTargetCompleted(token: string) { + 'use workflow'; + return { token, outcome: 'completed' }; +} + +/** + * Target: fails immediately. The token is embedded in the thrown error's + * message so the `onRunFailed` handler can find the observer without any + * backend reads (the hydrated cause is on the WorkflowRunFailedError it + * receives). + */ +export async function lifecycleHookTargetFailed(token: string) { + 'use workflow'; + throw new FatalError(`lifecycle-hook-target-failed:${token}`); +} + +/** + * Observer: parks on a hook until a lifecycle handler reports the target + * run's terminal transition, then returns the reported payload verbatim. + */ +export async function lifecycleHookObserver(token: string) { + 'use workflow'; + using hook = createHook>({ token }); + return await hook; +} diff --git a/workbench/nextjs-turbopack/instrumentation.ts b/workbench/nextjs-turbopack/instrumentation.ts index bd66985a97..9bfa71bee3 100644 --- a/workbench/nextjs-turbopack/instrumentation.ts +++ b/workbench/nextjs-turbopack/instrumentation.ts @@ -1,6 +1,16 @@ import { registerOTel } from '@vercel/otel'; -export function register() { +export async function register() { + if (process.env.NEXT_RUNTIME === 'nodejs') { + // Workflow lifecycle hooks are host-only. The import MUST be dynamic + // and inside the runtime guard (the canonical Next.js pattern for + // node-only instrumentation): a static top-level import would pull + // `workflow/api` → world-init → @workflow/world-local → fs into every + // compile target of instrumentation.ts, and the non-node ones cannot + // resolve `fs` (breaks the webpack workbench's build). + const { registerE2eLifecycleHooks } = await import('./lifecycle-hooks-e2e'); + registerE2eLifecycleHooks(); + } registerOTel({ serviceName: 'nextjs-turbopack', instrumentationConfig: { diff --git a/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts b/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts new file mode 100644 index 0000000000..e0cdfa4511 --- /dev/null +++ b/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts @@ -0,0 +1,55 @@ +import { registerLifecycleHooks, resumeHook } from 'workflow/api'; + +/** + * E2E coverage for `registerLifecycleHooks` (see the "lifecycle hooks" + * describe block in packages/core/e2e/e2e.test.ts and the fixtures in + * workflows/99_e2e.ts). + * + * The handlers observe the `lifecycleHookTarget*` workflows' terminal + * transitions and report them by resuming the `lifecycleHookObserver` + * workflow's hook. Resuming a durable hook is deliberately the observation + * channel: the handler runs on whichever instance wrote the terminal event, + * which on a deployed app is generally NOT the instance serving the e2e + * test's HTTP requests, so an in-memory buffer would not travel. + */ +export function registerE2eLifecycleHooks(): void { + registerLifecycleHooks({ + async onRunCompleted({ run }) { + // Fires for every completed run in the app, so filter cheaply by + // workflow name (a metadata read) before touching the return value. + const workflowName = await run.workflowName; + if (!workflowName?.includes('lifecycleHookTargetCompleted')) { + return; + } + // Lazy hydration: the return value is only fetched for matching runs. + const returnValue = (await run.returnValue) as { + token: string; + outcome: string; + }; + await resumeHook(returnValue.token, { + observed: 'completed', + runId: run.runId, + workflowName, + returnedOutcome: returnValue.outcome, + }); + }, + async onRunFailed({ run, error }) { + // The hydrated thrown value is already on the error, so filtering + // needs no backend reads. + const cause = error.cause; + const causeMessage = + cause instanceof Error ? cause.message : String(cause); + const match = causeMessage.match(/lifecycle-hook-target-failed:(\S+)/); + if (!match) { + return; + } + await resumeHook(match[1], { + observed: 'failed', + runId: run.runId, + errorCode: error.errorCode, + causeName: cause instanceof Error ? cause.name : typeof cause, + causeMessage, + }); + }, + }); +} diff --git a/workbench/nextjs-webpack/instrumentation.ts b/workbench/nextjs-webpack/instrumentation.ts index 007ff5f971..e41114618d 100644 --- a/workbench/nextjs-webpack/instrumentation.ts +++ b/workbench/nextjs-webpack/instrumentation.ts @@ -1,6 +1,16 @@ import { registerOTel } from '@vercel/otel'; -export function register() { +export async function register() { + if (process.env.NEXT_RUNTIME === 'nodejs') { + // Workflow lifecycle hooks are host-only. The import MUST be dynamic + // and inside the runtime guard (the canonical Next.js pattern for + // node-only instrumentation): a static top-level import would pull + // `workflow/api` → world-init → @workflow/world-local → fs into every + // compile target of instrumentation.ts, and the non-node ones cannot + // resolve `fs` (breaks this workbench's webpack build). + const { registerE2eLifecycleHooks } = await import('./lifecycle-hooks-e2e'); + registerE2eLifecycleHooks(); + } registerOTel({ serviceName: 'nextjs-webpack', instrumentationConfig: { diff --git a/workbench/nextjs-webpack/lifecycle-hooks-e2e.ts b/workbench/nextjs-webpack/lifecycle-hooks-e2e.ts new file mode 120000 index 0000000000..ed7314a288 --- /dev/null +++ b/workbench/nextjs-webpack/lifecycle-hooks-e2e.ts @@ -0,0 +1 @@ +../nextjs-turbopack/lifecycle-hooks-e2e.ts \ No newline at end of file