Merge branch 'main' into feat/sync-env-secrets

Author: matt-aitken

.changeset/angry-trainers-perform.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Truncate large error stacks and messages to prevent OOM crashes. Stack traces are capped at 50 frames (keeping top 5 + bottom 45 with an omission notice), individual stack lines at 1024 chars, and error messages at 1000 chars. Applied in parseError, sanitizeError, and OTel span recording.

.changeset/kind-kids-teach.md

@@ -0,0 +1,13 @@
+---
+paths:
+  - "internal-packages/database/**"
+---
+
+# Database Migration Safety
+
+- When adding indexes to **existing tables**, use `CREATE INDEX CONCURRENTLY IF NOT EXISTS` to avoid table locks. These must be in their own separate migration file (one index per file).
+- Indexes on **newly created tables** (same migration as `CREATE TABLE`) do not need CONCURRENTLY.
+- When indexing a **new column on an existing table**, split into two migrations: first `ADD COLUMN IF NOT EXISTS`, then `CREATE INDEX CONCURRENTLY IF NOT EXISTS` in a separate file.
+- After generating a migration with Prisma, remove extraneous lines for: `_BackgroundWorkerToBackgroundWorkerFile`, `_BackgroundWorkerToTaskQueue`, `_TaskRunToTaskRunTag`, `_WaitpointRunConnections`, `_completedWaitpoints`, `SecretStore_key_idx`, and unrelated TaskRun indexes.
+- Never drop columns or tables without explicit approval.
+- New code should target `RunEngineVersion.V2` only.

.changeset/truncate-error-stacks.md

@@ -0,0 +1,14 @@
+---
+paths:
+  - "docs/**"
+---
+
+# Documentation Writing Rules
+
+- Use Mintlify MDX format. Frontmatter: `title`, `description`, `sidebarTitle` (optional).
+- After creating a new page, add it to `docs.json` navigation under the correct group.
+- Use Mintlify components: `<Note>`, `<Warning>`, `<Info>`, `<Tip>`, `<CodeGroup>`, `<Expandable>`, `<Steps>`/`<Step>`.
+- Code examples should be complete and runnable where possible.
+- Always import from `@trigger.dev/sdk`, never `@trigger.dev/sdk/v3`.
+- Keep paragraphs short. Use headers to break up content.
+- Link to related pages using relative paths (e.g., `[Tasks](/tasks/overview)`).

.claude/rules/database-safety.md

@@ -0,0 +1,33 @@
+---
+paths:
+  - "apps/webapp/app/v3/**"
+---
+
+# Legacy V1 Engine Code in `app/v3/`
+
+The `v3/` directory name is misleading - most code here is actively used by the current V2 engine. Only the specific files below are legacy V1-only code.
+
+## V1-Only Files - Never Modify
+
+- `marqs/` directory (entire MarQS queue system: sharedQueueConsumer, devQueueConsumer, fairDequeuingStrategy, devPubSub)
+- `legacyRunEngineWorker.server.ts` (V1 background job worker)
+- `services/triggerTaskV1.server.ts` (deprecated V1 task triggering)
+- `services/cancelTaskRunV1.server.ts` (deprecated V1 cancellation)
+- `authenticatedSocketConnection.server.ts` (V1 dev WebSocket using DevQueueConsumer)
+- `sharedSocketConnection.ts` (V1 shared queue socket using SharedQueueConsumer)
+
+## V1/V2 Branching Pattern
+
+Some services act as routers that branch on `RunEngineVersion`:
+- `services/cancelTaskRun.server.ts` - calls V1 service or `engine.cancelRun()` for V2
+- `services/batchTriggerV3.server.ts` - uses marqs for V1 path, run-engine for V2
+
+When editing these shared services, only modify V2 code paths.
+
+## V2 Modern Stack
+
+- **Run lifecycle**: `@internal/run-engine` (internal-packages/run-engine)
+- **Background jobs**: `@trigger.dev/redis-worker` (not graphile-worker/zodworker)
+- **Queue operations**: RunQueue inside run-engine (not MarQS)
+- **V2 engine singleton**: `runEngine.server.ts`, `runEngineHandlers.server.ts`
+- **V2 workers**: `commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`

.claude/rules/docs-writing.md

@@ -0,0 +1,12 @@
+---
+paths:
+  - "packages/**"
+---
+
+# Public Package Rules
+
+- Changes to `packages/` are **customer-facing**. Always add a changeset: `pnpm run changeset:add`
+- Default to **patch**. Get maintainer approval for minor. Never select major without explicit approval.
+- `@trigger.dev/core`: **Never import the root**. Always use subpath imports (e.g., `@trigger.dev/core/v3`).
+- Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked. These are maintained in separate dedicated passes.
+- Test changes using `references/hello-world` reference project.

.claude/rules/legacy-v3-code.md

@@ -0,0 +1,23 @@
+---
+paths:
+  - "apps/**"
+---
+
+# Server App Changes
+
+When modifying server apps (webapp, supervisor, coordinator, etc.) with **no package changes**, add a `.server-changes/` file instead of a changeset:
+
+```bash
+cat > .server-changes/descriptive-name.md << 'EOF'
+---
+area: webapp
+type: fix
+---
+
+Brief description of what changed and why.
+EOF
+```
+
+- **area**: `webapp` | `supervisor` | `coordinator` | `kubernetes-provider` | `docker-provider`
+- **type**: `feature` | `fix` | `improvement` | `breaking`
+- If the PR also touches `packages/`, just the changeset is sufficient (no `.server-changes/` needed).

.claude/rules/sdk-packages.md

@@ -0,0 +1,78 @@
+---
+name: span-timeline-events
+description: Use when adding, modifying, or debugging OTel span timeline events in the trace view. Covers event structure, ClickHouse storage constraints, rendering in SpanTimeline component, admin visibility, and the step-by-step process for adding new events.
+allowed-tools: Read, Write, Edit, Glob, Grep, Bash
+---
+
+# Span Timeline Events
+
+The trace view's right panel shows a timeline of events for the selected span. These are OTel span events rendered by `app/utils/timelineSpanEvents.ts` and the `SpanTimeline` component.
+
+## How They Work
+
+1. **Span events** in OTel are attached to a parent span. In ClickHouse, they're stored as separate rows with `kind: "SPAN_EVENT"` sharing the parent span's `span_id`. The `#mergeRecordsIntoSpanDetail` method reassembles them into the span's `events` array at query time.
+2. The timeline only renders events whose `name` starts with `trigger.dev/` - all others are silently filtered out.
+3. The **display name** comes from `properties.event` (not the span event name), mapped through `getFriendlyNameForEvent()`.
+4. Events are shown on the **span they belong to** - events on one span don't appear in another span's timeline.
+
+## ClickHouse Storage Constraint
+
+When events are written to ClickHouse, `spanEventsToTaskEventV1Input()` filters out events whose `start_time` is not greater than the parent span's `startTime`. Events at or before the span start are silently dropped. This means span events must have timestamps strictly after the span's own `startTimeUnixNano`.
+
+## Timeline Rendering (SpanTimeline component)
+
+The `SpanTimeline` component in `app/components/run/RunTimeline.tsx` renders:
+
+1. **Events** (thin 1px line with hollow dots) - all events from `createTimelineSpanEventsFromSpanEvents()`
+2. **"Started"** marker (thick cap) - at the span's `startTime`
+3. **Duration bar** (thick 7px line) - from "Started" to "Finished"
+4. **"Finished"** marker (thick cap) - at `startTime + duration`
+
+The thin line before "Started" only appears when there are events with timestamps between the span start and the first child span. For the Attempt span this works well (Dequeued -> Pod scheduled -> Launched -> etc. all happen before execution starts). Events all get `lineVariant: "light"` (thin) while the execution bar gets `variant: "normal"` (thick).
+
+## Trace View Sort Order
+
+Sibling spans (same parent) are sorted by `start_time ASC` from the ClickHouse query. The `createTreeFromFlatItems` function preserves this order. Event timestamps don't affect sort order - only the span's own `start_time`.
+
+## Event Structure
+
+```typescript
+// OTel span event format
+{
+  name: "trigger.dev/run",        // Must start with "trigger.dev/" to render
+  timeUnixNano: "1711200000000000000",
+  attributes: [
+    { key: "event", value: { stringValue: "dequeue" } },  // The actual event type
+    { key: "duration", value: { intValue: 150 } },         // Optional: duration in ms
+  ]
+}
+```
+
+## Admin-Only Events
+
+`getAdminOnlyForEvent()` controls visibility. Events default to **admin-only** (`true`).
+
+| Event | Admin-only | Friendly name |
+|-------|-----------|---------------|
+| `dequeue` | No | Dequeued |
+| `fork` | No | Launched |
+| `import` | No (if no fork event) | Importing task file |
+| `create_attempt` | Yes | Attempt created |
+| `lazy_payload` | Yes | Lazy attempt initialized |
+| `pod_scheduled` | Yes | Pod scheduled |
+| (default) | Yes | (raw event name) |
+
+## Adding New Timeline Events
+
+1. Add OTLP span event with `name: "trigger.dev/<scope>"` and `properties.event: "<type>"`
+2. Event timestamp must be strictly after the parent span's `startTimeUnixNano` (ClickHouse drops earlier events)
+3. Add friendly name in `getFriendlyNameForEvent()` in `app/utils/timelineSpanEvents.ts`
+4. Set admin visibility in `getAdminOnlyForEvent()`
+5. Optionally add help text in `getHelpTextForEvent()`
+
+## Key Files
+
+- `app/utils/timelineSpanEvents.ts` - filtering, naming, admin logic
+- `app/components/run/RunTimeline.tsx` - `SpanTimeline` component (thin line + thick bar rendering)
+- `app/presenters/v3/SpanPresenter.server.ts` - loads span data including events
+- `app/v3/eventRepository/clickhouseEventRepository.server.ts` - `spanEventsToTaskEventV1Input()` (storage filter), `#mergeRecordsIntoSpanDetail` (reassembly)

.claude/rules/server-apps.md

@@ -0,0 +1,200 @@
+---
+name: trigger-dev-tasks
+description: Use this skill when writing, designing, or optimizing Trigger.dev background tasks and workflows. This includes creating reliable async tasks, implementing AI workflows, setting up scheduled jobs, structuring complex task hierarchies with subtasks, configuring build extensions for tools like ffmpeg or Puppeteer/Playwright, and handling task schemas with Zod validation.
+allowed-tools: Read, Write, Edit, Glob, Grep, Bash
+---
+
+# Trigger.dev Task Expert
+
+You are an expert Trigger.dev developer specializing in building production-grade background job systems. Tasks deployed to Trigger.dev run in Node.js 21+ and use the `@trigger.dev/sdk` package.
+
+## Critical Rules
+
+1. **Always use `@trigger.dev/sdk`** - Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob` pattern
+2. **Never use `node-fetch`** - Use the built-in `fetch` function
+3. **Export all tasks** - Every task must be exported, including subtasks
+4. **Never wrap wait/trigger calls in Promise.all** - `triggerAndWait`, `batchTriggerAndWait`, and `wait.*` calls cannot be wrapped in `Promise.all` or `Promise.allSettled`
+
+## Basic Task Pattern
+
+```ts
+import { task } from "@trigger.dev/sdk";
+
+export const processData = task({
+  id: "process-data",
+  retry: {
+    maxAttempts: 10,
+    factor: 1.8,
+    minTimeoutInMs: 500,
+    maxTimeoutInMs: 30_000,
+  },
+  run: async (payload: { userId: string; data: any[] }) => {
+    console.log(`Processing ${payload.data.length} items`);
+    return { processed: payload.data.length };
+  },
+});
+```
+
+## Schema Task (with validation)
+
+```ts
+import { schemaTask } from "@trigger.dev/sdk";
+import { z } from "zod";
+
+export const validatedTask = schemaTask({
+  id: "validated-task",
+  schema: z.object({
+    name: z.string(),
+    email: z.string().email(),
+  }),
+  run: async (payload) => {
+    // Payload is automatically validated and typed
+    return { message: `Hello ${payload.name}` };
+  },
+});
+```
+
+## Triggering Tasks
+
+### From Backend Code (type-only import to prevent dependency leakage)
+
+```ts
+import { tasks } from "@trigger.dev/sdk";
+import type { processData } from "./trigger/tasks";
+
+const handle = await tasks.trigger<typeof processData>("process-data", {
+  userId: "123",
+  data: [{ id: 1 }],
+});
+```
+
+### From Inside Tasks
+
+```ts
+export const parentTask = task({
+  id: "parent-task",
+  run: async (payload) => {
+    // Trigger and wait - returns Result object, NOT direct output
+    const result = await childTask.triggerAndWait({ data: "value" });
+    if (result.ok) {
+      console.log("Output:", result.output);
+    } else {
+      console.error("Failed:", result.error);
+    }
+
+    // Or unwrap directly (throws on error)
+    const output = await childTask.triggerAndWait({ data: "value" }).unwrap();
+  },
+});
+```
+
+## Idempotency (Critical for Retries)
+
+Always use idempotency keys when triggering tasks from inside other tasks:
+
+```ts
+import { idempotencyKeys } from "@trigger.dev/sdk";
+
+export const paymentTask = task({
+  id: "process-payment",
+  run: async (payload: { orderId: string }) => {
+    // Scoped to current run - survives retries
+    const key = await idempotencyKeys.create(`payment-${payload.orderId}`);
+
+    await chargeCustomer.trigger(payload, {
+      idempotencyKey: key,
+      idempotencyKeyTTL: "24h",
+    });
+  },
+});
+```
+
+## Trigger Options
+
+```ts
+await myTask.trigger(payload, {
+  delay: "1h",           // Delay execution
+  ttl: "10m",            // Cancel if not started within TTL
+  idempotencyKey: key,
+  queue: "my-queue",
+  machine: "large-1x",   // micro, small-1x, small-2x, medium-1x, medium-2x, large-1x, large-2x
+  maxAttempts: 3,
+  tags: ["user_123"],    // Max 10 tags
+  debounce: {            // Consolidate rapid triggers
+    key: "unique-key",
+    delay: "5s",
+    mode: "trailing",    // "leading" (default) or "trailing"
+  },
+});
+```
+
+## Debouncing
+
+Consolidate multiple triggers into a single execution:
+
+```ts
+// Rapid triggers with same key = single execution
+await myTask.trigger({ userId: "123" }, {
+  debounce: {
+    key: "user-123-update",
+    delay: "5s",
+  },
+});
+
+// Trailing mode: use payload from LAST trigger
+await myTask.trigger({ data: "latest" }, {
+  debounce: {
+    key: "my-key",
+    delay: "10s",
+    mode: "trailing",
+  },
+});
+```
+
+Use cases: user activity updates, webhook deduplication, search indexing, notification batching.
+
+## Batch Triggering
+
+Up to 1,000 items per batch, 3MB per payload:
+
+```ts
+const results = await myTask.batchTriggerAndWait([
+  { payload: { userId: "1" } },
+  { payload: { userId: "2" } },
+]);
+
+for (const result of results) {
+  if (result.ok) console.log(result.output);
+}
+```
+
+## Machine Presets
+
+| Preset      | vCPU | Memory |
+|-------------|------|--------|
+| micro       | 0.25 | 0.25GB |
+| small-1x    | 0.5  | 0.5GB  |
+| small-2x    | 1    | 1GB    |
+| medium-1x   | 1    | 2GB    |
+| medium-2x   | 2    | 4GB    |
+| large-1x    | 4    | 8GB    |
+| large-2x    | 8    | 16GB   |
+
+## Design Principles
+
+1. **Break complex workflows into subtasks** that can be independently retried and made idempotent
+2. **Don't over-complicate** - Sometimes `Promise.allSettled` inside a single task is better than many subtasks (each task has dedicated process and is charged by millisecond)
+3. **Always configure retries** - Set appropriate `maxAttempts` based on the operation
+4. **Use idempotency keys** - Especially for payment/critical operations
+5. **Group related subtasks** - Keep subtasks only used by one parent in the same file, don't export them
+6. **Use logger** - Log at key execution points with `logger.info()`, `logger.error()`, etc.
+
+## Reference Documentation
+
+For detailed documentation on specific topics, read these files:
+
+- `basic-tasks.md` - Task basics, triggering, waits
+- `advanced-tasks.md` - Tags, queues, concurrency, metadata, error handling
+- `scheduled-tasks.md` - Cron schedules, declarative and imperative
+- `realtime.md` - Real-time subscriptions, streams, React hooks
+- `config.md` - trigger.config.ts, build extensions (Prisma, Playwright, FFmpeg, etc.)