From f0aba9487f119f9c26c913b736a073a932a9f54f Mon Sep 17 00:00:00 2001 From: RulaKhaled Date: Wed, 9 Sep 2026 17:11:42 +0200 Subject: [PATCH 1/7] docs: Fix stale AI integration paths in AGENTS.md and add-ai-integration skill Co-Authored-By: Claude Opus 5 --- .agents/skills/add-ai-integration/SKILL.md | 78 +++++++++++----------- AGENTS.md | 13 ++-- 2 files changed, 46 insertions(+), 45 deletions(-) diff --git a/.agents/skills/add-ai-integration/SKILL.md b/.agents/skills/add-ai-integration/SKILL.md index 25d508ccc50a..83b57d4d4abf 100644 --- a/.agents/skills/add-ai-integration/SKILL.md +++ b/.agents/skills/add-ai-integration/SKILL.md @@ -12,30 +12,31 @@ argument-hint: Does the AI SDK have native OpenTelemetry support? |- YES -> Does it emit OTel spans automatically? | |- YES (like Vercel AI) -> Pattern 1: OTel Span Processors -| +- NO -> Pattern 2: OTel Instrumentation (wrap client) +| +- NO -> Pattern 2: Orchestrion Instrumentation (wrap client) +- NO -> Does the SDK provide hooks/callbacks? |- YES (like LangChain) -> Pattern 3: Callback/Hook Based +- NO -> Pattern 4: Client Wrapping ``` -## Runtime-Specific Placement +## Placement -If an AI SDK only works in one runtime, code lives exclusively in that runtime's package. Do NOT add it to `packages/core/`. +AI instrumentation lives in `packages/server-utils/`, not `packages/core/` and not the runtime packages: -- **Node.js-only** -> `packages/node/src/integrations/tracing/{provider}/` -- **Cloudflare-only** -> `packages/cloudflare/src/integrations/tracing/{provider}.ts` -- **Browser-only** -> `packages/browser/src/integrations/tracing/{provider}/` -- **Multi-runtime** -> shared core in `packages/core/src/tracing/{provider}/` with runtime-specific wrappers +- **Instrumentation logic** -> `packages/server-utils/src/ai/{provider}/` +- **Integration** (wires it up, registered in `getTracingIntegrations()`) -> `packages/server-utils/src/integrations/{provider}.ts` +- **Runtime packages** (`node`, `cloudflare`, `bun`, ...) re-export the integration from `@sentry/server-utils` -- they do not define their own + +Cloudflare-only client wrapping (Workers AI) is the exception: it is applied in `packages/cloudflare/src/instrumentations/worker/instrumentEnv.ts`, wrapping the binding from `env`. ## Span Hierarchy - `gen_ai.invoke_agent` — parent/pipeline spans (chains, agents, orchestration) - `gen_ai.chat`, `gen_ai.generate_text`, etc. — child spans (actual LLM calls) -## Shared Utilities (`packages/core/src/tracing/ai/`) +## Shared Utilities (`packages/server-utils/src/ai/core/`) - `gen-ai-attributes.ts` — OTel Semantic Convention attribute constants. **Always use these, never hardcode.** -- `utils.ts` — `setTokenUsageAttributes()`, `getTruncatedJsonString()`, `truncateGenAiMessages()`, `buildMethodPath()` +- `utils.ts` — `setTokenUsageAttributes()`, `buildMethodPath()`, `resolveAIRecordingOptions()`, `getGenAiSpanOp()`, `endStreamSpan()`, `extractSystemInstructions()` - Only use attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/). ## Streaming @@ -43,79 +44,78 @@ If an AI SDK only works in one runtime, code lives exclusively in that runtime's - **Non-streaming:** `startSpan()`, set attributes from response - **Streaming:** `startSpanManual()`, accumulate state via async generator or event listeners, set `GEN_AI_RESPONSE_STREAMING_ATTRIBUTE: true`, call `span.end()` in finally block - Detect via `params.stream === true` -- References: `openai/streaming.ts` (async generator), `anthropic-ai/streaming.ts` (event listeners) +- References: `ai/openai/streaming.ts` (async generator), `ai/anthropic-ai/streaming.ts` (event listeners) ## Token Accumulation - **Child spans:** Set tokens directly from API response via `setTokenUsageAttributes()` -- **Parent spans (`invoke_agent`):** Accumulate from children using event processor (see `vercel-ai/`) +- **Parent spans (`invoke_agent`):** Accumulate from children using event processor (see `ai/vercel-ai/`) ## Pattern 1: OTel Span Processors **Use when:** SDK emits OTel spans automatically (Vercel AI) -1. **Core:** Create `add{Provider}Processors()` in `packages/core/src/tracing/{provider}/index.ts` — registers `spanStart` listener + event processor -2. **Node.js:** Add `callWhenPatched()` optimization in `packages/node/src/integrations/tracing/{provider}/index.ts` — defers registration until package is imported -3. **Edge:** Direct registration in `packages/cloudflare/src/integrations/tracing/{provider}.ts` — no OTel, call processors immediately +1. Create `add{Provider}Processors()` in `packages/server-utils/src/ai/{provider}/index.ts` — registers `spanStart` listener + event processor +2. Wire it up in `packages/server-utils/src/integrations/{provider}.ts` and register in `getTracingIntegrations()` + +Reference: `packages/server-utils/src/ai/vercel-ai/` + `packages/server-utils/src/integrations/vercel-ai/` -Reference: `packages/node/src/integrations/tracing/vercelai/` +## Pattern 2: Orchestrion Instrumentation (Client Wrapping) -## Pattern 2: OTel Instrumentation (Client Wrapping) +**Use when:** SDK has no native telemetry of its own (OpenAI, Anthropic, Google GenAI) -**Use when:** SDK has no native OTel support (OpenAI, Anthropic, Google GenAI) +1. Create the span-building logic in `packages/server-utils/src/ai/{provider}/index.ts` +2. Declare the module/method targets in `packages/server-utils/src/orchestrion/config/{provider}.ts` +3. In `packages/server-utils/src/integrations/{provider}.ts`, call `invokeOrchestrionInstrumentation()` and bind the resulting `diagnostics_channel` tracing channels to spans via `bindTracingChannelToSpan()`. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. -1. **Core:** Create `instrument{Provider}Client()` in `packages/core/src/tracing/{provider}/index.ts` — Proxy to wrap client methods, create spans manually -2. **Node.js `instrumentation.ts`:** Patch module exports, wrap client constructor. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. -3. **Node.js `index.ts`:** Export integration function using `generateInstrumentOnce()` helper +Patching goes through orchestrion + Node `diagnostics_channel`, not OTel instrumentation packages. -Reference: `packages/node/src/integrations/tracing/openai/` +Reference: `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/ai/openai/` ## Pattern 3: Callback/Hook Based **Use when:** SDK provides lifecycle hooks (LangChain, LangGraph) -1. **Core:** Create `create{Provider}CallbackHandler()` — implement SDK's callback interface, create spans in callbacks -2. **Node.js `instrumentation.ts`:** Auto-inject callbacks by patching runnable methods. Disable underlying AI provider wrapping. +1. Create `create{Provider}CallbackHandler()` in `packages/server-utils/src/ai/{provider}/index.ts` — implement the SDK's callback/exporter interface, create spans in the callbacks +2. In `packages/server-utils/src/integrations/{provider}.ts`, auto-inject the handler by patching the relevant methods, and call `_INTERNAL_skipAiProviderWrapping()` to disable the underlying AI provider wrapping -Reference: `packages/node/src/integrations/tracing/langchain/` +Reference: `packages/server-utils/src/ai/langchain/`, and `packages/server-utils/src/ai/mastra/` for an exporter-shaped agent framework -## Auto-Instrumentation (Node.js) +## Registration -**Mandatory** for Node.js AI integrations. OTel only patches when the package is imported (zero cost if unused). +**Mandatory.** Patching only happens once the target package is imported (zero cost if unused). ### Steps -1. **Add to `getAutoPerformanceIntegrations()`** in `packages/node/src/integrations/tracing/index.ts` — LangChain MUST come first -2. **Add to `getOpenTelemetryInstrumentationToPreload()`** for OTel-based integrations -3. **Export from `packages/node/src/index.ts`**: integration function + options type +1. **Add to `getTracingIntegrations()`** in `packages/server-utils/src/integrations/index.ts` — LangChain MUST come first, so it can disable the AI provider integrations before they instrument +2. **Export from `packages/server-utils/src/index.ts`**: integration function + options type +3. **Re-export from the runtime packages** that support it (e.g. `packages/node/src/index.ts`, `packages/cloudflare/src/index.ts`) 4. **Add E2E tests:** - Node.js: `dev-packages/node-integration-tests/suites/tracing/{provider}/` - Cloudflare: `dev-packages/cloudflare-integration-tests/suites/tracing/{provider}/` - - Browser: `dev-packages/browser-integration-tests/suites/tracing/ai-providers/{provider}/` ## Key Rules 1. Respect `dataCollection.genAI` for recording input and output messages 2. Set `SEMANTIC_ATTRIBUTE_SENTRY_ORIGIN = 'auto.ai.{provider}'` (alphanumerics, `_`, `.` only) -3. Truncate large data with helper functions from `utils.ts` +3. Gate input/output message recording behind `resolveAIRecordingOptions()` 4. `gen_ai.invoke_agent` for parent ops, `gen_ai.chat` for child ops ## Checklist -- [ ] Runtime-specific code placed only in that runtime's package -- [ ] Added to `getAutoPerformanceIntegrations()` in correct order (Node.js) -- [ ] Added to `getOpenTelemetryInstrumentationToPreload()` (Node.js with OTel) -- [ ] Exported from appropriate package index +- [ ] Instrumentation in `packages/server-utils/src/ai/`, integration in `packages/server-utils/src/integrations/` +- [ ] Added to `getTracingIntegrations()` in correct order (LangChain first) +- [ ] Exported from `packages/server-utils/src/index.ts` and re-exported from the supported runtime packages - [ ] E2E tests added and verifying auto-instrumentation - [ ] Only used attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/) - [ ] JSDoc says "enabled by default" or "not enabled by default" - [ ] Documented how to disable (if auto-enabled) -- [ ] Verified OTel only patches when package imported (Node.js) +- [ ] Verified patching only happens when the target package is imported ## Reference Implementations -- **Pattern 1 (Span Processors):** `packages/node/src/integrations/tracing/vercelai/` -- **Pattern 2 (Client Wrapping):** `packages/node/src/integrations/tracing/openai/` -- **Pattern 3 (Callback/Hooks):** `packages/node/src/integrations/tracing/langchain/` +- **Pattern 1 (Span Processors):** `packages/server-utils/src/ai/vercel-ai/` +- **Pattern 2 (Client Wrapping):** `packages/server-utils/src/ai/openai/` + `packages/server-utils/src/integrations/openai.ts` +- **Pattern 3 (Callback/Hooks):** `packages/server-utils/src/ai/langchain/`, `packages/server-utils/src/ai/mastra/` **When in doubt, follow the pattern of the most similar existing integration.** diff --git a/AGENTS.md b/AGENTS.md index 9add28152464..39f39e4e2030 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,12 +68,13 @@ Uses **Git Flow** (see `docs/gitflow.md`). ## Architecture -- `packages/types/` is **deprecated — never modify it**. Types live in - `packages/core/`. -- An AI provider integration spans three places: core instrumentation in - `packages/core/src/tracing/{provider}/`, the Node integration in - `packages/node/src/integrations/tracing/{provider}/`, and the edge - runtime in `packages/cloudflare/src/integrations/tracing/{provider}.ts`. +- Types live in `packages/core/`. The `@sentry/types` package is gone. +- An AI provider integration spans two places, both in + `packages/server-utils/`: the gen-AI instrumentation logic in + `src/ai/{provider}/`, and the integration that wires it up in + `src/integrations/{provider}.ts`, registered in `getTracingIntegrations()`. + Runtime packages (`node`, `cloudflare`, ...) re-export from + `@sentry/server-utils` rather than defining their own. ## Linting & Formatting From 694550ab97f2f8a38c0059efb2795b3e56d4754e Mon Sep 17 00:00:00 2001 From: RulaKhaled Date: Wed, 9 Sep 2026 17:15:30 +0200 Subject: [PATCH 2/7] docs: Rewrite AI integration patterns to match orchestrion/diagnostics_channel The Vercel AI integration no longer consumes OTel spans: `ai` >= 7 is handled by subscribing to the SDK's native `ai:telemetry` tracing channel, and v4-v6 by orchestrion-injected channels. No span processor or event processor remains. Co-Authored-By: Claude Opus 5 --- .agents/skills/add-ai-integration/SKILL.md | 62 +++++++++++++--------- 1 file changed, 37 insertions(+), 25 deletions(-) diff --git a/.agents/skills/add-ai-integration/SKILL.md b/.agents/skills/add-ai-integration/SKILL.md index 83b57d4d4abf..f8025c3abd46 100644 --- a/.agents/skills/add-ai-integration/SKILL.md +++ b/.agents/skills/add-ai-integration/SKILL.md @@ -9,13 +9,11 @@ argument-hint: ## Decision Tree ``` -Does the AI SDK have native OpenTelemetry support? -|- YES -> Does it emit OTel spans automatically? -| |- YES (like Vercel AI) -> Pattern 1: OTel Span Processors -| +- NO -> Pattern 2: Orchestrion Instrumentation (wrap client) -+- NO -> Does the SDK provide hooks/callbacks? - |- YES (like LangChain) -> Pattern 3: Callback/Hook Based - +- NO -> Pattern 4: Client Wrapping +Does the SDK publish its own `diagnostics_channel` telemetry? +|- YES (ai >= 7) -> Pattern 1: Native tracing channel ++- NO -> Does the SDK expose callback/exporter hooks? + |- YES (LangChain, Mastra) -> Pattern 3: Callback/Exporter + +- NO (OpenAI, Anthropic, Google GenAI, ai < 7) -> Pattern 2: Orchestrion-injected channels ``` ## Placement @@ -49,32 +47,45 @@ Cloudflare-only client wrapping (Workers AI) is the exception: it is applied in ## Token Accumulation - **Child spans:** Set tokens directly from API response via `setTokenUsageAttributes()` -- **Parent spans (`invoke_agent`):** Accumulate from children using event processor (see `ai/vercel-ai/`) +- **Parent spans (`invoke_agent`):** Accumulate inside the channel subscriber as usage/finish chunks arrive, then set on the open parent span before ending it (see `integrations/vercel-ai/vercel-ai-dc-subscriber.ts`). There is no event processor doing this rollup. -## Pattern 1: OTel Span Processors +## Pattern 1: Native Tracing Channel -**Use when:** SDK emits OTel spans automatically (Vercel AI) +**Use when:** the SDK publishes to `diagnostics_channel` itself (`ai` >= 7 publishes `ai:telemetry`) -1. Create `add{Provider}Processors()` in `packages/server-utils/src/ai/{provider}/index.ts` — registers `spanStart` listener + event processor -2. Wire it up in `packages/server-utils/src/integrations/{provider}.ts` and register in `getTracingIntegrations()` +1. Write the subscriber in `packages/server-utils/src/integrations/{provider}/{provider}-dc-subscriber.ts` — read the channel payloads, open spans, set gen_ai attributes +2. Subscribe from the integration's `setupOnce()`, wrapped in `waitForTracingChannelBinding()` so it waits for the async-context binding: -Reference: `packages/server-utils/src/ai/vercel-ai/` + `packages/server-utils/src/integrations/vercel-ai/` +```ts +setupOnce() { + if (!dc.tracingChannel) return; + waitForTracingChannelBinding(() => { + subscribe{Provider}TracingChannel(dc.tracingChannel, options); + }); +} +``` + +Subscribing is a no-op on SDK versions that never publish, so it is always safe to call. + +Reference: `packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts` + +## Pattern 2: Orchestrion-Injected Channels -## Pattern 2: Orchestrion Instrumentation (Client Wrapping) +**Use when:** the SDK has no telemetry of its own (OpenAI, Anthropic, Google GenAI, `ai` < 7) -**Use when:** SDK has no native telemetry of its own (OpenAI, Anthropic, Google GenAI) +Orchestrion injects `diagnostics_channel` tracing channels into the target module's functions at load time; we then subscribe to those injected channels. This replaced the old OTel instrumentation packages — there is no `@opentelemetry/instrumentation-*` dependency in this path. -1. Create the span-building logic in `packages/server-utils/src/ai/{provider}/index.ts` -2. Declare the module/method targets in `packages/server-utils/src/orchestrion/config/{provider}.ts` -3. In `packages/server-utils/src/integrations/{provider}.ts`, call `invokeOrchestrionInstrumentation()` and bind the resulting `diagnostics_channel` tracing channels to spans via `bindTracingChannelToSpan()`. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. +1. Create the span-building/attribute logic in `packages/server-utils/src/ai/{provider}/` +2. Declare the module, version range, and methods to inject in `packages/server-utils/src/orchestrion/config/{provider}.ts` +3. In `packages/server-utils/src/integrations/{provider}.ts`, call `invokeOrchestrionInstrumentation(client, {provider}ModuleNames, fn, [options])` from `setup(client)`, and bind each injected channel to a span with `bindTracingChannelToSpan()`. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. -Patching goes through orchestrion + Node `diagnostics_channel`, not OTel instrumentation packages. +Reference: `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/orchestrion/config/openai.ts` -Reference: `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/ai/openai/` +**A provider can need both patterns.** `vercelAIIntegration` subscribes to the native `ai:telemetry` channel for `ai` >= 7 _and_ runs orchestrion injection for `ai` v4-v6, in the same integration. -## Pattern 3: Callback/Hook Based +## Pattern 3: Callback/Exporter -**Use when:** SDK provides lifecycle hooks (LangChain, LangGraph) +**Use when:** SDK provides lifecycle hooks or an exporter interface (LangChain, LangGraph, Mastra) 1. Create `create{Provider}CallbackHandler()` in `packages/server-utils/src/ai/{provider}/index.ts` — implement the SDK's callback/exporter interface, create spans in the callbacks 2. In `packages/server-utils/src/integrations/{provider}.ts`, auto-inject the handler by patching the relevant methods, and call `_INTERNAL_skipAiProviderWrapping()` to disable the underlying AI provider wrapping @@ -114,8 +125,9 @@ Reference: `packages/server-utils/src/ai/langchain/`, and `packages/server-utils ## Reference Implementations -- **Pattern 1 (Span Processors):** `packages/server-utils/src/ai/vercel-ai/` -- **Pattern 2 (Client Wrapping):** `packages/server-utils/src/ai/openai/` + `packages/server-utils/src/integrations/openai.ts` -- **Pattern 3 (Callback/Hooks):** `packages/server-utils/src/ai/langchain/`, `packages/server-utils/src/ai/mastra/` +- **Pattern 1 (Native channel):** `packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts` +- **Pattern 2 (Orchestrion channels):** `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/orchestrion/config/openai.ts` +- **Pattern 3 (Callback/Exporter):** `packages/server-utils/src/ai/langchain/`, `packages/server-utils/src/ai/mastra/` +- **Both patterns at once:** `packages/server-utils/src/integrations/vercel-ai/index.ts` **When in doubt, follow the pattern of the most similar existing integration.** From 3ad69e85dc371ac7c4133dfdb75ee217e00b34d8 Mon Sep 17 00:00:00 2001 From: RulaKhaled Date: Wed, 9 Sep 2026 17:21:12 +0200 Subject: [PATCH 3/7] docs: Correct truncation, span op, and streaming guidance in AI skill Resolves the open truncation question and sweeps the sections of the add-ai-integration skill that were not verified against the tree. Co-Authored-By: Claude Opus 5 --- .agents/skills/add-ai-integration/SKILL.md | 29 ++++++++++++++-------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/.agents/skills/add-ai-integration/SKILL.md b/.agents/skills/add-ai-integration/SKILL.md index f8025c3abd46..a402e0ca148a 100644 --- a/.agents/skills/add-ai-integration/SKILL.md +++ b/.agents/skills/add-ai-integration/SKILL.md @@ -29,20 +29,27 @@ Cloudflare-only client wrapping (Workers AI) is the exception: it is applied in ## Span Hierarchy - `gen_ai.invoke_agent` — parent/pipeline spans (chains, agents, orchestration) -- `gen_ai.chat`, `gen_ai.generate_text`, etc. — child spans (actual LLM calls) +- `gen_ai.chat`, `gen_ai.generate_content`, `gen_ai.embeddings`, `gen_ai.execute_tool` — child spans (actual LLM/tool calls) + +Do not hand-write the op string. Derive it with `getGenAiSpanOp(operationName)` from `ai/core/utils.ts`, and take the constants from `@sentry/conventions/op` (`GEN_AI_CHAT`, `GEN_AI_GENERATE_CONTENT`, `GEN_AI_EMBEDDINGS`, `GEN_AI_EXECUTE_TOOL`, `GEN_AI_HANDOFF`, `GEN_AI_INVOKE_AGENT`, `GEN_AI_RERANK` — that is the full set). An operation with no convention op (currently only `unknown`) falls back to the generic `function` op; the raw name is still preserved on `gen_ai.operation.name`. ## Shared Utilities (`packages/server-utils/src/ai/core/`) -- `gen-ai-attributes.ts` — OTel Semantic Convention attribute constants. **Always use these, never hardcode.** +- Attribute keys come from `@sentry/conventions/attributes` — import them there directly at the call site. **Never hardcode attribute strings.** +- `gen-ai-attributes.ts` — only the gap-fillers: attributes with no `@sentry/conventions` equivalent, Sentry-internal meta attributes, and keys we intentionally emit differently. Check conventions first; add here only if it genuinely has no equivalent. - `utils.ts` — `setTokenUsageAttributes()`, `buildMethodPath()`, `resolveAIRecordingOptions()`, `getGenAiSpanOp()`, `endStreamSpan()`, `extractSystemInstructions()` - Only use attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/). ## Streaming -- **Non-streaming:** `startSpan()`, set attributes from response -- **Streaming:** `startSpanManual()`, accumulate state via async generator or event listeners, set `GEN_AI_RESPONSE_STREAMING_ATTRIBUTE: true`, call `span.end()` in finally block -- Detect via `params.stream === true` -- References: `ai/openai/streaming.ts` (async generator), `ai/anthropic-ai/streaming.ts` (event listeners) +How the span is opened depends on the path: + +- **Channel path** (Patterns 1 & 2 — how auto-instrumentation actually runs): build the span with `startInactiveSpan()` inside the `getSpan` callback of `bindTracingChannelToSpan()` and let the binding own its lifecycle. For a streamed call, return `true` from the `deferSpanEnd` option to hand span-ending ownership to the stream wrapper; non-streaming results end through the normal `beforeSpanEnd` path. Detect the stream from the **result shape** (async-iterable, or the SDK's stream object), not from `params.stream` — see `wrapStreamResult()` in `integrations/openai.ts` and `integrations/anthropic.ts`. +- **Manual client wrapping** (`instrumentOpenAiClient()`, `instrumentAnthropicAiClient()`, ... in `ai/{provider}/index.ts`, the public manual-instrumentation API): non-streaming uses `startSpan()`; streaming uses `startSpanManual()` and detects via `params.stream === true` (or a method that always streams). + +Either way, do not set streaming response attributes by hand. Accumulate into a `StreamResponseState` and call `endStreamSpan(span, state, recordOutputs)` from `ai/core/utils.ts` — in a `finally` for an async generator, or from the stream's terminal event for a listener-based stream. It sets `GEN_AI_RESPONSE_STREAMING`, response id/model, token usage, finish reasons, output text and tool calls, and ends the span. + +References: `ai/openai/streaming.ts` (`instrumentStream`, async generator), `ai/anthropic-ai/streaming.ts` (`instrumentMessageStream`, event listeners) ## Token Accumulation @@ -107,9 +114,9 @@ Reference: `packages/server-utils/src/ai/langchain/`, and `packages/server-utils ## Key Rules -1. Respect `dataCollection.genAI` for recording input and output messages +1. Gate input/output message recording behind `resolveAIRecordingOptions()`, which resolves the integration's `recordInputs`/`recordOutputs` against the client's `dataCollection.genAI` settings. Never read `dataCollection.genAI` directly. 2. Set `SEMANTIC_ATTRIBUTE_SENTRY_ORIGIN = 'auto.ai.{provider}'` (alphanumerics, `_`, `.` only) -3. Gate input/output message recording behind `resolveAIRecordingOptions()` +3. **Do not truncate message payloads.** The `enableTruncation` flag and all AI truncation/media-stripping logic were removed in v11 (#23045); recorded messages are serialized with `stringify()` and set on the span as-is. Nothing downstream caps them either — `maxValueLength` only applies to `request.url` and exception values, and event normalization limits depth/breadth, not string length. Size limiting is handled server-side, so it is not a contributor concern. 4. `gen_ai.invoke_agent` for parent ops, `gen_ai.chat` for child ops ## Checklist @@ -118,9 +125,9 @@ Reference: `packages/server-utils/src/ai/langchain/`, and `packages/server-utils - [ ] Added to `getTracingIntegrations()` in correct order (LangChain first) - [ ] Exported from `packages/server-utils/src/index.ts` and re-exported from the supported runtime packages - [ ] E2E tests added and verifying auto-instrumentation -- [ ] Only used attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/) -- [ ] JSDoc says "enabled by default" or "not enabled by default" -- [ ] Documented how to disable (if auto-enabled) +- [ ] Only used attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/), with span ops derived via `getGenAiSpanOp()` +- [ ] Input/output recording gated on `resolveAIRecordingOptions()`; no truncation logic added +- [ ] JSDoc on the exported integration names the channels it subscribes to, the supported SDK versions, and the prerequisite (orchestrion-injected channels "require the Sentry runtime hook or bundler plugin") - [ ] Verified patching only happens when the target package is imported ## Reference Implementations From a56011b912d26ffdbe1bffd20dea60bdde343a5a Mon Sep 17 00:00:00 2001 From: RulaKhaled Date: Thu, 10 Sep 2026 12:14:03 +0200 Subject: [PATCH 4/7] docs: Trim add-ai-integration skill to what conventions don't already cover MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback: point at the gen-AI conventions, RFC 0153, and the OTel semconv rather than enumerating ops and attributes inline, which goes stale. Drop the parent-span token rollup guidance — cross-span accumulation does not work under span streaming and is computed product-side. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LEEG4nvszCnnckfDhq4mc5 --- .agents/skills/add-ai-integration/SKILL.md | 99 ++++++++-------------- 1 file changed, 34 insertions(+), 65 deletions(-) diff --git a/.agents/skills/add-ai-integration/SKILL.md b/.agents/skills/add-ai-integration/SKILL.md index a402e0ca148a..28a57cfb0885 100644 --- a/.agents/skills/add-ai-integration/SKILL.md +++ b/.agents/skills/add-ai-integration/SKILL.md @@ -6,6 +6,18 @@ argument-hint: # Adding a New AI Integration +## Read First + +Do not invent span names, ops, or attributes — they are specified elsewhere and change independently of this repo: + +- [Sentry gen_ai attributes](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/) and [gen_ai ops](https://getsentry.github.io/sentry-conventions/ops/#gen_ai) — the normative list +- [RFC 0153: Decoupling Sentry's generative AI conventions from OpenTelemetry](https://github.com/getsentry/rfcs/blob/main/text/0153-decoupling-sentrys-generative-ai-conventions-from-open-telemetry.md) — why we diverge from OTel +- [OTel gen-ai semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) — the upstream baseline + +In code: import attribute keys from `@sentry/conventions/attributes` and ops from `@sentry/conventions/op`. **Never hardcode either as a string.** Derive the op with `getGenAiSpanOp(operationName)` from `packages/server-utils/src/ai/core/utils.ts` rather than picking one by hand. + +`packages/server-utils/src/ai/core/gen-ai-attributes.ts` holds only gap-fillers: attributes with no `@sentry/conventions` equivalent, Sentry-internal meta attributes, and keys we intentionally emit differently. Check conventions first; add there only if it genuinely has no equivalent. + ## Decision Tree ``` @@ -22,57 +34,15 @@ AI instrumentation lives in `packages/server-utils/`, not `packages/core/` and n - **Instrumentation logic** -> `packages/server-utils/src/ai/{provider}/` - **Integration** (wires it up, registered in `getTracingIntegrations()`) -> `packages/server-utils/src/integrations/{provider}.ts` -- **Runtime packages** (`node`, `cloudflare`, `bun`, ...) re-export the integration from `@sentry/server-utils` -- they do not define their own +- **Runtime packages** (`node`, `cloudflare`, `bun`, ...) re-export the integration from `@sentry/server-utils` — they do not define their own Cloudflare-only client wrapping (Workers AI) is the exception: it is applied in `packages/cloudflare/src/instrumentations/worker/instrumentEnv.ts`, wrapping the binding from `env`. -## Span Hierarchy - -- `gen_ai.invoke_agent` — parent/pipeline spans (chains, agents, orchestration) -- `gen_ai.chat`, `gen_ai.generate_content`, `gen_ai.embeddings`, `gen_ai.execute_tool` — child spans (actual LLM/tool calls) - -Do not hand-write the op string. Derive it with `getGenAiSpanOp(operationName)` from `ai/core/utils.ts`, and take the constants from `@sentry/conventions/op` (`GEN_AI_CHAT`, `GEN_AI_GENERATE_CONTENT`, `GEN_AI_EMBEDDINGS`, `GEN_AI_EXECUTE_TOOL`, `GEN_AI_HANDOFF`, `GEN_AI_INVOKE_AGENT`, `GEN_AI_RERANK` — that is the full set). An operation with no convention op (currently only `unknown`) falls back to the generic `function` op; the raw name is still preserved on `gen_ai.operation.name`. - -## Shared Utilities (`packages/server-utils/src/ai/core/`) - -- Attribute keys come from `@sentry/conventions/attributes` — import them there directly at the call site. **Never hardcode attribute strings.** -- `gen-ai-attributes.ts` — only the gap-fillers: attributes with no `@sentry/conventions` equivalent, Sentry-internal meta attributes, and keys we intentionally emit differently. Check conventions first; add here only if it genuinely has no equivalent. -- `utils.ts` — `setTokenUsageAttributes()`, `buildMethodPath()`, `resolveAIRecordingOptions()`, `getGenAiSpanOp()`, `endStreamSpan()`, `extractSystemInstructions()` -- Only use attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/). - -## Streaming - -How the span is opened depends on the path: - -- **Channel path** (Patterns 1 & 2 — how auto-instrumentation actually runs): build the span with `startInactiveSpan()` inside the `getSpan` callback of `bindTracingChannelToSpan()` and let the binding own its lifecycle. For a streamed call, return `true` from the `deferSpanEnd` option to hand span-ending ownership to the stream wrapper; non-streaming results end through the normal `beforeSpanEnd` path. Detect the stream from the **result shape** (async-iterable, or the SDK's stream object), not from `params.stream` — see `wrapStreamResult()` in `integrations/openai.ts` and `integrations/anthropic.ts`. -- **Manual client wrapping** (`instrumentOpenAiClient()`, `instrumentAnthropicAiClient()`, ... in `ai/{provider}/index.ts`, the public manual-instrumentation API): non-streaming uses `startSpan()`; streaming uses `startSpanManual()` and detects via `params.stream === true` (or a method that always streams). - -Either way, do not set streaming response attributes by hand. Accumulate into a `StreamResponseState` and call `endStreamSpan(span, state, recordOutputs)` from `ai/core/utils.ts` — in a `finally` for an async generator, or from the stream's terminal event for a listener-based stream. It sets `GEN_AI_RESPONSE_STREAMING`, response id/model, token usage, finish reasons, output text and tool calls, and ends the span. - -References: `ai/openai/streaming.ts` (`instrumentStream`, async generator), `ai/anthropic-ai/streaming.ts` (`instrumentMessageStream`, event listeners) - -## Token Accumulation - -- **Child spans:** Set tokens directly from API response via `setTokenUsageAttributes()` -- **Parent spans (`invoke_agent`):** Accumulate inside the channel subscriber as usage/finish chunks arrive, then set on the open parent span before ending it (see `integrations/vercel-ai/vercel-ai-dc-subscriber.ts`). There is no event processor doing this rollup. - ## Pattern 1: Native Tracing Channel **Use when:** the SDK publishes to `diagnostics_channel` itself (`ai` >= 7 publishes `ai:telemetry`) -1. Write the subscriber in `packages/server-utils/src/integrations/{provider}/{provider}-dc-subscriber.ts` — read the channel payloads, open spans, set gen_ai attributes -2. Subscribe from the integration's `setupOnce()`, wrapped in `waitForTracingChannelBinding()` so it waits for the async-context binding: - -```ts -setupOnce() { - if (!dc.tracingChannel) return; - waitForTracingChannelBinding(() => { - subscribe{Provider}TracingChannel(dc.tracingChannel, options); - }); -} -``` - -Subscribing is a no-op on SDK versions that never publish, so it is always safe to call. +Write the subscriber next to the integration, and subscribe from `setupOnce()` wrapped in `waitForTracingChannelBinding()` so it waits for the async-context binding. Subscribing is a no-op on SDK versions that never publish, so it is always safe to call. Reference: `packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts` @@ -80,11 +50,11 @@ Reference: `packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscr **Use when:** the SDK has no telemetry of its own (OpenAI, Anthropic, Google GenAI, `ai` < 7) -Orchestrion injects `diagnostics_channel` tracing channels into the target module's functions at load time; we then subscribe to those injected channels. This replaced the old OTel instrumentation packages — there is no `@opentelemetry/instrumentation-*` dependency in this path. +Orchestrion injects tracing channels into the target module's functions at load time; we subscribe to those injected channels. This replaced the old OTel instrumentation packages — there is no `@opentelemetry/instrumentation-*` dependency in this path. -1. Create the span-building/attribute logic in `packages/server-utils/src/ai/{provider}/` -2. Declare the module, version range, and methods to inject in `packages/server-utils/src/orchestrion/config/{provider}.ts` -3. In `packages/server-utils/src/integrations/{provider}.ts`, call `invokeOrchestrionInstrumentation(client, {provider}ModuleNames, fn, [options])` from `setup(client)`, and bind each injected channel to a span with `bindTracingChannelToSpan()`. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. +1. Span-building/attribute logic in `packages/server-utils/src/ai/{provider}/` +2. Module, version range, and methods to inject in `packages/server-utils/src/orchestrion/config/{provider}.ts` +3. `invokeOrchestrionInstrumentation(...)` from `setup(client)` in the integration, binding each channel with `bindTracingChannelToSpan()`. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. Reference: `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/orchestrion/config/openai.ts` @@ -92,32 +62,38 @@ Reference: `packages/server-utils/src/integrations/openai.ts` + `packages/server ## Pattern 3: Callback/Exporter -**Use when:** SDK provides lifecycle hooks or an exporter interface (LangChain, LangGraph, Mastra) +**Use when:** the SDK provides lifecycle hooks or an exporter interface (LangChain, LangGraph, Mastra) -1. Create `create{Provider}CallbackHandler()` in `packages/server-utils/src/ai/{provider}/index.ts` — implement the SDK's callback/exporter interface, create spans in the callbacks -2. In `packages/server-utils/src/integrations/{provider}.ts`, auto-inject the handler by patching the relevant methods, and call `_INTERNAL_skipAiProviderWrapping()` to disable the underlying AI provider wrapping +Implement the SDK's callback/exporter interface in `packages/server-utils/src/ai/{provider}/`, auto-inject it from the integration by patching the relevant methods, and call `_INTERNAL_skipAiProviderWrapping()` to disable the underlying AI provider wrapping. Reference: `packages/server-utils/src/ai/langchain/`, and `packages/server-utils/src/ai/mastra/` for an exporter-shaped agent framework +## Streaming + +How the span is opened depends on the path: + +- **Channel path** (Patterns 1 & 2 — how auto-instrumentation actually runs): build the span with `startInactiveSpan()` inside the `getSpan` callback of `bindTracingChannelToSpan()` and let the binding own its lifecycle. For a streamed call, return `true` from `deferSpanEnd` to hand span-ending ownership to the stream wrapper; non-streaming results end through the normal `beforeSpanEnd` path. Detect the stream from the **result shape** (async-iterable, or the SDK's stream object), not from `params.stream`. +- **Manual client wrapping** (`instrumentOpenAiClient()`, `instrumentAnthropicAiClient()`, ... — the public manual-instrumentation API): non-streaming uses `startSpan()`; streaming uses `startSpanManual()` and detects via `params.stream === true`. + +Either way, do not set streaming response attributes by hand: accumulate into a `StreamResponseState` and call `endStreamSpan(span, state, recordOutputs)` from `ai/core/utils.ts` — in a `finally` for an async generator, or from the stream's terminal event for a listener-based stream. + +References: `ai/openai/streaming.ts` (async generator), `ai/anthropic-ai/streaming.ts` (event listeners), `integrations/openai.ts` and `integrations/anthropic.ts` (`wrapStreamResult()`) + ## Registration **Mandatory.** Patching only happens once the target package is imported (zero cost if unused). -### Steps - 1. **Add to `getTracingIntegrations()`** in `packages/server-utils/src/integrations/index.ts` — LangChain MUST come first, so it can disable the AI provider integrations before they instrument 2. **Export from `packages/server-utils/src/index.ts`**: integration function + options type 3. **Re-export from the runtime packages** that support it (e.g. `packages/node/src/index.ts`, `packages/cloudflare/src/index.ts`) -4. **Add E2E tests:** - - Node.js: `dev-packages/node-integration-tests/suites/tracing/{provider}/` - - Cloudflare: `dev-packages/cloudflare-integration-tests/suites/tracing/{provider}/` +4. **Add E2E tests:** `dev-packages/node-integration-tests/suites/tracing/{provider}/`, `dev-packages/cloudflare-integration-tests/suites/tracing/{provider}/` ## Key Rules 1. Gate input/output message recording behind `resolveAIRecordingOptions()`, which resolves the integration's `recordInputs`/`recordOutputs` against the client's `dataCollection.genAI` settings. Never read `dataCollection.genAI` directly. 2. Set `SEMANTIC_ATTRIBUTE_SENTRY_ORIGIN = 'auto.ai.{provider}'` (alphanumerics, `_`, `.` only) 3. **Do not truncate message payloads.** The `enableTruncation` flag and all AI truncation/media-stripping logic were removed in v11 (#23045); recorded messages are serialized with `stringify()` and set on the span as-is. Nothing downstream caps them either — `maxValueLength` only applies to `request.url` and exception values, and event normalization limits depth/breadth, not string length. Size limiting is handled server-side, so it is not a contributor concern. -4. `gen_ai.invoke_agent` for parent ops, `gen_ai.chat` for child ops +4. Set token usage on the span the SDK reports it for, via `setTokenUsageAttributes()`. Do not add cross-span rollup — totals over a span tree are computed product-side, and summing in the SDK does not survive span streaming. ## Checklist @@ -125,16 +101,9 @@ Reference: `packages/server-utils/src/ai/langchain/`, and `packages/server-utils - [ ] Added to `getTracingIntegrations()` in correct order (LangChain first) - [ ] Exported from `packages/server-utils/src/index.ts` and re-exported from the supported runtime packages - [ ] E2E tests added and verifying auto-instrumentation -- [ ] Only used attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/), with span ops derived via `getGenAiSpanOp()` +- [ ] Attributes and ops taken from `@sentry/conventions`, with the op derived via `getGenAiSpanOp()` - [ ] Input/output recording gated on `resolveAIRecordingOptions()`; no truncation logic added - [ ] JSDoc on the exported integration names the channels it subscribes to, the supported SDK versions, and the prerequisite (orchestrion-injected channels "require the Sentry runtime hook or bundler plugin") - [ ] Verified patching only happens when the target package is imported -## Reference Implementations - -- **Pattern 1 (Native channel):** `packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts` -- **Pattern 2 (Orchestrion channels):** `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/orchestrion/config/openai.ts` -- **Pattern 3 (Callback/Exporter):** `packages/server-utils/src/ai/langchain/`, `packages/server-utils/src/ai/mastra/` -- **Both patterns at once:** `packages/server-utils/src/integrations/vercel-ai/index.ts` - **When in doubt, follow the pattern of the most similar existing integration.** From 383a57ece2d42d0636f3ff145bd3297d65018196 Mon Sep 17 00:00:00 2001 From: RulaKhaled Date: Thu, 10 Sep 2026 12:25:02 +0200 Subject: [PATCH 5/7] docs: Correct why AI token rollup is dropped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Accumulating onto a still-open parent span does survive span streaming — the parent is snapshotted only when it ends. What does not survive is a rollup at serialization time: `captureSpan()` snapshots each span on its own end and no transaction event is assembled, so there is no finished tree to walk. The reason to drop it is that the product computes tree totals. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LEEG4nvszCnnckfDhq4mc5 --- .agents/skills/add-ai-integration/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/skills/add-ai-integration/SKILL.md b/.agents/skills/add-ai-integration/SKILL.md index 28a57cfb0885..db92e20cc18b 100644 --- a/.agents/skills/add-ai-integration/SKILL.md +++ b/.agents/skills/add-ai-integration/SKILL.md @@ -93,7 +93,7 @@ References: `ai/openai/streaming.ts` (async generator), `ai/anthropic-ai/streami 1. Gate input/output message recording behind `resolveAIRecordingOptions()`, which resolves the integration's `recordInputs`/`recordOutputs` against the client's `dataCollection.genAI` settings. Never read `dataCollection.genAI` directly. 2. Set `SEMANTIC_ATTRIBUTE_SENTRY_ORIGIN = 'auto.ai.{provider}'` (alphanumerics, `_`, `.` only) 3. **Do not truncate message payloads.** The `enableTruncation` flag and all AI truncation/media-stripping logic were removed in v11 (#23045); recorded messages are serialized with `stringify()` and set on the span as-is. Nothing downstream caps them either — `maxValueLength` only applies to `request.url` and exception values, and event normalization limits depth/breadth, not string length. Size limiting is handled server-side, so it is not a contributor concern. -4. Set token usage on the span the SDK reports it for, via `setTokenUsageAttributes()`. Do not add cross-span rollup — totals over a span tree are computed product-side, and summing in the SDK does not survive span streaming. +4. Set token usage on the span the SDK reports it for, via `setTokenUsageAttributes()`. Do not roll child usage up onto parent spans — tree totals are computed product-side, from the full span tree. A rollup done at serialization time is impossible anyway under span streaming: each span is snapshotted to JSON when it ends (`captureSpan()`), and no transaction event is assembled, so there is no finished tree to walk. ## Checklist From 9c92da9eed95f097334aed2c7a6082713b20999f Mon Sep 17 00:00:00 2001 From: RulaKhaled Date: Thu, 10 Sep 2026 12:30:16 +0200 Subject: [PATCH 6/7] docs: Trim add-ai-integration skill further Collapse the per-pattern prose into a table of use-case + reference file, merge the Registration section into the checklist it duplicated, and drop narration of things the cited reference files already show. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LEEG4nvszCnnckfDhq4mc5 --- .agents/skills/add-ai-integration/SKILL.md | 112 +++++++-------------- 1 file changed, 35 insertions(+), 77 deletions(-) diff --git a/.agents/skills/add-ai-integration/SKILL.md b/.agents/skills/add-ai-integration/SKILL.md index db92e20cc18b..de686bc6e565 100644 --- a/.agents/skills/add-ai-integration/SKILL.md +++ b/.agents/skills/add-ai-integration/SKILL.md @@ -6,19 +6,16 @@ argument-hint: # Adding a New AI Integration -## Read First +## Conventions First -Do not invent span names, ops, or attributes — they are specified elsewhere and change independently of this repo: +Span ops and attributes are specified outside this repo. Never invent or hardcode either: -- [Sentry gen_ai attributes](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/) and [gen_ai ops](https://getsentry.github.io/sentry-conventions/ops/#gen_ai) — the normative list -- [RFC 0153: Decoupling Sentry's generative AI conventions from OpenTelemetry](https://github.com/getsentry/rfcs/blob/main/text/0153-decoupling-sentrys-generative-ai-conventions-from-open-telemetry.md) — why we diverge from OTel -- [OTel gen-ai semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) — the upstream baseline +- [gen_ai attributes](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/) and [gen_ai ops](https://getsentry.github.io/sentry-conventions/ops/#gen_ai) — normative; import from `@sentry/conventions/attributes` and `@sentry/conventions/op` +- [RFC 0153](https://github.com/getsentry/rfcs/blob/main/text/0153-decoupling-sentrys-generative-ai-conventions-from-open-telemetry.md) — why Sentry's gen-AI conventions diverge from the [OTel gen-ai semconv](https://opentelemetry.io/docs/specs/semconv/gen-ai/) -In code: import attribute keys from `@sentry/conventions/attributes` and ops from `@sentry/conventions/op`. **Never hardcode either as a string.** Derive the op with `getGenAiSpanOp(operationName)` from `packages/server-utils/src/ai/core/utils.ts` rather than picking one by hand. +Derive the op with `getGenAiSpanOp()` from `ai/core/utils.ts` rather than picking one by hand. `ai/core/gen-ai-attributes.ts` is for gap-fillers only — keys with no `@sentry/conventions` equivalent — so check it last, not first. -`packages/server-utils/src/ai/core/gen-ai-attributes.ts` holds only gap-fillers: attributes with no `@sentry/conventions` equivalent, Sentry-internal meta attributes, and keys we intentionally emit differently. Check conventions first; add there only if it genuinely has no equivalent. - -## Decision Tree +## Which Pattern ``` Does the SDK publish its own `diagnostics_channel` telemetry? @@ -28,82 +25,43 @@ Does the SDK publish its own `diagnostics_channel` telemetry? +- NO (OpenAI, Anthropic, Google GenAI, ai < 7) -> Pattern 2: Orchestrion-injected channels ``` -## Placement - -AI instrumentation lives in `packages/server-utils/`, not `packages/core/` and not the runtime packages: - -- **Instrumentation logic** -> `packages/server-utils/src/ai/{provider}/` -- **Integration** (wires it up, registered in `getTracingIntegrations()`) -> `packages/server-utils/src/integrations/{provider}.ts` -- **Runtime packages** (`node`, `cloudflare`, `bun`, ...) re-export the integration from `@sentry/server-utils` — they do not define their own - -Cloudflare-only client wrapping (Workers AI) is the exception: it is applied in `packages/cloudflare/src/instrumentations/worker/instrumentEnv.ts`, wrapping the binding from `env`. - -## Pattern 1: Native Tracing Channel - -**Use when:** the SDK publishes to `diagnostics_channel` itself (`ai` >= 7 publishes `ai:telemetry`) - -Write the subscriber next to the integration, and subscribe from `setupOnce()` wrapped in `waitForTracingChannelBinding()` so it waits for the async-context binding. Subscribing is a no-op on SDK versions that never publish, so it is always safe to call. - -Reference: `packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts` - -## Pattern 2: Orchestrion-Injected Channels - -**Use when:** the SDK has no telemetry of its own (OpenAI, Anthropic, Google GenAI, `ai` < 7) - -Orchestrion injects tracing channels into the target module's functions at load time; we subscribe to those injected channels. This replaced the old OTel instrumentation packages — there is no `@opentelemetry/instrumentation-*` dependency in this path. - -1. Span-building/attribute logic in `packages/server-utils/src/ai/{provider}/` -2. Module, version range, and methods to inject in `packages/server-utils/src/orchestrion/config/{provider}.ts` -3. `invokeOrchestrionInstrumentation(...)` from `setup(client)` in the integration, binding each channel with `bindTracingChannelToSpan()`. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. - -Reference: `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/orchestrion/config/openai.ts` - -**A provider can need both patterns.** `vercelAIIntegration` subscribes to the native `ai:telemetry` channel for `ai` >= 7 _and_ runs orchestrion injection for `ai` v4-v6, in the same integration. - -## Pattern 3: Callback/Exporter - -**Use when:** the SDK provides lifecycle hooks or an exporter interface (LangChain, LangGraph, Mastra) - -Implement the SDK's callback/exporter interface in `packages/server-utils/src/ai/{provider}/`, auto-inject it from the integration by patching the relevant methods, and call `_INTERNAL_skipAiProviderWrapping()` to disable the underlying AI provider wrapping. - -Reference: `packages/server-utils/src/ai/langchain/`, and `packages/server-utils/src/ai/mastra/` for an exporter-shaped agent framework - -## Streaming - -How the span is opened depends on the path: - -- **Channel path** (Patterns 1 & 2 — how auto-instrumentation actually runs): build the span with `startInactiveSpan()` inside the `getSpan` callback of `bindTracingChannelToSpan()` and let the binding own its lifecycle. For a streamed call, return `true` from `deferSpanEnd` to hand span-ending ownership to the stream wrapper; non-streaming results end through the normal `beforeSpanEnd` path. Detect the stream from the **result shape** (async-iterable, or the SDK's stream object), not from `params.stream`. -- **Manual client wrapping** (`instrumentOpenAiClient()`, `instrumentAnthropicAiClient()`, ... — the public manual-instrumentation API): non-streaming uses `startSpan()`; streaming uses `startSpanManual()` and detects via `params.stream === true`. - -Either way, do not set streaming response attributes by hand: accumulate into a `StreamResponseState` and call `endStreamSpan(span, state, recordOutputs)` from `ai/core/utils.ts` — in a `finally` for an async generator, or from the stream's terminal event for a listener-based stream. +| Pattern | Use when | Reference | +| -------------------------- | ------------------------------------------ | --------------------------------------------------------------- | +| 1 — Native tracing channel | the SDK publishes to `diagnostics_channel` | `integrations/vercel-ai/vercel-ai-dc-subscriber.ts` | +| 2 — Orchestrion channels | the SDK has no telemetry of its own | `integrations/openai.ts` + `orchestrion/config/openai.ts` | +| 3 — Callback/exporter | the SDK exposes hooks or an exporter | `ai/langchain/`, `ai/mastra/` (exporter-shaped agent framework) | -References: `ai/openai/streaming.ts` (async generator), `ai/anthropic-ai/streaming.ts` (event listeners), `integrations/openai.ts` and `integrations/anthropic.ts` (`wrapStreamResult()`) +What the reference files won't tell you: -## Registration +- Pattern 2 replaced the old OTel instrumentation packages — there is no `@opentelemetry/instrumentation-*` dependency in this path. +- A provider can need two patterns at once: `vercelAIIntegration` subscribes to native `ai:telemetry` for `ai` >= 7 _and_ runs orchestrion injection for v4-v6. +- Pattern 1 subscribers are safe to register unconditionally — subscribing is a no-op on SDK versions that never publish. -**Mandatory.** Patching only happens once the target package is imported (zero cost if unused). +## Where The Code Goes -1. **Add to `getTracingIntegrations()`** in `packages/server-utils/src/integrations/index.ts` — LangChain MUST come first, so it can disable the AI provider integrations before they instrument -2. **Export from `packages/server-utils/src/index.ts`**: integration function + options type -3. **Re-export from the runtime packages** that support it (e.g. `packages/node/src/index.ts`, `packages/cloudflare/src/index.ts`) -4. **Add E2E tests:** `dev-packages/node-integration-tests/suites/tracing/{provider}/`, `dev-packages/cloudflare-integration-tests/suites/tracing/{provider}/` +- **Instrumentation** -> `packages/server-utils/src/ai/{provider}/` +- **Integration** -> `packages/server-utils/src/integrations/{provider}.ts` +- Runtime packages (`node`, `cloudflare`, `bun`, ...) re-export from `@sentry/server-utils` — they never define their own +- Exception: Workers AI is client-wrapped in `packages/cloudflare/src/instrumentations/worker/instrumentEnv.ts` -## Key Rules +## Gotchas -1. Gate input/output message recording behind `resolveAIRecordingOptions()`, which resolves the integration's `recordInputs`/`recordOutputs` against the client's `dataCollection.genAI` settings. Never read `dataCollection.genAI` directly. -2. Set `SEMANTIC_ATTRIBUTE_SENTRY_ORIGIN = 'auto.ai.{provider}'` (alphanumerics, `_`, `.` only) -3. **Do not truncate message payloads.** The `enableTruncation` flag and all AI truncation/media-stripping logic were removed in v11 (#23045); recorded messages are serialized with `stringify()` and set on the span as-is. Nothing downstream caps them either — `maxValueLength` only applies to `request.url` and exception values, and event normalization limits depth/breadth, not string length. Size limiting is handled server-side, so it is not a contributor concern. -4. Set token usage on the span the SDK reports it for, via `setTokenUsageAttributes()`. Do not roll child usage up onto parent spans — tree totals are computed product-side, from the full span tree. A rollup done at serialization time is impossible anyway under span streaming: each span is snapshotted to JSON when it ends (`captureSpan()`), and no transaction event is assembled, so there is no finished tree to walk. +1. **Detect streaming from the result shape** — an async-iterable or the SDK's stream object — not from `params.stream`. Only the manual `instrument{Provider}Client()` API keys off `params.stream === true`. +2. **Never set streamed response attributes by hand.** Accumulate into a `StreamResponseState` and call `endStreamSpan()` (`ai/openai/streaming.ts` for an async generator, `ai/anthropic-ai/streaming.ts` for a listener-based stream). +3. **Never truncate message payloads.** Truncation was removed in v11 (#23045) and nothing downstream caps them; size limiting is server-side. +4. **Never roll child token usage up onto parent spans.** Tree totals are computed product-side from the full span tree. +5. **Never read `dataCollection.genAI` directly.** Gate input/output recording on `resolveAIRecordingOptions()`. +6. **LangChain must be registered first** in `getTracingIntegrations()`, so it can disable the provider integrations before they instrument. +7. Set `SEMANTIC_ATTRIBUTE_SENTRY_ORIGIN = 'auto.ai.{provider}'` (alphanumerics, `_`, `.` only). ## Checklist -- [ ] Instrumentation in `packages/server-utils/src/ai/`, integration in `packages/server-utils/src/integrations/` -- [ ] Added to `getTracingIntegrations()` in correct order (LangChain first) -- [ ] Exported from `packages/server-utils/src/index.ts` and re-exported from the supported runtime packages -- [ ] E2E tests added and verifying auto-instrumentation -- [ ] Attributes and ops taken from `@sentry/conventions`, with the op derived via `getGenAiSpanOp()` -- [ ] Input/output recording gated on `resolveAIRecordingOptions()`; no truncation logic added -- [ ] JSDoc on the exported integration names the channels it subscribes to, the supported SDK versions, and the prerequisite (orchestrion-injected channels "require the Sentry runtime hook or bundler plugin") -- [ ] Verified patching only happens when the target package is imported +- [ ] Instrumentation in `src/ai/`, integration in `src/integrations/`, registered in `getTracingIntegrations()` (LangChain first) +- [ ] Exported from `packages/server-utils/src/index.ts`, re-exported from the supported runtime packages +- [ ] E2E tests in `dev-packages/node-integration-tests/suites/tracing/{provider}/` (and `cloudflare-integration-tests/` if supported) +- [ ] Ops and attributes from `@sentry/conventions`, op derived via `getGenAiSpanOp()` +- [ ] Recording gated on `resolveAIRecordingOptions()`; no truncation, no token rollup +- [ ] JSDoc names the channels subscribed to, the supported SDK versions, and — for Pattern 2 — that it requires the Sentry runtime hook or bundler plugin +- [ ] Patching happens only once the target package is imported (zero cost if unused) **When in doubt, follow the pattern of the most similar existing integration.** From 135c0ca148fa051ffeb269e958ea52691847cec5 Mon Sep 17 00:00:00 2001 From: RulaKhaled Date: Thu, 10 Sep 2026 15:02:18 +0200 Subject: [PATCH 7/7] docs: Drop stale OTel-path note from AI skill Review feedback: the OTel instrumentation path is gone entirely (no `@opentelemetry/instrumentation-*` dep in server-utils, no import in any AI code), so noting its absence only serves a stale mental model. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LEEG4nvszCnnckfDhq4mc5 --- .agents/skills/add-ai-integration/SKILL.md | 1 - 1 file changed, 1 deletion(-) diff --git a/.agents/skills/add-ai-integration/SKILL.md b/.agents/skills/add-ai-integration/SKILL.md index de686bc6e565..a32d2ef766c9 100644 --- a/.agents/skills/add-ai-integration/SKILL.md +++ b/.agents/skills/add-ai-integration/SKILL.md @@ -33,7 +33,6 @@ Does the SDK publish its own `diagnostics_channel` telemetry? What the reference files won't tell you: -- Pattern 2 replaced the old OTel instrumentation packages — there is no `@opentelemetry/instrumentation-*` dependency in this path. - A provider can need two patterns at once: `vercelAIIntegration` subscribes to native `ai:telemetry` for `ai` >= 7 _and_ runs orchestrion injection for v4-v6. - Pattern 1 subscribers are safe to register unconditionally — subscribing is a no-op on SDK versions that never publish.