diff --git a/src/assets/images/agents/tracing-agent_overview.png b/src/assets/images/agents/tracing-agent_overview.png new file mode 100644 index 00000000000..dd4358cb938 Binary files /dev/null and b/src/assets/images/agents/tracing-agent_overview.png differ diff --git a/src/assets/images/agents/tracing-classic-waterfall.png b/src/assets/images/agents/tracing-classic-waterfall.png new file mode 100644 index 00000000000..f297ad62878 Binary files /dev/null and b/src/assets/images/agents/tracing-classic-waterfall.png differ diff --git a/src/assets/images/agents/tracing-per-agent.png b/src/assets/images/agents/tracing-per-agent.png new file mode 100644 index 00000000000..da5932097e8 Binary files /dev/null and b/src/assets/images/agents/tracing-per-agent.png differ diff --git a/src/assets/images/agents/tracing-session-replay.png b/src/assets/images/agents/tracing-session-replay.png new file mode 100644 index 00000000000..92f0a9446a2 Binary files /dev/null and b/src/assets/images/agents/tracing-session-replay.png differ diff --git a/src/content/changelog/agents/2026-03-02-agents-sdk-v0.7.0.mdx b/src/content/changelog/agents/2026-03-02-agents-sdk-v0.7.0.mdx index 820b1c70666..1f34c6070b5 100644 --- a/src/content/changelog/agents/2026-03-02-agents-sdk-v0.7.0.mdx +++ b/src/content/changelog/agents/2026-03-02-agents-sdk-v0.7.0.mdx @@ -72,7 +72,7 @@ export default { The custom `Observability` override interface is still supported for users who need to filter or forward events to external services. -For the full event reference, refer to the [Observability documentation](/agents/runtime/operations/observability/). +For the full event reference, refer to the [Diagnostics channels documentation](/agents/runtime/operations/observability/diagnostics-channels/). ## `keepAlive()` and `keepAliveWhile()` diff --git a/src/content/docs/agents/harnesses/think/index.mdx b/src/content/docs/agents/harnesses/think/index.mdx index 53dd5f391d8..c31799827fa 100644 --- a/src/content/docs/agents/harnesses/think/index.mdx +++ b/src/content/docs/agents/harnesses/think/index.mdx @@ -132,6 +132,23 @@ tag = "v1" +### Tracing + +Think uses `wrapAISDK()` internally to instrument model turns, tool calls, and approval lifecycle segments. You do not need to wrap the AI SDK or configure an adapter. To send `invoke_agent`, `chat`, `execute_tool`, and `tool_approval` spans to Workers Observability, turn on Workers traces: + + + +```toml +[observability.traces] +enabled = true +``` + + + +Traces appear in the Agents view in the Cloudflare Dashboard. You can inspect conversations and trace timelines. To turn tracing off, set `observability.traces.enabled` to `false`. You do not need to change the Think agent class. + +For span attributes, payload controls, exporting traces, and direct AI SDK v6 or v7 setup, refer to [Tracing](/agents/runtime/operations/observability/tracing/). + ## Think vs AIChatAgent Both Think and [`AIChatAgent`](/agents/communication-channels/chat/chat-agents/) extend `Agent` and speak the same `cf_agent_chat_*` WebSocket protocol. They serve different goals. diff --git a/src/content/docs/agents/harnesses/think/recovery.mdx b/src/content/docs/agents/harnesses/think/recovery.mdx index 9c5844edbb8..5fe26f0ece8 100644 --- a/src/content/docs/agents/harnesses/think/recovery.mdx +++ b/src/content/docs/agents/harnesses/think/recovery.mdx @@ -45,7 +45,7 @@ export class MyAgent extends Think { -The same recovery events are available through `agents/observability` on the `chat` channel; transcript repairs are emitted on the `transcript` channel. Refer to [Observability](/agents/runtime/operations/observability/#chat-recovery-events). +The same recovery events are available through `agents/observability` on the `chat` channel; transcript repairs are emitted on the `transcript` channel. Refer to [Diagnostics channels](/agents/runtime/operations/observability/diagnostics-channels/#chat-recovery-events). ## onChatRecovery @@ -232,7 +232,7 @@ export class MyAgent extends Think { -Use either layer alone, or both together: the proactive guard avoids most overflows, and the reactive backstop catches any that still slip through (for example, a turn that starts already over budget, or a single tool result so large that compaction cannot help — in which case it terminalizes cleanly). Both apply to every turn entry path (WebSocket, sub-agent `chat()`, and programmatic `saveMessages()` / `submitMessages()`), and both emit a `chat:context:compacted` [observability event](/agents/runtime/operations/observability/#chat-context-events). +Use either layer alone, or both together: the proactive guard avoids most overflows, and the reactive backstop catches any that still slip through (for example, a turn that starts already over budget, or a single tool result so large that compaction cannot help — in which case it terminalizes cleanly). Both apply to every turn entry path (WebSocket, sub-agent `chat()`, and programmatic `saveMessages()` / `submitMessages()`), and both emit a `chat:context:compacted` [diagnostics channel event](/agents/runtime/operations/observability/diagnostics-channels/#chat-context-events). :::note A no-op compaction cannot rescue an over-budget turn, so recovery is only as effective as your compaction configuration. For tool-heavy histories, configure a `tokenCounter` on `compactAfter()` (refer to [Sessions](/agents/runtime/lifecycle/sessions/#auto-compaction)). diff --git a/src/content/docs/agents/runtime/agents-api.mdx b/src/content/docs/agents/runtime/agents-api.mdx index 8347424197c..bede1c78b73 100644 --- a/src/content/docs/agents/runtime/agents-api.mdx +++ b/src/content/docs/agents/runtime/agents-api.mdx @@ -85,28 +85,29 @@ flowchart TD ## Server-side API reference -| Feature | Methods | Documentation | -| --------------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | -| **State** | `setState()`, `onStateChanged()`, `initialState` | [Store and sync state](/agents/runtime/lifecycle/state/) | -| **Callable methods** | `@callable()` decorator | [Callable methods](/agents/runtime/lifecycle/callable-methods/) | -| **Scheduling** | `schedule()`, `scheduleEvery()`, `getScheduleById()`, `listSchedules()` | [Schedule tasks](/agents/runtime/execution/schedule-tasks/) | -| **Durable execution** | `runFiber()`, `startFiber()`, `stash()`, `onFiberRecovered()`, `keepAlive()`, `keepAliveWhile()` | [Durable execution](/agents/runtime/execution/durable-execution/) | -| **Queue** | `queue()`, `dequeue()`, `dequeueAll()`, `getQueue()` | [Queue tasks](/agents/runtime/execution/queue-tasks/) | -| **WebSockets** | `onConnect()`, `onMessage()`, `onClose()`, `broadcast()` | [WebSockets](/agents/runtime/communication/websockets/) | -| **HTTP/SSE** | `onRequest()` | [HTTP and SSE](/agents/runtime/communication/http-sse/) | -| **Email** | `onEmail()`, `replyToEmail()` | [Email routing](/agents/communication-channels/email/) | -| **Workflows** | `runWorkflow()`, `waitForApproval()` | [Run Workflows](/agents/runtime/execution/run-workflows/) | -| **MCP Client** | `addMcpServer()`, `removeMcpServer()`, `getMcpServers()` | [MCP Client API](/agents/model-context-protocol/apis/client-api/) | -| **AI Models** | Workers AI, OpenAI, Anthropic bindings | [Using AI models](/agents/runtime/operations/using-ai-models/) | -| **Protocol messages** | `shouldSendProtocolMessages()`, `isConnectionProtocolEnabled()` | [Protocol messages](/agents/runtime/communication/protocol-messages/) | -| **Context** | `getCurrentAgent()` | [getCurrentAgent()](/agents/runtime/lifecycle/get-current-agent/) | -| **Observability** | `subscribe()`, diagnostics channels, Tail Workers | [Observability](/agents/runtime/operations/observability/) | -| **Sub-agents** | `subAgent()`, `abortSubAgent()`, `deleteSubAgent()` | [Sub-agents](/agents/runtime/execution/sub-agents/) | -| **Agents as tools** | `runAgentTool()`, `clearAgentToolRuns()`, `hasAgentToolRun()` | [Agents as tools](/agents/runtime/execution/agent-tools/) | -| **Agent Skills** | `skills` registry, bundled skill sources, script runners | [Agent Skills](/agents/runtime/execution/agent-skills/) | -| **Sessions** | `Session.create()`, context blocks, compaction, search | [Sessions](/agents/runtime/lifecycle/sessions/) | -| **Think** | `Think` base class, workspace tools, lifecycle hooks, extensions | [Think](/agents/harnesses/think/) | -| **Chat SDK** | `createChatSdkState()`, `ChatSdkStateAgent` | [Chat SDK](/agents/runtime/communication/chat-sdk/) | +| Feature | Methods | Documentation | +| ------------------------ | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- | +| **State** | `setState()`, `onStateChanged()`, `initialState` | [Store and sync state](/agents/runtime/lifecycle/state/) | +| **Callable methods** | `@callable()` decorator | [Callable methods](/agents/runtime/lifecycle/callable-methods/) | +| **Scheduling** | `schedule()`, `scheduleEvery()`, `getScheduleById()`, `listSchedules()` | [Schedule tasks](/agents/runtime/execution/schedule-tasks/) | +| **Durable execution** | `runFiber()`, `startFiber()`, `stash()`, `onFiberRecovered()`, `keepAlive()`, `keepAliveWhile()` | [Durable execution](/agents/runtime/execution/durable-execution/) | +| **Queue** | `queue()`, `dequeue()`, `dequeueAll()`, `getQueue()` | [Queue tasks](/agents/runtime/execution/queue-tasks/) | +| **WebSockets** | `onConnect()`, `onMessage()`, `onClose()`, `broadcast()` | [WebSockets](/agents/runtime/communication/websockets/) | +| **HTTP/SSE** | `onRequest()` | [HTTP and SSE](/agents/runtime/communication/http-sse/) | +| **Email** | `onEmail()`, `replyToEmail()` | [Email routing](/agents/communication-channels/email/) | +| **Workflows** | `runWorkflow()`, `waitForApproval()` | [Run Workflows](/agents/runtime/execution/run-workflows/) | +| **MCP Client** | `addMcpServer()`, `removeMcpServer()`, `getMcpServers()` | [MCP Client API](/agents/model-context-protocol/apis/client-api/) | +| **AI Models** | Workers AI, OpenAI, Anthropic bindings | [Using AI models](/agents/runtime/operations/using-ai-models/) | +| **Protocol messages** | `shouldSendProtocolMessages()`, `isConnectionProtocolEnabled()` | [Protocol messages](/agents/runtime/communication/protocol-messages/) | +| **Context** | `getCurrentAgent()` | [getCurrentAgent()](/agents/runtime/lifecycle/get-current-agent/) | +| **Tracing** | `wrapAISDK()` | [Tracing](/agents/runtime/operations/observability/tracing/) | +| **Diagnostics channels** | `subscribe()`, diagnostics channels | [Diagnostics channels](/agents/runtime/operations/observability/diagnostics-channels/) | +| **Sub-agents** | `subAgent()`, `abortSubAgent()`, `deleteSubAgent()` | [Sub-agents](/agents/runtime/execution/sub-agents/) | +| **Agents as tools** | `runAgentTool()`, `clearAgentToolRuns()`, `hasAgentToolRun()` | [Agents as tools](/agents/runtime/execution/agent-tools/) | +| **Agent Skills** | `skills` registry, bundled skill sources, script runners | [Agent Skills](/agents/runtime/execution/agent-skills/) | +| **Sessions** | `Session.create()`, context blocks, compaction, search | [Sessions](/agents/runtime/lifecycle/sessions/) | +| **Think** | `Think` base class, workspace tools, lifecycle hooks, extensions | [Think](/agents/harnesses/think/) | +| **Chat SDK** | `createChatSdkState()`, `ChatSdkStateAgent` | [Chat SDK](/agents/runtime/communication/chat-sdk/) | ## SQL API diff --git a/src/content/docs/agents/runtime/operations/observability.mdx b/src/content/docs/agents/runtime/operations/observability/diagnostics-channels.mdx similarity index 97% rename from src/content/docs/agents/runtime/operations/observability.mdx rename to src/content/docs/agents/runtime/operations/observability/diagnostics-channels.mdx index 81d14903316..3a16bc44b8e 100644 --- a/src/content/docs/agents/runtime/operations/observability.mdx +++ b/src/content/docs/agents/runtime/operations/observability/diagnostics-channels.mdx @@ -1,16 +1,16 @@ --- -title: Observability -description: Subscribe to structured Agent events for RPC calls, state changes, schedules, workflows, and MCP connections via diagnostics channels. -pcx_content_type: concept +title: Diagnostics channels +description: Subscribe to structured agent events for RPC calls, state changes, schedules, workflows, and MCP connections. +pcx_content_type: reference sidebar: - order: 30 + order: 2 products: - agents --- import { TypeScriptExample, LinkCard } from "~/components"; -Agents emit structured events for every significant operation — RPC calls, state changes, schedule execution, workflow transitions, MCP connections, and more. These events are published to [diagnostics channels](/workers/runtime-apis/nodejs/diagnostics-channel/) and are silent by default (zero overhead when nobody is listening). +Agents publish structured events to [diagnostics channels](/workers/runtime-apis/nodejs/diagnostics-channel/) for every significant operation -- RPC calls, state changes, schedule execution, workflow transitions, MCP connections, and more. Publishing has zero overhead when nobody is listening. ## Event structure @@ -297,6 +297,12 @@ These events track chat message lifecycle, client-side tool interactions, and Th ## Next steps + + - - diff --git a/src/content/docs/agents/runtime/operations/observability/index.mdx b/src/content/docs/agents/runtime/operations/observability/index.mdx new file mode 100644 index 00000000000..59aca38fbbb --- /dev/null +++ b/src/content/docs/agents/runtime/operations/observability/index.mdx @@ -0,0 +1,12 @@ +--- +title: Observability +pcx_content_type: navigation +sidebar: + order: 30 + group: + hideIndex: true +--- + +import { DirectoryListing } from "~/components"; + + diff --git a/src/content/docs/agents/runtime/operations/observability/tracing.mdx b/src/content/docs/agents/runtime/operations/observability/tracing.mdx new file mode 100644 index 00000000000..69ba554b9c0 --- /dev/null +++ b/src/content/docs/agents/runtime/operations/observability/tracing.mdx @@ -0,0 +1,275 @@ +--- +title: Tracing +description: Trace model calls, tool runs, and approvals with Workers traces. +pcx_content_type: concept +sidebar: + order: 1 +products: + - agents +--- + +import { LinkCard, TypeScriptExample, WranglerConfig } from "~/components"; + +Agent tracing helps you understand what an agent did at every turn, including its model calls, tool runs, and approval requests. Use traces to investigate unexpected behavior, find slow operations, and review token usage. + +Agent activity appears alongside runtime events such as fetch calls, KV reads, and D1 queries in [Workers traces](/workers/observability/traces/). + +## Enable tracing + +Enable tracing in your Wrangler configuration: + + + +```toml +[observability] +enabled = true + +[observability.traces] +enabled = true +``` + + + +## View agent activity + +Open the [Agents tab](https://dash.cloudflare.com/?to=/:account/agents) in the Cloudflare Dashboard to see traced agents and subagents. The overview shows each agent's model, session count, runs, and total token usage. + +A session is a conversation made up of one or more turns. A turn is one request to an agent and its response. + +> **Image placeholder:** Agents dashboard listing traced agents with run and token usage metrics. + +Select an agent to see its traces. Each trace includes its duration, token breakdown, and status. + +> **Image placeholder:** Agents dashboard showing traces for one agent. + +There are two ways to follow what an agent did: **Session replay** and **Trace**. + +**Session replay** shows the recorded conversation across turns, including messages, reasoning, tool calls, and subagent activity. What appears depends on your [payload recording settings](/agents/runtime/operations/observability/tracing/#payload-privacy). + +> **Image placeholder:** Session replay showing model messages and tool calls. + +**Trace** is a waterfall of the operations performed during one turn. It shows when each operation started, how long it took, and which operation called it. + +> **Image placeholder:** Trace waterfall showing nested agent, model, and tool spans. + +## Trace structure + +Each turn produces a trace made of spans, one for each timed operation: + +```txt +invoke_agent {agent class} +├── chat {model} +└── execute_tool {tool} + └── tool_approval {tool} +``` + +The `invoke_agent` span covers the turn. Model calls, tool runs, and approvals appear as nested spans. Subagent work appears under the operation that invoked it. + +:::note +Approval spans represent lifecycle events within a Worker invocation. They do not measure the time a person waits before responding across invocations. +::: + +### Agent identity + +Agent spans use three fields to identify the work shown in the dashboard: + +- **Agent name** identifies the logical agent implementation. Use a shared name such as `booking-agent`. +- **Agent ID** identifies the stable agent instance or resource, such as `booking-agent-production`. +- **Conversation ID** identifies the current conversation or session. + +Do not derive the agent name from a request, conversation, or user identifier. This creates too many distinct agent names in the dashboard. + +### Payload privacy + +Message and tool payloads can contain personally identifiable information. Only record payloads that are safe to store. Each integration controls payload recording differently, so configure it in the relevant [Framework setup](/agents/runtime/operations/observability/tracing/#framework-setup) section. + +Think and `wrapAISDK()` use `storeMessages` to record input and output messages on `chat` spans. They use `storeTools` to record arguments and results on `execute_tool` spans. + +## Framework setup + +Tracing and payload controls depend on the integration. Think and Flue instrument turns automatically. Direct AI SDK calls and custom harnesses need additional setup. + +### Think + +[Think](/agents/harnesses/think/) uses `wrapAISDK()` internally and emits the [standard span structure](/agents/runtime/operations/observability/tracing/#trace-structure). No additional tracing setup is required. + +Think supplies these identity fields: + +| Attribute | Think default | +| ------------------------ | --------------------------- | +| `gen_ai.agent.name` | `this.constructor.name` | +| `gen_ai.agent.id` | `this.name` | +| `gen_ai.conversation.id` | `this.ctx.id.toString()` | + +#### Store payloads + +Think does not store message or tool payloads by default. To record them, override the properties on the agent class: + + + +```ts +import { Think } from "@cloudflare/think"; + +export class MyAgent extends Think { + override storeMessages = true; + override storeTools = true; +} +``` + + + +### Flue + +[Flue](https://flueframework.com/) also instruments turns automatically with the same span structure as Think. + +### AI SDK v6 and v7 + +For direct [AI SDK](https://sdk.vercel.ai/) calls, wrap the namespace once: + + + +```ts +import * as ai from "ai"; +import { wrapAISDK } from "agents/observability/ai"; + +const tracedAI = wrapAISDK(ai); +``` + + + +`wrapAISDK()` supports AI SDK v6 and v7. It instruments `generateText`, `streamText`, `generateObject`, and `streamObject`, creating the `invoke_agent` parent before model and tool work begins. + +Unlike Think, a direct AI SDK call has no Agent instance from which to infer dashboard identity. Supply the [agent identity fields](/agents/runtime/operations/observability/tracing/#agent-identity) on each call. + +Only include context that is safe to store. Do not include credentials, tokens, user input, or other secrets. + +#### AI SDK v7 + +Use native AI SDK v7 telemetry fields to supply identity. You can include additional scalar context, such as a tenant or route, to make traces easier to query: + + + +```ts +await tracedAI.generateText({ + model, + prompt: "Find an available appointment", + runtimeContext: { + agentId: "booking-agent-production", + conversationId: "conversation-123", + tenantId: "tenant-42", + }, + telemetry: { + functionId: "booking-agent", + includeRuntimeContext: { + agentId: true, + conversationId: true, + tenantId: true, + }, + }, +}); +``` + + + +This maps `functionId`, `agentId`, and `conversationId` to `gen_ai.agent.name`, `gen_ai.agent.id`, and `gen_ai.conversation.id`. Other included scalar values use the `cloudflare.agents.runtime_context.*` namespace. + +#### AI SDK v6 + +AI SDK v6 uses `experimental_telemetry.metadata` for the same identity and additional span data: + + + +```ts +await tracedAI.generateText({ + model, + prompt: "Find an available appointment", + experimental_telemetry: { + functionId: "booking-agent", + metadata: { + agentId: "booking-agent-production", + conversationId: "conversation-123", + tenantId: "tenant-42", + }, + }, +}); +``` + + + +Additional scalar metadata uses the `cloudflare.agents.metadata.*` namespace. + +#### Store payloads + +`wrapAISDK()` does not store message or tool payloads by default. To record them with AI SDK v6 or v7, pass storage options when wrapping the namespace: + + + +```ts +import * as ai from "ai"; +import { wrapAISDK } from "agents/observability/ai"; + +const tracedAI = wrapAISDK(ai, { + storeMessages: true, + storeTools: true, +}); +``` + + + +### Custom harnesses + +If your agent does not use one of our currently supported frameworks, instrument it with the [Workers custom spans API](/workers/observability/traces/custom-spans/). Create an `invoke_agent` span for each turn, with `chat` spans for model calls, `execute_tool` spans for tool runs, and `tool_approval` spans for approvals. + +For span names, attributes, and implementation examples, refer to the [OpenTelemetry GenAI reference implementations](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/reference/README.md). Adapt these examples to the Workers custom spans API. + +:::note +Workers does not yet support the [OpenTelemetry API](https://opentelemetry.io/) directly. Cloudflare is working to add support so libraries and agent frameworks that emit OpenTelemetry spans can integrate without manual custom-span instrumentation. +::: + +#### Add agent identity + +Add these attributes to both the `invoke_agent` and `chat` spans so the Agents dashboard can associate the telemetry with the agent and conversation: + +| Attribute | `invoke_agent` span | `chat` span | +| ------------------------ | --------------------------------------------- | --------------------------------------------- | +| `gen_ai.operation.name` | `invoke_agent` | `chat` | +| `gen_ai.agent.name` | A shared agent name, such as `booking-agent` | The same agent name | +| `gen_ai.agent.id` | A stable identifier for the agent instance | The same agent ID | +| `gen_ai.conversation.id` | The conversation, session, or thread identifier | The same conversation, session, or thread ID | + +#### Store payloads + +For custom spans, add payload attributes manually. Use `gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.system_instructions` on model spans. Use `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` on tool spans. The custom spans API accepts scalar attribute values, so serialize structured payloads with `JSON.stringify()`. These values are recorded whenever the invocation is sampled. Only add payloads that are safe to store. + +## Exporting traces + +Span attributes follow the [OpenTelemetry Generative AI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), so any tool that reads OpenTelemetry data can consume them. To send traces to an external destination, [configure an OpenTelemetry Protocol (OTLP) endpoint](/workers/observability/exporting-opentelemetry-data/) in Workers Observability. + +## Pricing + +Agent tracing is free while [Workers traces](/workers/observability/traces/) is in beta. Each span counts as one observability event, sharing the same quota as Workers Logs and Traces. + +| Tier | Included events | Retention | +| ------------ | --------------------------------------------------- | --------- | +| Workers Free | 200,000 per day | 3 days | +| Workers Paid | 20 million per month ($0.60 per additional million) | 7 days | + +## Next steps + + + + + +