diff --git a/docs/platforms/javascript/common/agent-tracing/index.mdx b/docs/platforms/javascript/common/agent-tracing/index.mdx index c5bfc04b11c8c..bc89b385b38dd 100644 --- a/docs/platforms/javascript/common/agent-tracing/index.mdx +++ b/docs/platforms/javascript/common/agent-tracing/index.mdx @@ -84,6 +84,11 @@ As of SDK version `11.0.0`, Sentry auto-instruments the most popular AI SDKs and title: "Together AI", icon: "together-ai", }, + { + to: "/configuration/integrations/typesafe/", + title: "TypeSafe", + icon: "typesafe-ai", + }, { to: "/agent-tracing/mastra/", href: "/platforms/javascript/guides/mastra/", diff --git a/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx new file mode 100644 index 0000000000000..a38bd65ab5f4a --- /dev/null +++ b/docs/platforms/javascript/common/configuration/integrations/typesafe.mdx @@ -0,0 +1,151 @@ +--- +title: TypeSafe +description: "Adds instrumentation for TypeSafe Jev evaluations. (default)" +supportedCategories: + - server + - serverless +notSupported: + - javascript.cloudflare + - javascript.mastra +--- + + + +Requires Sentry SDK version `11.1.0` or higher. + + + +_Import name: `Sentry.typesafeIntegration`_ + +This integration is enabled by default when tracing is enabled. To customize it, see [Modifying Default Integrations](./../#modifying-default-integrations). + +The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture agent tracing `gen_ai.evaluate` spans, recording model, token usage, latency, and (when enabled) evaluation inputs and answers. + +If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. + +```javascript +Sentry.init({ + dsn: "___PUBLIC_DSN___", + // Tracing must be enabled for agent tracing to work + tracesSampleRate: 1.0, + integrations: [Sentry.typesafeIntegration()], +}); +``` + + + +Run TypeSafe calls in the Node.js runtime. This automatic instrumentation isn't available in the Edge runtime. + + + + + +On Deno, preload Sentry's import hook so the TypeSafe SDK can be instrumented: + +```bash +deno run --allow-net --allow-env --allow-read --allow-sys --allow-write --preload=npm:@sentry/deno/import app.ts +``` + + + + + +On Bun, automatic instrumentation requires a build with `sentryBunPlugin()` and `@typesafe-ai/sdk` included in the bundle: + +```typescript {filename:build.ts} +import { sentryBunPlugin } from "@sentry/bun/plugin"; + +await Bun.build({ + entrypoints: ["./src/index.ts"], + outdir: "./dist", + target: "bun", + plugins: [sentryBunPlugin()], +}); +``` + +For unbundled applications started with `bun run`, use [Manual Instrumentation](#manual-instrumentation). + + + +## Supported Operations + +Tracing support is added to `TypeSafeClient.systemOne()` calls to evaluate state against a set of questions using Jev. Each call creates a `gen_ai.evaluate` span. Recorded answers include probabilities and confidence values when returned by TypeSafe. + +## Supported Versions + +- `@typesafe-ai/sdk`: `>=0.5.0 <1` + +## Options + +Evaluation state and answers can contain sensitive data. Input and output recording are enabled by default. To disable them for TypeSafe, configure the integration explicitly: + +```javascript +Sentry.init({ + dsn: "___PUBLIC_DSN___", + tracesSampleRate: 1.0, + integrations: [ + Sentry.typesafeIntegration({ + recordInputs: false, + recordOutputs: false, + }), + ], +}); +``` + +### `recordInputs` + +_Type: `boolean` (optional)_ + +Records evaluation `state` and `questions`. Defaults to `dataCollection.genAI.inputs`, which is `true` by default. + +### `recordOutputs` + +_Type: `boolean` (optional)_ + +Records evaluation `answers`. Defaults to `dataCollection.genAI.outputs`, which is `true` by default. + + + +## Manual Instrumentation + +_Import name: `Sentry.instrumentTypeSafeClient`_ + +If your application does not use automatic instrumentation, wrap your TypeSafe client after setting up Sentry. Use the returned client for your `systemOne()` calls. + + + +Import Sentry from the entry point for your runtime: `@sentry/hono/node`, `@sentry/hono/bun`, or `@sentry/hono/cloudflare`. + + + + + +```javascript +import * as Sentry from "___SDK_PACKAGE___"; +``` + + + +```javascript +import { TypeSafeClient, choice } from "@typesafe-ai/sdk"; + +const typesafe = new TypeSafeClient({ apiKey: "your-typesafe-api-key" }); +const client = Sentry.instrumentTypeSafeClient(typesafe, { + recordInputs: false, + recordOutputs: false, +}); + +const result = await client.systemOne({ + state: "I was charged twice. Please refund the extra charge.", + questions: { + department: choice("Which team should handle this?", { + billing: "Charges and refunds", + technical: "Bugs and outages", + }), + }, +}); +``` + +Use one instrumentation approach for each client. If automatic instrumentation is already active, you don't need to wrap the client. + + diff --git a/docs/platforms/javascript/common/configuration/integrations/vercelai.mdx b/docs/platforms/javascript/common/configuration/integrations/vercelai.mdx index cac73ce702cc1..a7fd32fe3c693 100644 --- a/docs/platforms/javascript/common/configuration/integrations/vercelai.mdx +++ b/docs/platforms/javascript/common/configuration/integrations/vercelai.mdx @@ -56,9 +56,43 @@ By default, tracing support is added to the following `ai` functions: - `embed()` - `embedMany()` - `rerank()` +- `experimental_evaluate()` — requires Sentry SDK `11.1.0` or higher and AI SDK 7 Plus `generate()` and `stream()` on [`ToolLoopAgent`](https://ai-sdk.dev/docs/agents/overview#toolloopagent-class). +## Trace Jev Evaluations + +With Sentry SDK `11.1.0` or higher and AI SDK 7, `vercelAIIntegration` automatically traces `experimental_evaluate()` calls, including calls to TypeSafe's Jev model through Vercel AI Gateway. Keep your existing Sentry setup with tracing enabled; no additional integration or telemetry registration is needed. + +Configure [AI Gateway authentication](https://ai-sdk.dev/docs/ai-sdk-core/evaluation#default-provider-strings) with `AI_GATEWAY_API_KEY` or Vercel OIDC, then call `experimental_evaluate()`: + +```javascript +import { experimental_evaluate } from "ai"; + +const result = await experimental_evaluate({ + model: "typesafe-ai/jev", + state: "I was charged twice. Please refund the extra charge.", + questions: { + department: { + type: "choice", + instructions: "Which team should handle this?", + criteria: { + billing: "Charges and refunds", + technical: "Bugs and outages", + }, + }, + }, +}); + +console.log(result.answers.department); +``` + +Each call creates a `gen_ai.evaluate` span with model, latency, and token usage when available. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages. For TypeSafe answers, Sentry also includes confidence values from `providerMetadata.typesafe.confidence` when present. + +Evaluation inputs and outputs follow the same [`recordInputs`](#recordinputs) and [`recordOutputs`](#recordoutputs) options as other Vercel AI calls. Disable these options if evaluation state or answers contain sensitive data. + +The AI SDK's [evaluation API](https://ai-sdk.dev/docs/ai-sdk-core/evaluation) is experimental and may change in patch releases. If you use `TypeSafeClient.systemOne()` directly instead, see the [TypeSafe integration](/platforms/javascript/guides/node/configuration/integrations/typesafe/). + ## Supported Versions - `ai`: `>=3.0.0 <=7` diff --git a/platform-includes/configuration/integrations/javascript.astro.mdx b/platform-includes/configuration/integrations/javascript.astro.mdx index 305ea4bd3cbee..a0da2518a03b4 100644 --- a/platform-includes/configuration/integrations/javascript.astro.mdx +++ b/platform-includes/configuration/integrations/javascript.astro.mdx @@ -93,6 +93,7 @@ Depending on whether an integration enhances the functionality of a particular r | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`firebaseIntegration`](./firebase) | ✓ | | ✓ | | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`pinoIntegration`](./pino) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.aws-lambda.mdx b/platform-includes/configuration/integrations/javascript.aws-lambda.mdx index daf01fb6677d3..be0f437474b86 100644 --- a/platform-includes/configuration/integrations/javascript.aws-lambda.mdx +++ b/platform-includes/configuration/integrations/javascript.aws-lambda.mdx @@ -26,6 +26,7 @@ | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`amqplibIntegration`](./amqplib) | | | ✓ | | | [`anrIntegration`](./anr) | | ✓ | | | | [`captureConsoleIntegration`](./captureconsole) | | | | ✓ | diff --git a/platform-includes/configuration/integrations/javascript.bun.mdx b/platform-includes/configuration/integrations/javascript.bun.mdx index 30c3ec00901c1..d6122d40e41b7 100644 --- a/platform-includes/configuration/integrations/javascript.bun.mdx +++ b/platform-includes/configuration/integrations/javascript.bun.mdx @@ -38,6 +38,7 @@ | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`firebaseIntegration`](./firebase) | ✓ | | ✓ | | | [`bunRuntimeMetricsIntegration`](./bunruntimemetrics) | | | | ✓ | | [`captureConsoleIntegration`](./captureconsole) | | | | ✓ | diff --git a/platform-includes/configuration/integrations/javascript.deno.mdx b/platform-includes/configuration/integrations/javascript.deno.mdx index bc060c536aaf5..fdd9fc3df31ab 100644 --- a/platform-includes/configuration/integrations/javascript.deno.mdx +++ b/platform-includes/configuration/integrations/javascript.deno.mdx @@ -21,3 +21,4 @@ | [`trpcMiddleware`](./trpc) | | ✓ | ✓ | | ✓ | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.fastify.mdx b/platform-includes/configuration/integrations/javascript.fastify.mdx index c928ec5d0d60d..7c5077bcc18d3 100644 --- a/platform-includes/configuration/integrations/javascript.fastify.mdx +++ b/platform-includes/configuration/integrations/javascript.fastify.mdx @@ -38,6 +38,7 @@ | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`anrIntegration`](./anr) | | ✓ | | | | [`captureConsoleIntegration`](./captureconsole) | | | | ✓ | | [`eventLoopBlockIntegration`](./event-loop-block) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.gcp-functions.mdx b/platform-includes/configuration/integrations/javascript.gcp-functions.mdx index 5914b98376820..73494ecfd4e11 100644 --- a/platform-includes/configuration/integrations/javascript.gcp-functions.mdx +++ b/platform-includes/configuration/integrations/javascript.gcp-functions.mdx @@ -26,6 +26,7 @@ | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`firebaseIntegration`](./firebase) | ✓ | | ✓ | | | [`amqplibIntegration`](./amqplib) | | | ✓ | | | [`anrIntegration`](./anr) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.hapi.mdx b/platform-includes/configuration/integrations/javascript.hapi.mdx index 40090a1bf4908..f808916317e36 100644 --- a/platform-includes/configuration/integrations/javascript.hapi.mdx +++ b/platform-includes/configuration/integrations/javascript.hapi.mdx @@ -39,6 +39,7 @@ | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`anrIntegration`](./anr) | | ✓ | | | | [`captureConsoleIntegration`](./captureconsole) | | | | ✓ | | [`eventLoopBlockIntegration`](./event-loop-block) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.nestjs.mdx b/platform-includes/configuration/integrations/javascript.nestjs.mdx index 1c0a32c016555..4e9964a5dc4e9 100644 --- a/platform-includes/configuration/integrations/javascript.nestjs.mdx +++ b/platform-includes/configuration/integrations/javascript.nestjs.mdx @@ -52,5 +52,6 @@ | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`pinoIntegration`](./pino) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.nextjs.mdx b/platform-includes/configuration/integrations/javascript.nextjs.mdx index e5fbd3eb908cd..47e1a7bece612 100644 --- a/platform-includes/configuration/integrations/javascript.nextjs.mdx +++ b/platform-includes/configuration/integrations/javascript.nextjs.mdx @@ -100,6 +100,7 @@ Depending on whether an integration enhances the functionality of a particular r | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`pinoIntegration`](./pino) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.node.mdx b/platform-includes/configuration/integrations/javascript.node.mdx index 489437d96bed0..4c43bf85481aa 100644 --- a/platform-includes/configuration/integrations/javascript.node.mdx +++ b/platform-includes/configuration/integrations/javascript.node.mdx @@ -51,6 +51,7 @@ | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`firebaseIntegration`](./firebase) | ✓ | | ✓ | | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`pinoIntegration`](./pino) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.nuxt.mdx b/platform-includes/configuration/integrations/javascript.nuxt.mdx index 45e809cc438e3..ad261d98c0db8 100644 --- a/platform-includes/configuration/integrations/javascript.nuxt.mdx +++ b/platform-includes/configuration/integrations/javascript.nuxt.mdx @@ -93,5 +93,6 @@ Depending on whether an integration enhances the functionality of a particular r | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`pinoIntegration`](./pino) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.remix.mdx b/platform-includes/configuration/integrations/javascript.remix.mdx index 0a099873775dd..27031afcce658 100644 --- a/platform-includes/configuration/integrations/javascript.remix.mdx +++ b/platform-includes/configuration/integrations/javascript.remix.mdx @@ -93,5 +93,6 @@ Depending on whether an integration enhances the functionality of a particular r | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`pinoIntegration`](./pino) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.solidstart.mdx b/platform-includes/configuration/integrations/javascript.solidstart.mdx index a71907c8ea8f8..f0b4ffb9da789 100644 --- a/platform-includes/configuration/integrations/javascript.solidstart.mdx +++ b/platform-includes/configuration/integrations/javascript.solidstart.mdx @@ -93,5 +93,6 @@ Depending on whether an integration enhances the functionality of a particular r | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`pinoIntegration`](./pino) | | ✓ | | | diff --git a/platform-includes/configuration/integrations/javascript.sveltekit.mdx b/platform-includes/configuration/integrations/javascript.sveltekit.mdx index 031724444dab9..83ec4a187a238 100644 --- a/platform-includes/configuration/integrations/javascript.sveltekit.mdx +++ b/platform-includes/configuration/integrations/javascript.sveltekit.mdx @@ -93,5 +93,6 @@ Depending on whether an integration enhances the functionality of a particular r | [`mistralAIIntegration`](./mistral) | ✓ | | ✓ | | | [`groqIntegration`](./groq) | ✓ | | ✓ | | | [`togetherAIIntegration`](./together-ai) | ✓ | | ✓ | | +| [`typesafeIntegration`](./typesafe) | ✓ | | ✓ | | | [`zodErrorsIntegration`](./zodErrors) | | | | ✓ | | [`pinoIntegration`](./pino) | | ✓ | | |