Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
size-limit report 📦
|
mydea
added this pull request to stack #24458
September 17, 2026 11:50
mydea
force-pushed
the
feat/gen-ai-libraries-e2e
branch
from
September 17, 2026 11:59
0832515 to
69679f1
Compare
mydea
force-pushed
the
feat/gen-ai-libraries-e2e
branch
from
September 17, 2026 12:21
2771d43 to
48619f0
Compare
mydea
marked this pull request as ready for review
September 17, 2026 12:51
mydea
force-pushed
the
feat/gen-ai-libraries-e2e
branch
from
September 18, 2026 07:25
478faed to
e771719
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e771719. Configure here.
Base automatically changed from
feat/groq-together-ai-integrations
to
develop
September 18, 2026 08:15
mydea
requested review from
andreiborza,
isaacs and
s1gr1d
and removed request for
a team
September 18, 2026 08:15
Adds an optional `gen-ai-libraries` e2e app that drives every instrumented gen-AI
*library* — OpenAI, Anthropic, Mistral, Together and the Vercel AI SDK — against
a real model through the single `E2E_OPENROUTER_API_KEY`, with a chat query and a
forced tool call each. Shared, framework-agnostic handlers (`src/libraries.ts`)
run in two runtimes selected by a `RUNTIME` variant (following `hono-4`):
- node - express + runtime channel injection (`--import`)
- cloudflare - a worker built with `vite` + `@sentry/cloudflare/vite`, on workerd
Every call uses `openai/gpt-4o-mini`: OpenRouter serves an OpenAI-compatible
`/api/v1/chat/completions` and an Anthropic-compatible `/api/v1/messages`
("Anthropic skin"), so all five SDKs reach it; the model is incidental, the SDK
code path is what is under test.
Folds in and removes `node-mistral` — Mistral is now one of the covered libraries.
Google GenAI and Groq are omitted: OpenRouter serves neither the native Gemini
`generateContent` format nor Groq's `/openai/v1` request path. Groq shares
Together's OpenAI-compatible instrumentation code path (exercised here) and is
covered by the node-integration-tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scope every test to its own request's trace with `collectStreamedSpans` and a provider-aware span match, so a late-arriving span from another request (the four direct SDKs all report `gen_ai.chat`) can no longer satisfy the wait and flake the suite. Assert tool-call capture concretely instead of scanning span JSON for a substring that also appears in the prompt: the direct SDKs must carry `gen_ai.response.tool_calls` with the tool, and the Vercel AI case — split into its own test rather than hidden behind a conditional — must emit a `gen_ai.execute_tool` span whose `gen_ai.tool.name` is the tool. The prompt no longer names the tool, so the string only appears in a real tool call. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mydea
force-pushed
the
feat/gen-ai-libraries-e2e
branch
from
September 18, 2026 08:15
e771719 to
4b6da1c
Compare
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
s1gr1d
approved these changes
Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #24454.
Adds an optional
gen-ai-librariese2e app that drives every gen-AI library we instrument — OpenAI, Anthropic, Mistral, Together and the Vercel AI SDK — against a real model through the singleE2E_OPENROUTER_API_KEYthe other AI e2e apps use, with one chat-query route and one forced-tool-call route each.node-mistralis folded in and removed; Mistral is now one of the covered libraries.Structure
Shared, framework-agnostic handlers in
src/libraries.tsare wired into two runtime entries, selected by aRUNTIMEvariant (tests/constants.ts), so both instrumentation paths get real-call coverage from one app:src/entry.node.ts, an express app with runtime channel injection preloaded vianode --import ./src/instrument.node.ts.optionalVariants,RUNTIME=cloudflare) —src/entry.cloudflare.ts, built withvite+@sentry/cloudflare/vite(build-time channel injection) and run on workerd viawrangler devagainst the build's generated config (dist/gen_ai_libraries/wrangler.json).playwright.config.tspicks the start command and port perRUNTIME; the tests (tests/chat.test.ts,tests/tools.test.ts) are runtime-agnostic — they hitbaseURLand assert the streamed gen_ai spans, parametrized over the library list intests/utils.ts.Why these five
Every call uses
openai/gpt-4o-mini. OpenRouter serves an OpenAI-compatible/api/v1/chat/completionsand an Anthropic-compatible/api/v1/messages("Anthropic skin"), so all five SDKs reach it (Together/Mistral via lenient OpenAI-shape parsing, the Vercel AI SDK via@openrouter/ai-sdk-provider). The model is incidental; each SDK's own request/response code path is what's under test.Two instrumented libraries can't reach OpenRouter and are omitted with a code comment: Google GenAI (native Gemini
generateContentformat isn't served) and Groq (its SDK hardcodes a/openai/v1request path OpenRouter doesn't expose). Groq shares Together's OpenAI-compatible instrumentation code path — exercised here — and is covered by the node-integration-tests in the base PR.Notes
together_ai,anthropic,mistralai,openai, plus the Vercel AI SDK'sgen_ai.generate_content), including recorded tool calls. The Cloudflare variant runs the same suite in CI.node-mistraldrops its Mistral-specific stream-drain edge cases (tee/pipe, dataloader co-instrumentation) in favour of the uniform query+tool-call coverage across libraries.optional, so it only runs when the OpenRouter key is present.