Skip to content

test(node): Cover Vercel AI v7 cache tokens from the SDK usage object #24358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mydea merged 2 commits into from
Sep 14, 2026
+69 −0
Merged

test(node): Cover Vercel AI v7 cache tokens from the SDK usage object #24358

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d344641
test(node): Cover Vercel AI v7 cache tokens from the SDK usage object
mydea Sep 14, 2026
27897d2
cleanup
mydea Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions dev-packages/node-integration-tests/suites/tracing/vercelai/v6_v7/scenario-cache-tokens.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import * as Sentry from '@sentry/node';
import { generateText } from 'ai';
import { MockLanguageModelV3 } from 'ai/test';

// Calls routed through the Vercel AI Gateway carry a `gateway` key in `providerMetadata` instead of a
// provider name, so cache counts can only be derived from the SDK's normalized usage object.
async function run() {
await Sentry.startSpan({ op: 'function', name: 'main' }, async () => {
await generateText({
experimental_telemetry: { isEnabled: true, recordInputs: true, recordOutputs: true },
model: new MockLanguageModelV3({
doGenerate: async () => ({
finishReason: { unified: 'stop', raw: 'stop' },
usage: {
inputTokens: { total: 120, noCache: 20, cacheRead: 80, cacheWrite: 20 },
outputTokens: { total: 10, noCache: 10, cached: 0 },
totalTokens: { total: 130, noCache: 30, cached: 100 },
},
content: [{ type: 'text', text: 'Cache token span!' }],
warnings: [],
providerMetadata: {
gateway: { routing: {} },
},
}),
}),
messages: [{ role: 'user', content: 'Tell me something' }],
});
});
}

run();
38 changes: 38 additions & 0 deletions dev-packages/node-integration-tests/suites/tracing/vercelai/v6_v7/test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import {
GEN_AI_TOOL_DEFINITIONS,
GEN_AI_TOOL_DESCRIPTION,
GEN_AI_TOOL_NAME,
GEN_AI_USAGE_CACHE_CREATION_INPUT_TOKENS,
GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS,
GEN_AI_USAGE_INPUT_TOKENS,
GEN_AI_USAGE_OUTPUT_TOKENS,
Expand Down Expand Up @@ -811,6 +812,43 @@ describe.each(matrix)('Vercel AI integration (version %s)', (version, vercelAiVe
},
);

createEsmTests(
__dirname,
'scenario-cache-tokens.mjs',
'instrument.mjs',
(createRunner, test) => {
test('reads cache token counts from the SDK usage object', async () => {
await createRunner()
.expect({ transaction: { transaction: 'main' } })
.expect({
span: container => {
const generateContent = container.items.find(
span => span.attributes['sentry.op']?.value === 'gen_ai.generate_content',
)!;
expect(generateContent).toBeDefined();
expect(generateContent.attributes[GEN_AI_USAGE_INPUT_TOKENS]?.value).toBe(120);
expect(generateContent.attributes[GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS]?.value).toBe(80);
expect(generateContent.attributes[GEN_AI_USAGE_CACHE_CREATION_INPUT_TOKENS]?.value).toBe(20);

const invokeAgent = container.items.find(
span => span.attributes['sentry.op']?.value === 'gen_ai.invoke_agent',
)!;
expect(invokeAgent).toBeDefined();
expect(invokeAgent.attributes[GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS]?.value).toBe(80);
expect(invokeAgent.attributes[GEN_AI_USAGE_CACHE_CREATION_INPUT_TOKENS]?.value).toBe(20);
},
})
.start()
.completed();
});

@cursor cursor Bot Sep 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cache token test not skipped on v6

Medium Severity

The new cache-token test is registered for both matrix versions, but v6 cannot derive cache counts from this gateway scenario. Nearby version-specific cases use test.skipIf, so the v6 entry will assert gen_ai.usage.input_tokens.cache_read and cache_creation values that that path never writes.

Fix in Cursor Fix in Web

Triggered by project rule: PR Review Guidelines for Cursor Bot

Reviewed by Cursor Bugbot for commit 58c1aaa. Configure here.

@mydea mydea Sep 14, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems to work just fine

Comment on lines +833 to +843

@sentry sentry Bot Sep 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The test 'reads cache token counts from the SDK usage object' is missing a test.skipIf(version === '6') condition, causing it to fail for the v6 test matrix.
Severity: LOW

Suggested Fix

Wrap the test case 'reads cache token counts from the SDK usage object' with a skip condition to prevent it from running against v6. Use test.skipIf(version === '6') around the test definition, similar to other version-specific tests in the suite.

Prompt for AI Agent 
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location:
dev-packages/node-integration-tests/suites/tracing/vercelai/v6_v7/test.ts#L820-L843

Potential issue: The integration test `'reads cache token counts from the SDK usage
object'` is intended to run for both v6 and v7 of the Vercel AI SDK. However, the test
is missing a required skip condition for v6. The test asserts that cache token counts
are derived from the SDK's usage object, a feature specific to v7. The v6 OTel
integration derives these counts from `providerMetadata` and cannot handle the `gateway`
provider metadata shape used in this test scenario. As a result, the test will
incorrectly execute and fail during CI runs for v6.

Did we get this right? 👍 / 👎 to inform future reviews.

},
{
additionalDependencies: {
ai: vercelAiVersion,
},
},
);

createEsmTests(
__dirname,
'scenario-embeddings.mjs',
Expand Down
Loading