Skip to content

test(ci): Benchmark batching integration scenarios - #26381

Draft
nicohrubec wants to merge 27 commits into
developfrom
feat/benchmark-batch-node-scenarios
Draft

nicohrubec wants to merge 27 commits into
developfrom
feat/benchmark-batch-node-scenarios

Conversation

@nicohrubec

@nicohrubec nicohrubec commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

No need to review yet this is just an experiment.

Summary

Reduce repeated SDK/app startup by running compatible integration scenarios in shared processes. Keep SDK configurations, module formats and runtime coverage separate.

Measured result Before After Improvement
Slowest required Node/Bun/shared-Deno integration test step 2m46s 2m21s 25s (15%)
Test-step time summed across those 36 parallel jobs 63m59s 56m38s 7m21s (11%) less runner time

The first row is the integration bottleneck improvement. The second sums parallel jobs; it is not minutes off the PR wait. These are full-run observations, so they include runner variation. Baseline: 37bc9a0; final: 9eb2691. Setup, queues and optional Node 27 are excluded from the table.

Latest time to required checks: 5m37s from workflow creation (5m35s excluding initial waiting). Node-compatible integration jobs finished at 5m01s; Nuxt E2E finished last at 5m33s. This does not establish a matching end-to-end CI speedup: the original baseline failed elsewhere and had an unrelated cache outlier. All final required checks and controlled suite comparisons passed. PR remains draft.

The experiment history and suite-level measurements follow.

What

Batch compatible Express, HTTP and continueTrace integration cases within their existing test files, stacked on #26379. Keep module formats and sampling configurations isolated.

Why

Avoid repeated child-process/SDK initialization while retaining the unique request and header cases and the existing file-level sharding.

Complete-suite test times, averaged over two before/after measurements on the same ARM runner (warm-up, then ABBA order):

Suite Node 20 Node 24 Deno Bun 1.4
express/tracing 56.0 → 41.7s 49.3 → 38.0s 52.4 → 39.6s Excluded
tracing/httpIntegration 24.8 → 22.6s 21.9 → 20.1s 16.0 → 13.9s Excluded
tracing/continueTrace 11.2 → 2.9s 7.8 → 2.0s 8.8 → 2.3s 9.8 → 2.7s
Total saved 24.8s 18.9s 21.4s 7.2s

Times sum test-case durations including unchanged cases and both module formats. Bun sums its Node-SDK and Bun-SDK projects.

In the full CI runs, shard 5 (which contains continueTrace) also finished its test step sooner:

Node Before After Saved
20 108s 98s 10s
22 136s 104s 32s
24 101s 89s 12s
26 96s 89s 7s
27 alpha 90s 86s 4s

These exclude setup/cache restoration and are single-run observations; differences include other tests and runner variation, not just continueTrace.

The corrected full run passed all required checks, but this pair of full runs does not establish a critical-path improvement. The baseline had an unrelated Deno E2E startup timeout; its cache-restoration outlier is excluded from the job-time comparison. The suite profiler and CI-trigger comment are temporary benchmark scaffolding.

Additional body-test batching

Measured independently against the previous successful batching revision, with the same warm-up and two measurements per version. These are only the newly changed body-test groups, including ESM and CJS:

Test group Node 20 Node 24 Deno
Express content types / ignored bodies 11.0 → 6.3s 7.5 → 5.5s 10.3 → 6.1s
Request-size limits 26.7 → 17.3s 18.2 → 13.7s 24.8 → 16.4s
Additional work saved 14.1s 6.5s 12.6s

Both groups stay on shard 6. In the full runs, its test step changed as follows (single-run observations, including other tests and runner variation):

Node Before After Saved
20 167s 151s 16s
22 179s 165s 14s
24 144s 136s 8s
26 142s 123s 19s
27 alpha 139s 122s 17s

Trace-header server reuse

A separate experiment keeps all 15 header tests and assertions, sharing one CJS server instead of starting 15. Outgoing requests to the placeholder host are aborted after reading their headers.

Runtime Before After Saved
Node 20 5.8s 0.5s 5.3s (92%)
Node 24 4.0s 0.3s 3.7s (91%)
Deno 4.8s 0.4s 4.4s (92%)

Again, two measurements per version on the same runner. All targeted comparisons passed, and the trace-header trial passed every required check. Dependency-install reuse is deferred to a separate PR and is not included here.

Additional AI and Hono batching

Complete-suite command times, including fixture setup; mean of two measurements per version on the same ARM runner after a warm-up:

Suite Node 20: before → after Node 24: before → after
Vercel AI v6/v7 83.0s → 59.2s 68.1s → 47.3s
OpenAI 40.7s → 32.7s 35.1s → 29.5s
LangChain v1 54.8s → 38.1s 54.6s → 37.7s
Hono 37.0s → 20.4s 31.2s → 17.0s

Existing named tests and assertions are retained. Compatible AI scenarios use separate isolation scopes in one process per SDK configuration/module format. The selected AI groups use 36 → 8 processes. Hono shares one server for nine non-error tests (18 → 2 processes across ESM/CJS); error scenarios remain separate. Repeated identical route requests share their captured trace. The Anthropic experiment was dropped because it was slower.

All controlled comparisons passed with unchanged test counts and no new skips. These are suite savings, not overall CI savings. Dependency-install aliases and duration-balanced sharding are not included.

Further independent comparisons

Complete-suite command seconds, including fixture setup; means of two measurements per version after warm-up on the same runner. Expanded OpenAI/Vercel rows measure the additional batching beyond the earlier table.

Suite / additional batch Node 20 before → after Node 24 before → after
Filesystem 25.0s → 10.5s 21.1s → 8.9s
Express happy paths 38.0s → 27.1s 35.6s → 25.8s
Hono SDK 15.7s → 9.7s 12.7s → 7.8s
Hapi 20.2s → 15.6s 17.3s → 13.4s
Koa 14.7s → 10.5s 12.7s → 9.1s
Fastify 11.6s → 10.5s 9.9s → 8.9s
Anthropic 31.8s → 24.8s 26.2s → 20.9s
Anthropic 0.129 16.6s → 8.3s 13.8s → 7.0s
Google GenAI v2 9.9s → 5.3s 7.4s → 4.3s
OpenAI v7 — 17.1s → 14.7s
Mistral 30.5s → 27.8s 21.2s → 19.3s
Mistral v2 9.6s → 5.6s 10.0s → 5.7s
Mastra — 95.5s → 92.8s
OpenAI: more conversation/root cases 33.3s → 26.3s 29.3s → 24.0s
Vercel AI: agent/concurrency cases 54.5s → 43.3s 49.8s → 39.2s
Legacy LangChain 17.7s → 7.4s 13.7s → 5.7s
Express span renaming 4.4s → 2.4s 3.8s → 2.2s

Mastra requires Node ≥22; its Node 22 comparison was 99.5s → 95.1s. OpenAI v7 is also gated off on Node 20. All completed comparisons retain test counts and assertions; runtime skips are unchanged.

Final bottleneck experiments

Same-host complete-suite command timings, including setup; mean of two measurements per version after warm-up:

Suite Runtime Before → after Saved
Google GenAI v1 Node 20 26.7s → 12.2s 14.5s
Google GenAI v1 Node 24 21.0s → 10.0s 11.0s
pi-durable Node 22 21.9s → 19.1s 2.7s
pi-durable Node 24 16.8s → 14.8s 2.0s

Google GenAI keeps each SDK configuration in a separate process. pi-durable reuses identical tool/lifecycle executions across their existing assertion views, preserving the ordered error/sentinel checks. Existing runtime guards, tests and assertions are retained. All four comparisons and all 42 Node/Bun/Deno integration jobs passed; the final required gate passed. No duration-balanced sharding or dependency-install aliases were added.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

size-limit report 📦

Compared: 7a07ce4 (PR test merge using develop at a34b36f)

Against: 005db14 (develop) · base run #66164

⚠️ Baseline mismatch: 13 develop commit(s) in the PR build are missing from the baseline. Their size changes also affect this comparison. Compare commits. Re-run once develop's baseline build for a34b36f finishes.

Path Size % Change Change
@sentry/browser 30.1 kB - -
@sentry/browser - with treeshaking flags 28.3 kB - -
@sentry/browser - with treeshaking flags tracing without tracing 28.2 kB - -
@sentry/browser (incl. Tracing) 52.28 kB - -
@sentry/browser (incl. Tracing + Span Streaming) 52.29 kB - -
@sentry/browser (incl. Tracing, Profiling) 55.25 kB - -
@sentry/browser (incl. Tracing, Replay) 92.82 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 81.6 kB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 97.21 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 110.6 kB - -
@sentry/browser (incl. Feedback) 47.73 kB - -
@sentry/browser (incl. sendFeedback) 30.69 kB - -
@sentry/browser (incl. FeedbackAsync) 35.69 kB - -
@sentry/browser (incl. Metrics) 31.43 kB - -
@sentry/browser (incl. Logs) 31.71 kB - -
@sentry/browser (incl. Metrics & Logs) 32.37 kB - -
@sentry/react 32.11 kB - -
@sentry/react (incl. Tracing) 54.95 kB - -
@sentry/vue 38.3 kB - -
@sentry/vue (incl. Tracing) 55.24 kB - -
@sentry/svelte 30.12 kB - -
@sentry/remix (Remix 3 client bundle) 58.06 kB - -
CDN Bundle 31.87 kB - -
CDN Bundle (incl. Tracing) 52.7 kB - -
CDN Bundle (incl. Logs, Metrics) 34.67 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 55.26 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 75.7 kB - -
CDN Bundle (incl. Tracing, Replay) 90.72 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 93.33 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 96.09 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 98.69 kB - -
CDN Bundle - uncompressed 92.98 kB - -
CDN Bundle (incl. Tracing) - uncompressed 155.45 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 101.33 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 163.08 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 231.44 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 275.61 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 283.24 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 287.4 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 295.02 kB - -
@sentry/nextjs (client) 58.37 kB - -
@sentry/sveltekit (client) 53.48 kB - -
@sentry/core/server 44.73 kB - -
@sentry/core/browser 14.3 kB - -
@sentry/node 150.46 kB -0.01% -8 B 🔽
@sentry/node/import (ESM hook with diagnostics-channel injection) 83.3 kB - -
@sentry/node - without tracing 88.55 kB -0.01% -5 B 🔽
@sentry/node - without channel injection 139.42 kB -0.01% -7 B 🔽
@sentry/aws-serverless 98.46 kB -0.01% -4 B 🔽
@sentry/cloudflare (withSentry) - minified 205.19 kB - -
@sentry/cloudflare (withSentry) 475.96 kB - -
@sentry/nextjs/cloudflare (withSentry) - minified 230.5 kB - -

View base workflow run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

runner.makeRequest('get', '/test', { headers: { 'X-Forwarded-For': '203.0.113.7, 10.0.0.1' } });
await runner.makeRequest('get', '/test?a=1&b=2#hash');
await runner.makeRequest('post', '/test?a=1&b=2#hash', { data: 'test body' });
await runner.makeRequest('get', '/test', { headers: { 'X-Forwarded-For': '203.0.113.7, 10.0.0.1' } });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Batched requests assume envelope order

Medium Severity

The GET, POST, and forwarded cases now share one child process and three ordered transaction expects, without .unordered() or unique matchers. makeRequest only waits for the HTTP response, so the three envelopes can arrive in any order—especially on Bun and Deno—and the GET and forwarded cases are both GET /test. That makes this batched test likely to flake, unlike the other suites in this PR that match by unique attributes.

Fix in Cursor Fix in Web

Triggered by project rule: PR Review Guidelines for Cursor Bot

Reviewed by Cursor Bugbot for commit 37918b3. Configure here.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Optional E2E tests

Some optional checks were not successful. They do not block the merge. 4 not successful, 55 successful. See the run.

Check Status
❌ CI: Optional E2E Tests / E2E node-eve (latest) Test (optional) Failing after 2m Details
❌ CI: Optional E2E Tests / E2E nextjs-16-cacheComponents (cloudflare) Test (optional) Failing after 11m Details
❌ CI: Optional E2E Tests / E2E node-eve (orchestrion) Test (optional) Failing after 2m Details
❌ CI: Optional E2E Tests / E2E node-eve Test (optional) Failing after 2m Details
Successful checks
Check Status
✅ CI: Optional E2E Tests / E2E cloudflare-mastra Test (optional) Successful in 46s Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext, bun) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (cloudflare) Test (optional) Successful in 43s Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (turbo) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E cloudflare-flue-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-mastra Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E astro-7-static Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v1 Test (optional) Successful in 32s Details
✅ CI: Optional E2E Tests / E2E node-flue (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-agent Test (optional) Successful in 45s Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v2 Test (optional) Successful in 35s Details
✅ CI: Optional E2E Tests / E2E cloudflare-pi-durable Test (optional) Successful in 46s Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (latest) Test (optional) Successful in 46s Details
✅ CI: Optional E2E Tests / E2E cloudflare-pi-durable (latest) Test (optional) Successful in 45s Details
✅ CI: Optional E2E Tests / E2E cloudflare-think (ai v6) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-pi-durable Test (optional) Successful in 36s Details
✅ CI: Optional E2E Tests / E2E node-mastra (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5 Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E cloudflare-workers-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nextjs-16-tunnel (cloudflare) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (latest) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-think Test (optional) Successful in 57s Details
✅ CI: Optional E2E Tests / E2E astro-7 Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-flue Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry-static Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps Test (optional) Successful in 32s Details
✅ CI: Optional E2E Tests / E2E deno (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (cloudflare, latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (cloudflare) Test (optional) Successful in 33s Details
✅ CI: Optional E2E Tests / E2E nextjs-orpc Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (deno) Test (optional) Successful in 57s Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (bun) Test (optional) Successful in 51s Details
✅ CI: Optional E2E Tests / E2E nuxt-5 (kv-cache) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (client source maps kept) Test (optional) Successful in 33s Details
✅ CI: Optional E2E Tests / E2E debug-id-sourcemaps Test (optional) Successful in 43s Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-cloudflare Test (optional) Successful in 31s Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-sourcemaps Test (optional) Successful in 30s Details
✅ CI: Optional E2E Tests / E2E nextjs-14 (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries Test (optional) Successful in 47s Details
✅ CI: Optional E2E Tests / E2E node-openai-decisions Test (optional) Successful in 26s Details
✅ CI: Optional E2E Tests / E2E node-openai-decisions (latest) Test (optional) Successful in 25s Details
✅ CI: Optional E2E Tests / E2E solidstart-2 Test (optional) Successful in 54s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-cloudflare Test (optional) Successful in 27s Details
✅ CI: Optional E2E Tests / E2E node-fastify-6 Test (optional) Successful in 36s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (static / nuxt generate) Test (optional) Successful in 36s Details
✅ CI: Optional E2E Tests / E2E nuxt-5 (tunnel-dev) Test (optional) Successful in 57s Details
✅ CI: Optional E2E Tests / E2E supabase-nextjs Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E react-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E vue-3 (no Options API) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-pi-durable (latest) Test (optional) Successful in 40s Details
✅ CI: Optional E2E Tests / E2E react-router-7-framework-instrumentation Test (optional) Successful in 2m Details

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@nicohrubec
nicohrubec added this pull request to stack #26383 November 16, 2026 08:08
[
"import * as Sentry from '@sentry/node';",
...imports,
`async function run() { ${runs.join('\n')} await Sentry.flush(2000); }`,
nicohrubec and others added 23 commits November 16, 2026 11:21
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
nicohrubec and others added 4 commits November 16, 2026 11:22
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
@nicohrubec
nicohrubec force-pushed the feat/benchmark-batch-node-scenarios branch from 9eb2691 to bd05f5c Compare November 16, 2026 10:24
@nicohrubec
nicohrubec force-pushed the feat/benchmark-inline-bundles branch from 8ae2d2f to 2e35c86 Compare November 16, 2026 10:24
@nicohrubec
nicohrubec removed this pull request from stack #26383 November 16, 2026 10:28

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit bd05f5c. Configure here.

});
},
})
.start();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Batched cases use for-loops

Low Severity

Several suites that used test.each now assert multiple cases inside one test with a for loop (error routes, array routes, request bodies, body-size limits, and continueTrace variants). Shared-process batching can stay on a lazy helper like getSpans; looping inside a single test hides which case failed. The testing conventions ask for test.each instead of loops over scenarios in one test.

Additional Locations (2)
Fix in Cursor Fix in Web

Triggered by project rule: PR Review Guidelines for Cursor Bot

Reviewed by Cursor Bugbot for commit bd05f5c. Configure here.

Base automatically changed from feat/benchmark-inline-bundles to develop November 16, 2026 10:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants