Repository navigation
test(ci): Benchmark batching integration scenarios - #26381
nicohrubec wants to merge 27 commits into
Conversation
size-limit report 📦Compared: 7a07ce4 (PR test merge using develop at a34b36f) Against: 005db14 (develop) · base run #66164
|
| runner.makeRequest('get', '/test', { headers: { 'X-Forwarded-For': '203.0.113.7, 10.0.0.1' } }); | ||
| await runner.makeRequest('get', '/test?a=1&b=2#hash'); | ||
| await runner.makeRequest('post', '/test?a=1&b=2#hash', { data: 'test body' }); | ||
| await runner.makeRequest('get', '/test', { headers: { 'X-Forwarded-For': '203.0.113.7, 10.0.0.1' } }); |
There was a problem hiding this comment.
Batched requests assume envelope order
Medium Severity
The GET, POST, and forwarded cases now share one child process and three ordered transaction expects, without .unordered() or unique matchers. makeRequest only waits for the HTTP response, so the three envelopes can arrive in any order—especially on Bun and Deno—and the GET and forwarded cases are both GET /test. That makes this batched test likely to flake, unlike the other suites in this PR that match by unique attributes.
Triggered by project rule: PR Review Guidelines for Cursor Bot
Reviewed by Cursor Bugbot for commit 37918b3. Configure here.
Optional E2E testsSome optional checks were not successful. They do not block the merge. 4 not successful, 55 successful. See the run.
Successful checks
|
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
9eb2691 to
bd05f5c
Compare
8ae2d2f to
2e35c86
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit bd05f5c. Configure here.
| }); | ||
| }, | ||
| }) | ||
| .start(); |
There was a problem hiding this comment.
Batched cases use for-loops
Low Severity
Several suites that used test.each now assert multiple cases inside one test with a for loop (error routes, array routes, request bodies, body-size limits, and continueTrace variants). Shared-process batching can stay on a lazy helper like getSpans; looping inside a single test hides which case failed. The testing conventions ask for test.each instead of loops over scenarios in one test.
Additional Locations (2)
Triggered by project rule: PR Review Guidelines for Cursor Bot
Reviewed by Cursor Bugbot for commit bd05f5c. Configure here.


No need to review yet this is just an experiment.
Summary
Reduce repeated SDK/app startup by running compatible integration scenarios in shared processes. Keep SDK configurations, module formats and runtime coverage separate.
The first row is the integration bottleneck improvement. The second sums parallel jobs; it is not minutes off the PR wait. These are full-run observations, so they include runner variation. Baseline: 37bc9a0; final: 9eb2691. Setup, queues and optional Node 27 are excluded from the table.
Latest time to required checks: 5m37s from workflow creation (5m35s excluding initial waiting). Node-compatible integration jobs finished at 5m01s; Nuxt E2E finished last at 5m33s. This does not establish a matching end-to-end CI speedup: the original baseline failed elsewhere and had an unrelated cache outlier. All final required checks and controlled suite comparisons passed. PR remains draft.
The experiment history and suite-level measurements follow.
What
Batch compatible Express, HTTP and
continueTraceintegration cases within their existing test files, stacked on #26379. Keep module formats and sampling configurations isolated.Why
Avoid repeated child-process/SDK initialization while retaining the unique request and header cases and the existing file-level sharding.
Complete-suite test times, averaged over two before/after measurements on the same ARM runner (warm-up, then ABBA order):
express/tracingtracing/httpIntegrationtracing/continueTraceTimes sum test-case durations including unchanged cases and both module formats. Bun sums its Node-SDK and Bun-SDK projects.
In the full CI runs, shard 5 (which contains
continueTrace) also finished its test step sooner:These exclude setup/cache restoration and are single-run observations; differences include other tests and runner variation, not just
continueTrace.The corrected full run passed all required checks, but this pair of full runs does not establish a critical-path improvement. The baseline had an unrelated Deno E2E startup timeout; its cache-restoration outlier is excluded from the job-time comparison. The suite profiler and CI-trigger comment are temporary benchmark scaffolding.
Additional body-test batching
Measured independently against the previous successful batching revision, with the same warm-up and two measurements per version. These are only the newly changed body-test groups, including ESM and CJS:
Both groups stay on shard 6. In the full runs, its test step changed as follows (single-run observations, including other tests and runner variation):
Trace-header server reuse
A separate experiment keeps all 15 header tests and assertions, sharing one CJS server instead of starting 15. Outgoing requests to the placeholder host are aborted after reading their headers.
Again, two measurements per version on the same runner. All targeted comparisons passed, and the trace-header trial passed every required check. Dependency-install reuse is deferred to a separate PR and is not included here.
Additional AI and Hono batching
Complete-suite command times, including fixture setup; mean of two measurements per version on the same ARM runner after a warm-up:
Existing named tests and assertions are retained. Compatible AI scenarios use separate isolation scopes in one process per SDK configuration/module format. The selected AI groups use 36 → 8 processes. Hono shares one server for nine non-error tests (18 → 2 processes across ESM/CJS); error scenarios remain separate. Repeated identical route requests share their captured trace. The Anthropic experiment was dropped because it was slower.
All controlled comparisons passed with unchanged test counts and no new skips. These are suite savings, not overall CI savings. Dependency-install aliases and duration-balanced sharding are not included.
Further independent comparisons
Complete-suite command seconds, including fixture setup; means of two measurements per version after warm-up on the same runner. Expanded OpenAI/Vercel rows measure the additional batching beyond the earlier table.
Mastra requires Node ≥22; its Node 22 comparison was 99.5s → 95.1s. OpenAI v7 is also gated off on Node 20. All completed comparisons retain test counts and assertions; runtime skips are unchanged.
Final bottleneck experiments
Same-host complete-suite command timings, including setup; mean of two measurements per version after warm-up:
Google GenAI keeps each SDK configuration in a separate process. pi-durable reuses identical tool/lifecycle executions across their existing assertion views, preserving the ordered error/sentinel checks. Existing runtime guards, tests and assertions are retained. All four comparisons and all 42 Node/Bun/Deno integration jobs passed; the final required gate passed. No duration-balanced sharding or dependency-install aliases were added.