Skip to content

perf(ci): Benchmark browser integration tests on ARM - #26434

Closed
nicohrubec wants to merge 3 commits into
developfrom
feat/benchmark-browser-playwright-arm
Closed

nicohrubec wants to merge 3 commits into
developfrom
feat/benchmark-browser-playwright-arm

Conversation

@nicohrubec

@nicohrubec nicohrubec commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

What

Benchmark browser integration tests on ARM, preserving browsers, shards and six workers. Current x86 runners have eight cores; ARM has four. Architecture-separated browser caches enable a warm comparison.

Why

Keep the current x86 runners for now. Warm ARM was slower at the median and failed 10/35 browser jobs.

Setup Chromium median test / job* Chromium longest job* Firefox test / job Browser jobs passed
x86 baseline 39.5 / 86s 177s 17 / 64s 35/35
First ARM run 56 / 117s 203s 22 / 65s 25/35
Warm ARM run 56.5 / 104s 152s 23 / 67s 25/35

*The same 24 Chromium jobs that passed in all three runs. Complete-job time includes setup/cleanup. Warm ARM browser-cache restoration was verified.

ARM consistently failed eight Chromium Replay jobs on click-coordinate snapshots. Both WebKit jobs could not launch: missing libgtk-4.so.1, libgraphene-1.0.so.0, libsecret-1.so.0, and libx264.so. All 9,926 case identities were selected; assertions were unchanged. This draft retains the benchmark experiment for inspection.

Co-Authored-By: OpenAI Codex <codex@openai.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

size-limit report 📦

Compared: f944acd (PR test merge using develop at c7b765f)

Against: c7b765f (develop) · base run #66318

✅ Baseline matches: this diff contains only this PR's changes.

Path Size % Change Change
@sentry/browser 30.1 kB - -
@sentry/browser - with treeshaking flags 28.3 kB - -
@sentry/browser - with treeshaking flags tracing without tracing 28.2 kB - -
@sentry/browser (incl. Tracing) 52.28 kB - -
@sentry/browser (incl. Tracing + Span Streaming) 52.29 kB - -
@sentry/browser (incl. Tracing, Profiling) 55.25 kB - -
@sentry/browser (incl. Tracing, Replay) 92.82 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 81.6 kB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 97.21 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 110.6 kB - -
@sentry/browser (incl. Feedback) 47.73 kB - -
@sentry/browser (incl. sendFeedback) 30.69 kB - -
@sentry/browser (incl. FeedbackAsync) 35.69 kB - -
@sentry/browser (incl. Metrics) 31.43 kB - -
@sentry/browser (incl. Logs) 31.71 kB - -
@sentry/browser (incl. Metrics & Logs) 32.37 kB - -
@sentry/react 32.11 kB - -
@sentry/react (incl. Tracing) 54.95 kB - -
@sentry/vue 38.3 kB - -
@sentry/vue (incl. Tracing) 55.24 kB - -
@sentry/svelte 30.12 kB - -
@sentry/remix (Remix 3 client bundle) 58.06 kB - -
CDN Bundle 31.87 kB - -
CDN Bundle (incl. Tracing) 52.7 kB - -
CDN Bundle (incl. Logs, Metrics) 34.67 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 55.26 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 75.7 kB - -
CDN Bundle (incl. Tracing, Replay) 90.72 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 93.33 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 96.09 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 98.69 kB - -
CDN Bundle - uncompressed 92.98 kB - -
CDN Bundle (incl. Tracing) - uncompressed 155.45 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 101.33 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 163.08 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 231.44 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 275.61 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 283.24 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 287.4 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 295.02 kB - -
@sentry/nextjs (client) 58.57 kB - -
@sentry/sveltekit (client) 53.57 kB - -
@sentry/core/server 44.8 kB - -
@sentry/core/browser 14.4 kB - -
@sentry/node 151.36 kB -0.01% -6 B 🔽
@sentry/node/import (ESM hook with diagnostics-channel injection) 83.3 kB - -
@sentry/node - without tracing 88.56 kB -0.01% -5 B 🔽
@sentry/node - without channel injection 140.3 kB -0.01% -5 B 🔽
@sentry/aws-serverless 98.93 kB -0.01% -7 B 🔽
@sentry/cloudflare (withSentry) - minified 206.47 kB - -
@sentry/cloudflare (withSentry) 479.22 kB - -
@sentry/nextjs/cloudflare (withSentry) - minified 231.83 kB - -

View base workflow run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e42696a. Configure here.


env:
HEAD_COMMIT: ${{ github.event.inputs.commit || github.sha }}
HEAD_COMMIT: ${{ github.event.inputs.commit || github.event.pull_request.head.sha || github.sha }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR CI checks out unmerged head

High Severity

HEAD_COMMIT now prefers github.event.pull_request.head.sha over github.sha, so PR jobs check out the branch tip instead of GitHub's merge commit. job_get_metadata still receives github.sha, and Nx affected compares that head against pull_request.base.sha from a pair of depth-1 checkouts. Those two commits usually do not share fetched ancestry, so git merge-base can fail and take down job_build for multi-commit or out-of-date PRs. Even when it works, CI no longer tests the merged result.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit e42696a. Configure here.

uses: actions/checkout@v7
with:
ref: ${{ env.HEAD_COMMIT }}
ref: ${{ github.sha }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bundler tests checkout mismatches build

Medium Severity

job_bundler_plugin_integration_tests now checks out github.sha while job_build and the rest of the workflow use HEAD_COMMIT (pull_request.head.sha on PRs). This job still restores job_build's node_modules cache and build-output artifacts, so on a PR it runs merge-commit sources against binaries built from the PR tip. Those trees diverge whenever the PR is behind develop.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit e42696a. Configure here.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Optional E2E tests

Some optional checks were not successful. They do not block the merge. 2 not successful, 61 successful. See the run.

Check Status
❌ CI: Optional E2E Tests / E2E cloudflare-workers-send-to-sentry Test (optional) Failing after 3m Details
❌ CI: Optional E2E Tests / E2E nextjs-16-cacheComponents (cloudflare) Test (optional) Failing after 10m Details
Successful checks
Check Status
✅ CI: Optional E2E Tests / E2E gen-ai-libraries Test (optional) Successful in 47s Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-flue-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E debug-id-sourcemaps Test (optional) Successful in 31s Details
✅ CI: Optional E2E Tests / E2E cloudflare-think Test (optional) Successful in 58s Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (latest) Test (optional) Successful in 54s Details
✅ CI: Optional E2E Tests / E2E astro-7-static Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-eve (orchestrion) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (bun) Test (optional) Successful in 51s Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v2 Test (optional) Successful in 34s Details
✅ CI: Optional E2E Tests / E2E nextjs-14 (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (cloudflare) Test (optional) Successful in 41s Details
✅ CI: Optional E2E Tests / E2E cloudflare-pi-durable Test (optional) Successful in 52s Details
✅ CI: Optional E2E Tests / E2E cloudflare-pi-durable (latest) Test (optional) Successful in 48s Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-sourcemaps Test (optional) Successful in 32s Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (latest) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E node-openai-decisions (latest) Test (optional) Successful in 27s Details
✅ CI: Optional E2E Tests / E2E cloudflare-workers-without-tracing-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-eve Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E astro-7 Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-eve (content off, latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-eve (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (latest) Test (optional) Successful in 50s Details
✅ CI: Optional E2E Tests / E2E cloudflare-mastra Test (optional) Successful in 49s Details
✅ CI: Optional E2E Tests / E2E node-flue (latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nuxt-5 (development) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-flue Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-mastra (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-16-tunnel (cloudflare) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-think (ai v6) Test (optional) Successful in 55s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (client source maps kept) Test (optional) Successful in 34s Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (turbo) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext, bun) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-mastra Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-cloudflare Test (optional) Successful in 31s Details
✅ CI: Optional E2E Tests / E2E cloudflare-agent Test (optional) Successful in 43s Details
✅ CI: Optional E2E Tests / E2E node-fastify-6 Test (optional) Successful in 42s Details
✅ CI: Optional E2E Tests / E2E node-pi-durable (latest) Test (optional) Successful in 35s Details
✅ CI: Optional E2E Tests / E2E react-router-7-framework-instrumentation Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E deno (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-pi-durable Test (optional) Successful in 36s Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nextjs-orpc Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-openai-decisions Test (optional) Successful in 31s Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (cloudflare, latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E supabase-nextjs Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E vue-3 (no Options API) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v1 Test (optional) Successful in 35s Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry-static Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E solidstart-2 Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E solidstart-2-tunnel Test (optional) Successful in 42s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (static / nuxt generate) Test (optional) Successful in 31s Details
✅ CI: Optional E2E Tests / E2E nuxt-5 (kv-cache) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nestjs-12 (deno) Test (optional) Successful in 56s Details
✅ CI: Optional E2E Tests / E2E nuxt-5 Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-tunnel Test (optional) Successful in 39s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-cloudflare Test (optional) Successful in 32s Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (cloudflare) Test (optional) Successful in 45s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps Test (optional) Successful in 41s Details
✅ CI: Optional E2E Tests / E2E react-send-to-sentry Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react (latest) Test (optional) Successful in 2m Details

nicohrubec and others added 2 commits November 16, 2026 20:30
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
@nicohrubec nicohrubec closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant