Skip to content

ci(e2e): benchmark build reuse and worker count - #45

Merged
msrivas-7 merged 5 commits into
mainfrom
dev/ci-build-cache-workers
Aug 31, 2026
Merged

ci(e2e): benchmark build reuse and worker count#45
msrivas-7 merged 5 commits into
mainfrom
dev/ci-build-cache-workers

Conversation

@msrivas-7

@msrivas-7 msrivas-7 commented Aug 31, 2026

Copy link
Copy Markdown
Owner

What this PR does

  • Keeps the measured 16-shard blocking suite and 20-shard ceiling unchanged.
  • Builds immutable backend, runner, and frontend images once, then reuses their exact digests across the blocking browser jobs.
  • Keeps two Playwright workers per shard after independently measuring 2, 3, and 4 workers.
  • Requires every candidate shard to pass without retries and applies the existing 20-second / 5% improvement threshold.
  • Deletes successful runs' exact temporary images, while a daily retention sweep preserves one fallback and removes versions older than 48 hours.

Benchmark safeguards

The three immutable images build in parallel, and their complete build/publish critical path is charged to the reuse candidate. Reuse cannot win merely because its shard stage is faster. The benchmark remains at 16 shards per stage, while 20 is the hard ceiling so the normal PR's other jobs retain headroom under the account-wide concurrency limit.

CI isolation repair

The first ordinary run proved all Docker stacks shared one anonymous IP counter and collectively crossed the real 100-run daily cap. Each lane, shard, attempt, and benchmark stage now gets a deterministic reserved synthetic IPv6 address through the existing trusted Vite proxy. The quota remains enforced; unrelated jobs no longer consume one another’s allowance.

The worker benchmark also exposed a teardown-only socket failure after product assertions had passed. Session cleanup now treats transport loss during its best-effort drain consistently with the existing explicit cleanup path, while HTTP failures and product assertions remain unchanged.

Product journey impact

None. These are CI-only identities and measurements; production/local browser behavior, the complete Chromium suite, and Firefox/WebKit coverage remain unchanged.

Validation

  • 58 workflow, release, harness, and benchmark contract tests passed before the PR opened.
  • Focused benchmark/capacity/permission suite: 25 passed.
  • Anonymous identity/workflow contract suite: 21 passed.
  • Image-preparation accounting suite: 8 passed.
  • Frontend typecheck and production build passed.
  • A live Vite proxy probe preserved the exact synthetic X-Forwarded-For identity.
  • Same-commit benchmark run 33397529873: local build 369s; prebuilt including 24s preparation 338s; 31s / 8.4% gain.
  • Two workers passed all 16 shards; three workers failed two shards and was rejected; four workers was skipped.
  • Actionlint 1.7.12 and Docker Compose validation passed.
  • E2E fixture typecheck passed after the teardown transport repair.
  • All 11 workflow/release contract suites, five retention cases, and the 439-test/16-shard capacity guard pass.
  • Final adoption run 33400350423 passed all 16 Chromium shards, Firefox, WebKit, the advisory lane, three image preparations, shadow evidence, and exact-tag cleanup.

Rollout

The measured winner is adopted: 16 shards, two workers, and build-once image reuse. Twenty remains only the account/topology ceiling. Failed runs retain their images for isolated reruns; successful runs clean immediately and the scheduled sweep bounds abandoned versions.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-31T14:17:13.306805Z f3ee51e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 76fc7f2e41

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/e2e-runtime-benchmark.mjs Outdated
@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://gentle-flower-093ba7e0f-45.eastus2.7.azurestaticapps.net

@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://gentle-flower-093ba7e0f-45.eastus2.7.azurestaticapps.net

@msrivas-7 msrivas-7 added the ci-runtime-benchmark Run the controlled Docker reuse and Playwright worker benchmark label Aug 31, 2026
@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://gentle-flower-093ba7e0f-45.eastus2.7.azurestaticapps.net

@msrivas-7 msrivas-7 added ci-runtime-benchmark Run the controlled Docker reuse and Playwright worker benchmark and removed ci-runtime-benchmark Run the controlled Docker reuse and Playwright worker benchmark labels Aug 31, 2026
@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://gentle-flower-093ba7e0f-45.eastus2.7.azurestaticapps.net

@msrivas-7 msrivas-7 removed the ci-runtime-benchmark Run the controlled Docker reuse and Playwright worker benchmark label Aug 31, 2026
@msrivas-7

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: f3ee51e3cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@msrivas-7
msrivas-7 merged commit 4edcd73 into main Aug 31, 2026
44 checks passed
@msrivas-7
msrivas-7 deleted the dev/ci-build-cache-workers branch August 31, 2026 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant