Skip to content

ci: target a four-minute full PR quality loop - #47

Merged
msrivas-7 merged 13 commits into
mainfrom
dev/ci-four-minute-target
Aug 31, 2026
Merged

ci: target a four-minute full PR quality loop#47
msrivas-7 merged 13 commits into
mainfrom
dev/ci-four-minute-target

Conversation

@msrivas-7

@msrivas-7 msrivas-7 commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Goal

Move normal full PR CI toward a 4-minute median / 5-minute p95 without removing tests, weakening browser/security coverage, relying on retries, or requiring paid runners.

Changes

  • Balance all 439 Chromium tests by learned duration instead of test count; every test is assigned exactly once.
  • Learn timing only from clean exhaustive runs, with a tracked cold-start seed and trusted cache.
  • Split the complete 63-test advisory critical lane across isolated zero-retry jobs.
  • Pull immutable backend, runner, and frontend images concurrently.
  • Run duration learning, shadow evidence, and image cleanup in parallel after testing.
  • Remove unused frontend/browser setup from the API-only security suite while preserving every security scenario and tcpdump sentinel.
  • Keep the measured 16-full/2-critical topology; no tests or browser/security lanes are removed or demoted.
  • Make Docker cancellation completion race-safe after the confirmation run exposed a real fast-Stop hang.

User journey change

Before: a very fast Docker kill could occasionally leave Editor stuck on “Stopping this run…”.
Now: Stop always settles to “Run stopped,” restores Run, and the learner can execute again immediately.

Measured results

Workflow Before Final-head clean 1 Final-head clean 2
Full E2E 6m41s 4m30s 4m42s
Security suite 3m58s 2m54s
Platform CI 3m25s

The two clean exact-head E2E samples have a 4m36s observed median, and both finish below five minutes. All 439 Chromium tests, the complete advisory critical lane, Firefox, WebKit, and every security scenario passed without retry dependence. Two samples do not establish a statistical p95; trusted clean history will continue measuring that target.

The first overlapping final-head E2E attempt exposed one transient bootstrap data-load failure and was not counted. Its retry passed, but the workflow remained red by design. The two complete reruns above then passed with zero flaky tests; no timeout, assertion, or retry policy was weakened.

Rejected experiments

  • 20 full / 4 critical: increased aggregate database pressure and produced failed shards; the clean test-path gain was below the tracked 20-second adoption threshold.
  • Per-stack database pool caps: a moderate cap reproduced the upstream postgres.js 3.4.9 transaction-reservation defect; one connection avoided the immediate error but missed the runtime target. The experiment was removed completely.
  • No unsafe or unreliable experiment remains in the final PR diff.

Validation

  • 439/439 Chromium selectors accepted exactly once.
  • Workflow, duration-planning, image-pull, security, harness, benchmark, and release contract tests pass.
  • Pinned actionlint and diff checks pass.
  • The Stop fix passed 31 focused backend tests, the complete 1,424-test backend suite, 10/10 zero-retry focused E2E repetitions, and a real authenticated in-app-browser journey with two Stop cycles plus a successful run between them.

Final gate

The exact final commit passed two complete clean 16/2 E2E runs, security, platform CI, and Codex review. After merge, timings will be verified again on main.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-31T22:37:11.817881Z 733ba45 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5adf6bdc62

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/e2e.yml
Comment thread .github/workflows/security.yml
@msrivas-7 msrivas-7 added the ci-duration-benchmark Run duration-balanced E2E topology benchmark label Aug 31, 2026
@msrivas-7 msrivas-7 added ci-duration-benchmark Run duration-balanced E2E topology benchmark and removed ci-duration-benchmark Run duration-balanced E2E topology benchmark labels Aug 31, 2026
@msrivas-7 msrivas-7 added ci-duration-benchmark Run duration-balanced E2E topology benchmark and removed ci-duration-benchmark Run duration-balanced E2E topology benchmark labels Aug 31, 2026
@msrivas-7 msrivas-7 added the ci-duration-benchmark Run duration-balanced E2E topology benchmark label Aug 31, 2026
@msrivas-7 msrivas-7 removed the ci-duration-benchmark Run duration-balanced E2E topology benchmark label Aug 31, 2026
@msrivas-7

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 733ba45e5b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@msrivas-7
msrivas-7 merged commit 9756217 into main Aug 31, 2026
98 of 99 checks passed
@msrivas-7
msrivas-7 deleted the dev/ci-four-minute-target branch August 31, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant