improvement(ci): run affected suites and reuse production builds by waleedlatif1 · Pull Request #9297 · simstudioai/sim

waleedlatif1 · 2026-10-22T00:16:27Z

Summary

CI repeatedly compiled the same app in dev servers and ran broad suites for unrelated changes. Reuse the production build across HTTP, browser, Project Files, and Electron acceptance; use Turborepo's affected-task query to select the relevant test, type-check, build, and acceptance boundaries.

Partition shared runtime, server, workspace UI, landing/SEO, integration, and acceptance harness inputs. Propagate package dependencies and explicit cross-workspace reads, including the embedded CLI and docs-content tests. Upload changed files, selected tasks, reasons, and fallback decisions as ci-plan.
Generate job matrices from the selected tasks. Unknown inputs, missing history, CI/root-script changes, or a planner error run everything. Strengthen the aggregate gate to reject missing dependencies as well as failures, cancellations, and unauthorized skips.
Run integration, HTTP/Project Files, workspace-only tests, planning, and the gate on free standard GitHub runners after comparing all 16 integration jobs on both providers. Keep the memory-heavy builds and browser/app-test workloads on Blacksmith. Reduce Electron from 16 to 4 vCPUs after hosted comparisons. Align fallback checkout with the cache provider.
Balance both database provisioning paths across eight shards using refreshed timings for 289 files; split each mobile browser across two complete viewport partitions.
Partition the separate macOS shell smoke suite across three isolated runners, preserving serial file groups and its desktop-specific triggers. Retain packaged-app smoke and the weekly canary.
Run SEO once independently of browser shards; remove its redundant MCP positive-page control; use one measured CLI iteration after warm-up; skip Electron's route-compilation warm-up with a prebuilt app. Preserve long-import, renderer-crash/lease, permissions, and network-recovery cases. Move the computer-tool-unload test's cold import outside the timed assertion without changing its assertions or timeout.

The production SCIM fixture uses a hosted hostname mapped to loopback. A new HTTP regression verifies that canceling Enterprise entitlement rejects administration and existing bearer access, then restoring it restores access. It failed on the previous configuration and passed on the corrected one using the same production artifact.

Selection ownership and fallback behavior are documented in .github/CI.md.

Validation

Final head d3a59c8d03: Cubic and Greptile both 5/5, no actionable findings; all review threads resolved. Includes current staging 63814736c7.
Full CI passed in 8m 06s; macOS passed in 8m 00s, both first attempts. The final E2E production-build job took 3m 21s, including 3m of compilation. Earlier warm-cache full CI took 7m 39s. Recent successful full-PR baseline was 12m 24s median / 14m 52s p95 across 57 runs; these new runs are individual observations, not new percentiles.
Uploaded report comparison preserves all 289 integration files and 3,166 assertions per provisioning path (including the same existing skips), 97 unique checks per browser, nine Project Files suites, and all 12 Electron cases without retries. Final HTTP reports contain 249 checks, preserving the earlier controls and adding the hosted SCIM entitlement regression. macOS preserves all 111 case names/statuses: 96 passed, the same 15 existing skips, no retries; live-app and packaged-app checks pass in their dedicated jobs.
The hosted SCIM regression failed against the previous production configuration with HTTP 200 instead of 403 after subscription cancellation. The corrected configuration passed all 19 SCIM and nine version-comparison checks using the exact same production artifact, on both GitHub and Blacksmith.
Dependency regressions failed before their fixes: sibling Vitest configs/setup, workspace stream integration, and landing-page UI/backend dependencies now select their owning checks. All 55 real Git/Turbo planner and aggregate-gate checks pass.
A representative SDK-only hosted run passed in 1m 54s, selecting SDK tests/type-checks plus repository guards and skipping application builds/services/acceptance. The final selector regression suite preserves that behavior.
Same-source integration comparison: all 32 jobs passed with identical coverage. Longest job: 2-vCPU Blacksmith 6m 04s versus GitHub 6m 55s. The owner selected free GitHub integration, accepting that observed tradeoff. Electron stays on 4-vCPU Blacksmith after both shards passed within 14 seconds of the 8-vCPU controls; the free-runner lease experiment stalled.
Workflow validation, shell syntax, formatting/diff checks and the focused planner tests ran locally. Full audits, type-checks, builds, databases, browser/Electron suites and performance comparisons ran on hosted runners.

Final full CI · Final macOS · SDK-only proof · Paired integration comparison · SCIM regression before fix · Hosted SCIM on GitHub · Hosted SCIM on Blacksmith

vercel · 2026-10-22T00:16:34Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 22, 2026 6:33am UTC

…orts

cubic-dev-ai

No issues found across 7 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-22T00:47:35Z

Confidence Score: 5/5

[Medium impact] The current head appears safe to merge, with no actionable findings remaining.

Summary

This PR selects affected CI suites, shares one production build across acceptance jobs, and splits longer suites across more runners.

CI selects checks from the tasks affected by each change.
Acceptance jobs share a production build and split large suites across runners.
Acceptance suites check hosted SCIM access and avoid repeated setup work.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Changes[Changed files] --> Planner[Affected task plan]
  Planner --> Checks[Selected tests and type checks]
  Planner --> Build[Shared production build]
  Build --> HTTP[HTTP acceptance]
  Build --> Browser[Mobile browser acceptance]
  Build --> Electron[Live Electron acceptance]
  HTTP --> SCIM[Hosted SCIM fixture mapped to loopback]
  Checks --> Gate[Aggregate CI gate]
  HTTP --> Gate
  Browser --> Gate
  Electron --> Gate

_{Reviews (8) · Last reviewed commit: "fix(ci): preserve hosted SCIM entitlemen..." · Reviewed by Greptile}

waleedlatif1 · 2026-10-22T02:01:08Z

@greptileai Please re-review the complete PR at current head bff452a before squash merge. The previous review covered fd8616a. Focus on affected-task coverage, fail-closed CI gating, production artifact reuse, runner compatibility, and preservation of test coverage.

waleedlatif1 · 2026-10-22T02:01:09Z

@cubic-dev-ai review this PR at current head bff452a. Please review the complete PR before squash merge, including changes since fd8616a. Focus on affected-task coverage, fail-closed CI gating, production artifact reuse, runner compatibility, and preservation of test coverage.

cubic-dev-ai · 2026-10-22T02:01:20Z

@cubic-dev-ai review this PR at current head bff452a. Please review the complete PR before squash merge, including changes since fd8616a. Focus on affected-task coverage, fail-closed CI gating, production artifact reuse, runner compatibility, and preservation of test coverage.

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 21 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-22T02:09:22Z

@greptileai Please re-review the complete current head 8e97feb before merge. Both reported dependency gaps are fixed, with regressions demonstrated red before and green after. Please check for any remaining skipped coverage.

waleedlatif1 · 2026-10-22T02:09:23Z

@cubic-dev-ai review this PR at current head 8e97feb. The sibling harness inputs and workspace-stream integration dependency are fixed. All 52 real Git/Turbo planner and aggregate gate checks pass. Please verify the complete PR has no remaining issues.

cubic-dev-ai · 2026-10-22T02:09:34Z

@cubic-dev-ai review this PR at current head 8e97feb. The sibling harness inputs and workspace-stream integration dependency are fixed. All 52 real Git/Turbo planner and aggregate gate checks pass. Please verify the complete PR has no remaining issues.

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 21 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 21 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-22T02:30:50Z

@cubic-dev-ai review this PR at current head c4e196f. The benchmark wording now explicitly says 16 jobs per provider (32 total). The artifact-name warning was tested on the unchanged prior head: manual rerun job 119603757890 passed with the pinned upload-artifact v7.0.2, producing a second e2e-reports-seo artifact while preserving the first (IDs 12396807154 and 12397536990). See the review-thread reply. All prior dependency fixes passed full CI/macOS and all 55 planner/gate regressions. Please verify no actionable findings remain.

waleedlatif1 · 2026-10-22T02:30:51Z

@greptileai Please re-review the final head c4e196f. The only change since your clean 5/5 review is clarifying the paired benchmark count: 16 jobs per provider, 32 total. Prior head efb9f8c passed full CI and macOS. A manual SEO rerun also passed and preserved both report artifacts. Please confirm the final PR remains clean.

cubic-dev-ai · 2026-10-22T02:30:54Z

@cubic-dev-ai review this PR at current head c4e196f. The benchmark wording now explicitly says 16 jobs per provider (32 total). The artifact-name warning was tested on the unchanged prior head: manual rerun job 119603757890 passed with the pinned upload-artifact v7.0.2, producing a second e2e-reports-seo artifact while preserving the first (IDs 12396807154 and 12397536990). See the review-thread reply. All prior dependency fixes passed full CI/macOS and all 55 planner/gate regressions. Please verify no actionable findings remain.

@waleedlatif1 I've dismissed this issue:

P3 .github/workflows/checks.yml:289: When a failed suite has produced JSON, this always() step uploads the stable artifact name; rerunning the failed job then fails because upload-artifact defaults to rejecting an existing name. Set…

cubic-dev-ai · 2026-10-22T02:30:59Z

@cubic-dev-ai review this PR at current head c4e196f. The benchmark wording now explicitly says 16 jobs per provider (32 total). The artifact-name warning was tested on the unchanged prior head: manual rerun job 119603757890 passed with the pinned upload-artifact v7.0.2, producing a second e2e-reports-seo artifact while preserving the first (IDs 12396807154 and 12397536990). See the review-thread reply. All prior dependency fixes passed full CI/macOS and all 55 planner/gate regressions. Please verify no actionable findings remain.

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 21 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-22T06:07:22Z

@cubic-dev-ai Please review final head 333a30c. The owner has selected free GitHub integration runners after considering the paired benchmark; both runs-on and cache provider now explicitly use GitHub. Docs describe the 51-second observed tradeoff and 16 jobs per provider (32 total). All previous dependency fixes remain intact. The artifact-name warning is a verified false positive for the pinned v7.0.2: unchanged manual rerun job 119603757890 passed and preserved two same-name reports (IDs 12396807154 and 12397536990). Please verify no actionable issues remain at this final head.

waleedlatif1 · 2026-10-22T06:07:22Z

@greptileai Please review final head 333a30c. The owner has selected free GitHub integration runners after considering the paired benchmark; both runs-on and cache provider now explicitly use GitHub. Docs describe the 51-second observed tradeoff and 16 jobs per provider (32 total). All previous dependency fixes remain intact. The artifact-name warning is a verified false positive for the pinned v7.0.2: unchanged manual rerun job 119603757890 passed and preserved two same-name reports (IDs 12396807154 and 12397536990). Please verify no actionable issues remain at this final head.

cubic-dev-ai · 2026-10-22T06:07:33Z

@cubic-dev-ai Please review final head 333a30c. The owner has selected free GitHub integration runners after considering the paired benchmark; both runs-on and cache provider now explicitly use GitHub. Docs describe the 51-second observed tradeoff and 16 jobs per provider (32 total). All previous dependency fixes remain intact. The artifact-name warning is a verified false positive for the pinned v7.0.2: unchanged manual rerun job 119603757890 passed and preserved two same-name reports (IDs 12396807154 and 12397536990). Please verify no actionable issues remain at this final head.

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 21 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

…20261021

waleedlatif1 · 2026-10-22T06:10:56Z

@cubic-dev-ai Please verify final head 4530ec9, now including current staging 6381473. Integration remains on free GitHub by owner choice, with matching cache setup. The dev-push documentation exception is corrected. All earlier dependency findings remain fixed; the pinned artifact upload was verified on a successful unchanged manual rerun (job 119603757890). Please check the complete PR for remaining actionable findings. Fresh hosted CI is running against this head.

waleedlatif1 · 2026-10-22T06:10:57Z

@greptileai Please verify final head 4530ec9, now including current staging 6381473. Integration remains on free GitHub by owner choice, with matching cache setup. The dev-push documentation exception is corrected. All earlier dependency findings remain fixed; the pinned artifact upload was verified on a successful unchanged manual rerun (job 119603757890). Please check the complete PR for remaining actionable findings. Fresh hosted CI is running against this head.

cubic-dev-ai · 2026-10-22T06:11:07Z

@cubic-dev-ai Please verify final head 4530ec9, now including current staging 6381473. Integration remains on free GitHub by owner choice, with matching cache setup. The dev-push documentation exception is corrected. All earlier dependency findings remain fixed; the pinned artifact upload was verified on a successful unchanged manual rerun (job 119603757890). Please check the complete PR for remaining actionable findings. Fresh hosted CI is running against this head.

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 21 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-22T06:30:31Z

@cubic-dev-ai Please review the complete current head d3a59c8. The hosted SCIM finding is fixed with an actual hosted fixture origin mapped to loopback, retaining production-build reuse. Its new HTTP entitlement-lapse regression demonstrably failed on the old setup (run 39733908970: 200 instead of 403) and passed after the fix (run 39734104391: all 19 SCIM and 9 version-comparison checks). Original checks remain; no production security override or retry was added. Integration remains on free GitHub by owner choice. Fresh full CI is running. Please verify no actionable findings remain.

waleedlatif1 · 2026-10-22T06:30:32Z

@greptileai Please review the complete current head d3a59c8. The hosted SCIM finding is fixed with an actual hosted fixture origin mapped to loopback, retaining production-build reuse. Its new HTTP entitlement-lapse regression demonstrably failed on the old setup (run 39733908970: 200 instead of 403) and passed after the fix (run 39734104391: all 19 SCIM and 9 version-comparison checks). Original checks remain; no production security override or retry was added. Integration remains on free GitHub by owner choice. Fresh full CI is running. Please verify no actionable findings remain.

cubic-dev-ai · 2026-10-22T06:30:45Z

@cubic-dev-ai Please review the complete current head d3a59c8. The hosted SCIM finding is fixed with an actual hosted fixture origin mapped to loopback, retaining production-build reuse. Its new HTTP entitlement-lapse regression demonstrably failed on the old setup (run 39733908970: 200 instead of 403) and passed after the fix (run 39734104391: all 19 SCIM and 9 version-comparison checks). Original checks remain; no production security override or retry was added. Integration remains on free GitHub by owner choice. Fresh full CI is running. Please verify no actionable findings remain.

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 23 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

improvement(ci): reuse E2E builds and rebalance integration runners

4924e4c

vercel Bot deployed to Preview October 22, 2026 00:21

View deployment

improvement(ci): use free integration runners and shard browser viewp…

fd8616a

…orts

vercel Bot deployed to Preview October 22, 2026 00:34

View deployment

waleedlatif1 changed the title ~~improvement(ci): reuse E2E builds and rebalance integration runners~~ improvement(ci): reuse E2E builds and move integration tests to free runners Oct 22, 2026

waleedlatif1 marked this pull request as ready for review October 22, 2026 00:43

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

improvement(ci): share production builds across HTTP and Electron suites

ff3d930

vercel Bot deployed to Preview October 22, 2026 00:57

View deployment

improvement(ci): select affected task groups with Turborepo

9499b73

waleedlatif1 requested a review from a team as a code owner October 22, 2026 01:11

vercel Bot deployed to Preview October 22, 2026 01:17

View deployment

improvement(ci): scope type checks and right-size hosted runners

965c58f

vercel Bot deployed to Preview October 22, 2026 01:32

View deployment

waleedlatif1 changed the title ~~improvement(ci): reuse E2E builds and move integration tests to free runners~~ improvement(ci): run affected suites and reuse production builds Oct 22, 2026

improvement(ci): partition macOS desktop smoke coverage

bff452a

vercel Bot deployed to Preview October 22, 2026 01:47

View deployment

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread apps/sim/turbo.json Outdated

greptile-apps Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread apps/sim/turbo.json Outdated

Comment thread apps/sim/turbo.json Outdated

fix(ci): retain integration coverage for shared UI and harness changes

8e97feb

vercel Bot deployed to Preview October 22, 2026 02:12

View deployment

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread apps/sim/turbo.json

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread .github/CI.md Outdated

Comment thread .github/workflows/checks.yml

docs(ci): clarify paired provider benchmark count

c4e196f

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread .github/CI.md Outdated

vercel Bot deployed to Preview October 22, 2026 02:34

View deployment

ci: retain free GitHub integration runners

333a30c

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

waleedlatif1 added 2 commits October 21, 2026 23:10

docs(ci): clarify push and manual check selection

8ee4afa

Merge remote-tracking branch 'origin/staging' into codex/ci-velocity-…

4530ec9

…20261021

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

vercel Bot deployed to Preview October 22, 2026 06:16

View deployment

greptile-apps Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread .github/workflows/checks.yml

fix(ci): preserve hosted SCIM entitlement coverage

d3a59c8

vercel Bot deployed to Preview October 22, 2026 06:33

View deployment

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

waleedlatif1 merged commit 003c352 into staging Oct 22, 2026
71 checks passed

waleedlatif1 deleted the codex/ci-velocity-20261021 branch October 22, 2026 18:05

waleedlatif1 mentioned this pull request Oct 22, 2026

v0.9.32: ec2 integration, nextjs minor version bump, logs perf improvements, memory pressure reduced #9320

Merged

Conversation

waleedlatif1 commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

vercel Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

waleedlatif1 commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 22, 2026 •

edited

Loading

vercel Bot commented Oct 22, 2026 •

edited

Loading

greptile-apps Bot commented Oct 22, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading