Skip to content

Commit 003c352

Browse files
authored
improvement(ci): run affected suites and reuse production builds (#9297)
* improvement(ci): reuse E2E builds and rebalance integration runners * improvement(ci): use free integration runners and shard browser viewports * improvement(ci): share production builds across HTTP and Electron suites * improvement(ci): select affected task groups with Turborepo * improvement(ci): scope type checks and right-size hosted runners * improvement(ci): partition macOS desktop smoke coverage * fix(ci): retain integration coverage for shared UI and harness changes * fix(ci): preserve landing dependencies and favor integration latency * docs(ci): clarify paired provider benchmark count * ci: retain free GitHub integration runners * docs(ci): clarify push and manual check selection * fix(ci): preserve hosted SCIM entitlement coverage
1 parent eea2dcc commit 003c352

23 files changed

Lines changed: 1411 additions & 713 deletions

‎.github/CI.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# CI selection and acceptance ownership
2+
3+
The `changes` job runs Turborepo 2.9.14's `affectedTasks` query before installing application dependencies or provisioning services. `turbo.json` records package dependency propagation; `apps/sim/turbo.json` partitions the app into shared runtime, server, workspace UI, and acceptance tasks. `.github/ci-suites.json` turns those tasks into job matrices. The uploaded `ci-plan` artifact names changed files, selected tasks, attribution reasons, and any fallback.
4+
5+
A PR compares its merge base with the checked-out merge commit. Main/staging pushes execute all checks; the parent `ci` workflow skips checks on `dev` pushes, while PRs targeting `dev` still run checks. A manual `checks` run executes all checks unless supplied a full base commit SHA to reproduce a scoped plan. Missing history, an empty diff, unknown files outside the workspace graph, root script/CI changes, or a planner error run every check. Missing selection outputs fail the aggregate gate. The gate also rejects an omitted dependency, any failed/cancelled job, and any skip that its explicit selection flag did not authorize.
6+
7+
| Change | Expensive checks selected |
8+
| --- | --- |
9+
| Independent SDK | That workspace's tests and type-check; no app build, services, or browsers |
10+
| App unit test | App test shards and test type check |
11+
| Integration test | Both provision paths; no HTTP/browser job merely for the test edit |
12+
| Workspace UI | Database integration, mobile, SEO and live Electron, plus app tests/build |
13+
| Landing content | SEO, app tests/build; no database integration or Electron |
14+
| Shared runtime or transitive dependency | All dependent boundaries |
15+
| One E2E harness | Its acceptance group and the shared production build |
16+
17+
The server boundary deliberately includes all API routes and backend domains. Shared modules and unclassified app source remain broad inputs: this is safer than pretending a route's static imports describe database, dynamic registry, or background-worker dependencies. Files read outside a package graph need explicit inputs: CLI acceptance imports `packages/sim-cli/src/embed.ts`, app tests read docs content, and live Electron and realtime acceptance cross app boundaries. When adding such a dependency, update the corresponding task inputs/dependencies and the real Git/Turbo selector scenarios.
18+
19+
Integration also depends on workspace UI because the desktop executor integration suite runs the production workspace stream dispatcher against PostgreSQL and Redis. Its explicit harness inputs include the database and realtime Vitest configs and integration setup files, which runtime tasks exclude.
20+
21+
SEO depends on workspace UI and server code: landing pages reuse workspace pricing and preview components, provider models/pricing, and the block registry. These directory boundaries are not isolated application dependency graphs.
22+
23+
`affectedTasks` already uses task inputs. Do not enable `futureFlags.affectedUsingTaskInputs` just for this planner: that flag also changes pruning, which can pull CI-only cross-app task dependencies into production Docker contexts.
24+
25+
Affected type-checks follow the package graph too; repository audits and lint run whenever the checks workflow runs. Workspace-only unit jobs use a free GitHub runner and do not start the app test shards.
26+
27+
Database integration and lightweight HTTP suites run on free standard GitHub runners with matching GitHub cache setup. A same-source integration comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on free GitHub, with identical coverage. GitHub trades the observed 51 seconds for zero runner compute cost. This is one paired sample, not a long-run percentile estimate. These explicitly selected GitHub jobs do not follow `CI_PROVIDER`.
28+
29+
Memory-heavy builds and browser jobs retain appropriately sized Blacksmith runners. Electron uses 4-vCPU Blacksmith: both shards passed within 14 seconds of the 8-vCPU control; the free GitHub rest shard was slower and its lease run stalled.
30+
31+
Every HTTP and browser group consumes the same production build in an isolated job/database. Electron also uses that build and skips its development-route warm-up. Local HTTP and Electron development runs retain their dev-server behavior. Build inputs exclude test-only files so a harness edit can reuse an unchanged production build.
32+
33+
SCIM's production app uses a `sim.ai` fixture hostname mapped to loopback, since `NEXT_PUBLIC_FORCE_HOSTED` is ignored in production. Readiness, HTTP tests, and internal API requests still connect to loopback. The SCIM suite verifies that ending an Enterprise subscription rejects both owner administration and an existing bearer token, and that restoring it restores access.
34+
35+
## Coverage removed or consolidated
36+
37+
- SEO is plain HTTP, independent of browser and viewport. Its complete suite runs once in the SEO job, instead of repeating in each of four mobile shards.
38+
- The MCP suite's homepage-returns-200 control duplicated SEO's stronger real-homepage Markdown/content-negotiation check. SEO owns that positive control; MCP retains host isolation, ownership challenge, discovery, API catalogs, unknown-page/API behavior, and refusal of Markdown.
39+
- CLI CI uses one cold/warm-up iteration and one measured iteration instead of three identical measured repetitions. Both retain run completion, selected output, file-reference size, trace, and omitted-workflow-snapshot assertions. The standalone benchmark keeps its configurable sample size.
40+
- A precompiled Electron app needs no route-compilation warm-up, which repeated read/import/chat-switch/Stop flows before each shard. The actual acceptance tests still exercise those behaviors.
41+
42+
The long-import and lease-expiration checks remain: they prove distinct survival/uncertain-outcome behavior across real process failure. Likewise, the two integration provision paths remain because migrations create constraints/triggers that schema push does not.
43+
44+
Local planner validation is `bun run test:scripts scripts/ci-changes.test.ts scripts/ci-gate.test.ts`. It executes real Turbo queries in small Git fixtures; no application build or database is required. Full builds, services, browser suites, and Electron checks run on hosted CI.
45+
46+
The separate `desktop-e2e` workflow retains its desktop-specific path filters. Its macOS shell smoke suite is partitioned by Playwright across three isolated runners; serial test-file groups stay intact. Packaging smoke and the weekly Electron canary remain separate. Each shard uploads its small JSON coverage/timing report.

‎.github/ci-suites.json‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{
2+
"http": [
3+
{
4+
"group": "scim",
5+
"task": "ci:e2e-scim"
6+
},
7+
{
8+
"group": "cli",
9+
"task": "ci:e2e-cli"
10+
},
11+
{
12+
"group": "stop-after",
13+
"task": "ci:e2e-stop-after"
14+
},
15+
{
16+
"group": "desktop-inbox",
17+
"task": "ci:e2e-desktop-inbox"
18+
},
19+
{
20+
"group": "seo",
21+
"task": "ci:e2e-seo"
22+
},
23+
{
24+
"group": "project-files",
25+
"task": "ci:e2e-project-files"
26+
}
27+
],
28+
"production": [
29+
{
30+
"group": "mobile",
31+
"name": "mobile, chromium, 1/2",
32+
"browser": "chromium",
33+
"shard": "1/2",
34+
"artifact": "mobile-chromium-1",
35+
"task": "ci:e2e-mobile"
36+
},
37+
{
38+
"group": "mobile",
39+
"name": "mobile, chromium, 2/2",
40+
"browser": "chromium",
41+
"shard": "2/2",
42+
"artifact": "mobile-chromium-2",
43+
"task": "ci:e2e-mobile"
44+
},
45+
{
46+
"group": "mobile",
47+
"name": "mobile, webkit, 1/2",
48+
"browser": "webkit",
49+
"shard": "1/2",
50+
"artifact": "mobile-webkit-1",
51+
"task": "ci:e2e-mobile"
52+
},
53+
{
54+
"group": "mobile",
55+
"name": "mobile, webkit, 2/2",
56+
"browser": "webkit",
57+
"shard": "2/2",
58+
"artifact": "mobile-webkit-2",
59+
"task": "ci:e2e-mobile"
60+
}
61+
]
62+
}

‎.github/scripts/ci-changes.sh‎

Lines changed: 5 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -1,65 +1,6 @@
11
#!/usr/bin/env bash
2-
# Prints which checks a pull request's changes can affect, as `<flag>=true|false` lines for
3-
# $GITHUB_OUTPUT. A flag is `false` only when every changed file is on an allowlist whose jobs
4-
# fully cover it. A file on no allowlist sets every flag but `desktop_live` (which keeps its own
5-
# skip list) to `true`; a change under .github/, an empty diff, or any failure to work out the
6-
# diff sets all of them.
7-
#
8-
# full integration, e2e, e2e-build, e2e-mobile: any file not allowlisted below
9-
# test the Vitest shards and test-types: content, docs app, and desktop changes,
10-
# plus `full`
11-
# build the app's `next build`: content changes, plus `full`
12-
# desktop_live the live Electron suite: any file outside its own skip list
13-
#
14-
# lint always runs and is not gated.
15-
#
16-
# Usage: ci-changes.sh <base-sha>
17-
set -u
18-
19-
# Posts, author profiles, and the public assets only posts reference. The test shards read them
20-
# (lib/content/og-image.test.ts) and `next build` prerenders every post with its MDX components,
21-
# which check:library-content compiles but never renders. Components under content/ are code.
22-
content='^apps/sim/content/([^/]+/)*[^/]+\.mdx?$|^apps/sim/content/authors/[^/]+\.json$|^apps/sim/public/(authors|blog|changelog|library)/'
23-
# Its own workspace tests, its type-check and docs-manifest audit in lint, and apps/sim tests
24-
# that read its pages (tools/netsuite/netsuite.test.ts).
25-
docs='^apps/docs/'
26-
# Nothing outside the Electron app imports it; the desktop bridge and protocol packages are
27-
# consumed by apps/sim and stay unlisted.
28-
desktop='^apps/desktop/'
29-
# Agent guidance, read only by the lint job's audits.
30-
guidance='^(\.agents|\.claude|\.cursor)/|^([^/]+/)*(AGENTS|CLAUDE)\.md$|^README\.md$'
31-
# The live desktop suite drives apps/desktop against the app, so it skips more than the rest.
32-
desktop_live_skip="$guidance"'|^(apps/docs/|apps/pii/|apps/sim/content/|apps/sim/public/(authors|blog|changelog|library)/|packages/(python-sdk|ts-sdk)/)|\.mdx?$|(^|/)LICENSE$'
33-
34-
emit() {
35-
printf 'full=%s\ntest=%s\nbuild=%s\ndesktop_live=%s\n' "$1" "$2" "$3" "$4"
36-
}
37-
run_all() {
38-
emit true true true true
39-
echo "Running every check: $1" >&2
40-
exit 0
41-
}
42-
matches() { printf '%s\n' "$names" | grep -qE "$1"; }
43-
outside() { printf '%s\n' "$names" | grep -qvE "$1"; }
44-
45-
base=${1:-}
46-
[ -n "$base" ] || run_all 'no base commit'
47-
git fetch --quiet --depth=1 origin "$base" || run_all "could not fetch $base"
48-
names=$(git diff --no-renames --name-only "$base" HEAD) || run_all "could not diff against $base"
49-
[ -n "$names" ] || run_all 'no changed files listed'
50-
if matches '^\.github/'; then run_all 'a CI file changed'; fi
51-
52-
desktop_live=false
53-
if outside "$desktop_live_skip"; then desktop_live=true; fi
54-
if outside "$content|$docs|$desktop|$guidance"; then
55-
emit true true true "$desktop_live"
56-
unlisted=$(printf '%s\n' "$names" | grep -vE "$content|$docs|$desktop|$guidance" | head -n 1)
57-
echo "Running every check (desktop_live=$desktop_live): $unlisted is not on an allowlist" >&2
58-
exit 0
59-
fi
60-
61-
test=false build=false
62-
if matches "$content|$docs|$desktop"; then test=true; fi
63-
if matches "$content"; then build=true; fi
64-
emit false "$test" "$build" "$desktop_live"
65-
echo "Scoped checks: test=$test build=$build desktop_live=$desktop_live" >&2
2+
# Select tasks from the committed Turborepo graph; unknown files or failed attribution run all.
3+
# Usage: ci-changes.sh [base-sha]. No base runs the complete push/dispatch gate.
4+
set -euo pipefail
5+
script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
6+
exec bun "$script_dir/../../scripts/ci-plan.ts" "${1:-}"

‎.github/scripts/ci-gate.sh‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,7 @@
55
#
66
# A skip passes only for a gated job whose `changes` flag is exactly `false`:
77
#
8-
# integration, e2e, e2e-build, e2e-mobile full
9-
# test, test-types test
10-
# build build
11-
# desktop-live desktop_live
8+
# Each job maps to its own explicit selection flag below. No inferred skip is accepted.
129
#
1310
# Every other skip fails: an ungated job (lint, `changes` itself), a gated job whose flag is
1411
# `true`, missing, or anything else (a failed or skipped `changes` leaves no outputs). A failure
@@ -24,13 +21,21 @@ if ! jq -e 'type == "object"' <<< "$needs" > /dev/null 2>&1; then
2421
fi
2522

2623
failed="$(jq -r '
27-
{"integration": "full", "e2e": "full", "e2e-build": "full", "e2e-mobile": "full", "test": "test",
28-
"test-types": "test", "build": "build", "desktop-live": "desktop_live"} as $gate
24+
{"integration": "integration", "e2e": "http", "e2e-build": "e2e_build", "e2e-production": "production", "test": "test",
25+
"test-types": "test_app", "build": "build", "desktop-live": "desktop_live"} as $gate
26+
| . as $needs
27+
| (($gate | keys) + ["changes", "lint"]) as $required
28+
| ($required - keys) as $missing
29+
| if ($missing | length) > 0 then
30+
$missing[] | "\(.): missing"
31+
else
32+
$needs
2933
| (.changes.outputs // {}) as $flags
3034
| to_entries[]
3135
| select(.value.result != "success")
3236
| select(.value.result != "skipped" or $gate[.key] == null or $flags[$gate[.key]] != "false")
3337
| "\(.key): \(.value.result)"
38+
end
3439
' <<< "$needs")"
3540
if [ -n "$failed" ]; then
3641
echo "::error::Checks did not pass:"

‎.github/scripts/ci-plan.gql‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
query($base: String!, $head: String!) {
2+
packages { items { name path } }
3+
affectedTasks(base: $base, head: $head, tasks: [
4+
"test", "build", "type-check", "ci:integration", "ci:desktop-live",
5+
"ci:e2e-scim", "ci:e2e-cli", "ci:e2e-stop-after", "ci:e2e-desktop-inbox",
6+
"ci:e2e-project-files", "ci:e2e-mobile", "ci:e2e-seo"
7+
]) {
8+
items {
9+
name
10+
fullName
11+
package { name }
12+
reason {
13+
__typename
14+
... on TaskFileChanged { filePath }
15+
... on TaskGlobalFileChanged { filePath }
16+
... on TaskGlobalDepsChanged { filePath }
17+
... on TaskDependencyTaskChanged { taskName packageName }
18+
... on TaskPackageDependencyChanged { packageName }
19+
... on TaskAllChanged { description }
20+
}
21+
}
22+
}
23+
}

‎.github/scripts/http-e2e.sh‎

Lines changed: 19 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
#!/usr/bin/env bash
22
# Runs one end-to-end suite group over real HTTP, each against its own app: `next dev`, except
3-
# mobile, which serves the production build the `e2e-build` job uploads with `next start` (the
4-
# job extracts it into apps/sim/.next first).
3+
# CI groups, which reuse the production build from `e2e-build`.
4+
# The job extracts it into apps/sim/.next before running these suites.
55
#
6-
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|project-files|mobile> (run from apps/sim)
6+
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|project-files|mobile|seo> (run from apps/sim)
77
#
88
# The job provides DATABASE_URL, BETTER_AUTH_SECRET and ENCRYPTION_KEY; each group sets the rest of
99
# its app's environment here. Reports and server logs land in $RUNNER_TEMP/e2e.
@@ -20,7 +20,7 @@
2020
# telemetry flush runs detached and still writes .next/dev).
2121
set -euo pipefail
2222

23-
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|project-files|mobile>}
23+
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|project-files|mobile|seo>}
2424
report_dir="$RUNNER_TEMP/e2e"
2525
ready_timeout_seconds=600
2626
mkdir -p "$report_dir"
@@ -33,8 +33,9 @@ realtime_pid=''
3333
realtime_tag=''
3434
realtime_log=''
3535
browsers_pid=''
36+
local_app_url=''
3637
# `dev` or `start`; a group sets it before start_app.
37-
app_server=dev
38+
app_server=${E2E_APP_SERVER:-dev}
3839

3940
finish() {
4041
local status=$?
@@ -78,7 +79,8 @@ start_app() {
7879
status_log="$report_dir/$name-http-status.log"
7980
fi
8081
app_tag="$name-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT-$$"
81-
export NEXT_PUBLIC_APP_URL="http://127.0.0.1:$port"
82+
local_app_url="http://127.0.0.1:$port"
83+
export NEXT_PUBLIC_APP_URL="${E2E_APP_PUBLIC_URL:-$local_app_url}"
8284
export BETTER_AUTH_URL="$NEXT_PUBLIC_APP_URL"
8385
export DISABLE_TELEMETRY=true NEXT_TELEMETRY_DISABLED=1
8486
if [ "$app_server" = start ]; then
@@ -96,7 +98,7 @@ start_app() {
9698
--port "$port" > "$server_log" 2>&1 &
9799
fi
98100
server_pid=$!
99-
wait_ready "$server_pid" "$NEXT_PUBLIC_APP_URL/api/health" "$label app"
101+
wait_ready "$server_pid" "$local_app_url/api/health" "$label app"
100102
}
101103

102104
# start_realtime <name> <port>: the Socket.IO server the app's browser clients connect to, so pages
@@ -143,12 +145,12 @@ case "$group" in
143145
export DB_TX_TRIPWIRE=throw
144146
export NEXT_PUBLIC_CHAT_DISABLED=true
145147
start_app scim 3017 SCIM record-http-status
146-
SCIM_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
148+
SCIM_E2E_BASE_URL="$local_app_url" \
147149
SCIM_E2E_DATABASE_URL="$DATABASE_URL" \
148150
SCIM_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
149151
SCIM_E2E_REPORT_PATH="$report_dir/scim-e2e-report.json" \
150152
bun run test:scim:e2e
151-
VERSION_COMPARE_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
153+
VERSION_COMPARE_E2E_BASE_URL="$local_app_url" \
152154
VERSION_COMPARE_E2E_DATABASE_URL="$DATABASE_URL" \
153155
VERSION_COMPARE_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
154156
VERSION_COMPARE_E2E_REPORT_PATH="$report_dir/version-compare-http-report.json" \
@@ -174,7 +176,7 @@ case "$group" in
174176
bun --no-env-file scripts/test-mcp-host-e2e.ts
175177
CLI_LATENCY_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
176178
CLI_LATENCY_E2E_DATABASE_URL="$DATABASE_URL" \
177-
CLI_LATENCY_E2E_RUNS=3 \
179+
CLI_LATENCY_E2E_RUNS=1 \
178180
CLI_LATENCY_E2E_WARMUP=1 \
179181
CLI_LATENCY_E2E_REPORT_PATH="$report_dir/cli-run-latency-report.json" \
180182
bun run test:cli-run-latency:e2e
@@ -246,7 +248,13 @@ case "$group" in
246248
MOBILE_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
247249
MOBILE_E2E_REPORT_PATH="$report_dir/mobile-e2e-report.json" \
248250
bun run test:mobile:e2e
249-
# The same production build answers as a crawler sees it: statuses, robots, sitemap, JSON-LD.
251+
;;
252+
253+
seo)
254+
app_server=start
255+
export NEXT_PUBLIC_FORCE_HOSTED=false
256+
start_app seo 3024 'landing SEO' record-http-status
257+
# Plain HTTP assertions have no browser-specific behavior.
250258
LANDING_SEO_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
251259
LANDING_SEO_E2E_REPORT_PATH="$report_dir/landing-seo-e2e-report.json" \
252260
bun run test:landing-seo:e2e

0 commit comments

Comments
 (0)