Skip to content

Commit 0cd1133

Browse files
authored
ci: mutate only the changed files in the gate's mutation step (#582)
The ts-fast lane (and so --quick, --only ts and the no-mistakes Test step) ran the full Stryker pass over all six metadata files on every run: 10-45 minutes of CPU per gate round on a loaded host, most of that step's wall time. The step now diffs the stryker.conf.mjs `mutate` list against the merge base with the base branch (MO_CI_MUTATION_BASE, else origin/$GITHUB_BASE_REF, else whichever of origin/main and origin/release/* HEAD is closest to) and runs `stryker run --mutate` on only the changed files. When none changed it prints a SKIP naming the base. When no base resolves, or stryker.conf.mjs itself changed, it says so and runs the full pass. Thresholds and concurrency are unchanged; the list stays in stryker.conf.mjs only. The full pass stays: the flagless run's mutation lane, a new `--only mutation` section for release prep and on demand, and conformance.yml (tags, nightly, dispatch) via MO_CI_MUTATION_SCOPE=full. Stryker's sandbox goes under DEV_SCRATCH when that names a directory, and is removed afterwards.
1 parent 059563c commit 0cd1133

4 files changed

Lines changed: 126 additions & 15 deletions

File tree

‎.github/workflows/conformance.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ name: conformance
1818
# gates — fixture-lint, doc-template drift, embedded-library drift, leak scan and
1919
# the other offline repo gates (`--only gates`)
2020
# ts-fast — workspace build + typecheck, TS conformance, completeness (mutation) gate
21+
# (the FULL pass here: MO_CI_MUTATION_SCOPE=full on that job)
2122
# csharp / java-fast (java + kotlin) / python — each port's conformance corpora
2223
# java-reactor — full-reactor `mvn clean install`, tests on (`--only java-slow`)
2324
# The Java lanes point the script's Maven repository at ~/.m2/repository so setup-java's
@@ -129,6 +130,9 @@ jobs:
129130
- name: Conformance (${{ matrix.lane }})
130131
env:
131132
LANE: ${{ matrix.lane }}
133+
# Tag, nightly and dispatch runs mutate EVERY stryker.conf.mjs file, not only the
134+
# changed ones ts-fast mutates elsewhere (see gate_completeness_diff).
135+
MO_CI_MUTATION_SCOPE: full
132136
run: |
133137
METAOBJECTS_CI_M2_REPO="$HOME/.m2/repository" \
134138
scripts/ci-local.sh --only "$LANE" --no-integration --strict-toolchains

‎.github/workflows/local-ci.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -116,8 +116,9 @@ jobs:
116116
# gate runs only named conformance FILES, so the rest of those suites ran nowhere
117117
# and a regression could sit red on `main` under a green ts lane.
118118
# Deliberately separate from ts-fast rather than folded into it: ts-fast carries
119-
# the 30-min mutation gate, which flakes under runner contention, and a mutation
120-
# flake must not mask a unit-test failure. The suites total ~5s, so this lane is
119+
# the mutation gate (diff-scoped: on a push to main it mutates nothing new and
120+
# SKIPs; the full pass runs in conformance.yml), which flaked under runner
121+
# contention, and a mutation flake must not mask a unit-test failure. The suites total ~5s, so this lane is
121122
# nearly free and gives an independent, fast signal.
122123
ts-unit:
123124
needs: detect

‎.no-mistakes.yaml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,12 @@
1111
# along the script's own `--only` section boundaries so neither step repeats the other.
1212
# `scripts/ci-local.sh --help` enumerates what each section contains, and
1313
# `MO_CI_LIST_ONLY=1 scripts/ci-local.sh --only <section>` prints its exact step list.
14-
# Measured green on a warm tree 2026-09-18: test 256s (it carries the mutation gate and is
15-
# almost all of the cost), lint 23s.
14+
# Measured green on a warm tree 2026-09-18: test 256s, lint 23s. The test step's mutation
15+
# gate is DIFF-SCOPED (gate_completeness_diff in the script): it mutates only the
16+
# stryker.conf.mjs `mutate` files the branch changed against its base, and SKIPs when it
17+
# changed none. Before 2026-11-14 it ran the full pass every round, 10-45 min under load
18+
# and most of this step's wall time. The full pass is `--only mutation`, the flagless
19+
# run's mutation lane, and conformance.yml (tags, nightly, dispatch).
1620
#
1721
# ORDER IS LOAD-BEARING, and it is why the work is split this way round rather than one
1822
# `--quick` in `test`. The pipeline runs test BEFORE lint. The `gates` section contains

‎scripts/ci-local.sh‎

Lines changed: 113 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,8 @@
2121
# scripts/ci-local.sh --quick # FAST inner-loop tier: the cheap TS-centric
2222
# # gates only (leak-scan, pom parity, fixture-
2323
# # lint, ts build+typecheck, ts conformance,
24-
# # completeness, drift). Skips the other-
24+
# # completeness on the CHANGED files only, drift).
25+
# # Skips the other-
2526
# # language ports, the reactor, and docker.
2627
# scripts/ci-local.sh --only <section> [--only <section> ...]
2728
# # Run one or more named sections. Mutually
@@ -31,10 +32,23 @@
3132
# # leak-scan → the public-repo leak scan ALONE
3233
# # (hygiene.yml's PR gate; also in gates)
3334
# # ts → ts build+typecheck, ts conformance,
34-
# # completeness-gate, full unit suites
35+
# # completeness-gate (changed files),
36+
# # full unit suites
3537
# # (ts-unit), integration-tests ts
3638
# # ts-fast → build+typecheck, conformance,
37-
# # completeness (the pure-code signal)
39+
# # completeness (the pure-code signal).
40+
# # Its mutation step mutates only the
41+
# # stryker.conf.mjs `mutate` files this
42+
# # change touches vs the base branch, and
43+
# # prints a SKIP when it touches none
44+
# # (MO_CI_MUTATION_SCOPE=full forces all;
45+
# # MO_CI_MUTATION_BASE sets the base ref)
46+
# # mutation → the FULL mutation pass over every
47+
# # `mutate` file (~10-45 min). For release
48+
# # prep and on demand; the flagless run
49+
# # and conformance.yml also run it. It
50+
# # replaces ts-fast's diff-scoped step
51+
# # when both are selected
3852
# # ts-unit → full per-package unit suites for
3953
# # metadata/render/runtime-ts + the
4054
# # client/web packages — runs in
@@ -136,8 +150,8 @@ while [ $# -gt 0 ]; do
136150
--no-integration) NO_INTEG=1 ;;
137151
--integration-only) INTEG_ONLY=1 ;;
138152
--only) shift; case "${1:-}" in
139-
gates|leak-scan|ts|ts-fast|ts-unit|ts-slow|java|java-fast|java-slow|python|csharp|cube) ONLY="$ONLY ${1}" ;;
140-
*) echo "--only expects gates|leak-scan|ts|ts-fast|ts-unit|ts-slow|java|java-fast|java-slow|python|csharp|cube, got '${1:-}'" >&2; exit 2 ;;
153+
gates|leak-scan|ts|ts-fast|ts-unit|ts-slow|mutation|java|java-fast|java-slow|python|csharp|cube) ONLY="$ONLY ${1}" ;;
154+
*) echo "--only expects gates|leak-scan|ts|ts-fast|ts-unit|ts-slow|mutation|java|java-fast|java-slow|python|csharp|cube, got '${1:-}'" >&2; exit 2 ;;
141155
esac ;;
142156
-h|--help) awk 'NR==1{next} /^set -uo/{exit} {sub(/^# ?/,""); print}' "$0"; exit 0 ;;
143157
*) echo "unknown arg: $1 (see --help)" >&2; exit 2 ;;
@@ -635,8 +649,9 @@ gate_conf_ts() {
635649
# conformance gate runs only NAMED conformance files in metadata/render, so the rest
636650
# of those suites (and runtime-ts + the client/web packages) ran nowhere — which is
637651
# how a @formExclude regression sat red on `main`. Deliberately its OWN lane rather
638-
# than folded into ts-fast: ts-fast carries the 30-min mutation gate, which flakes
639-
# under runner contention, and a mutation flake must not mask a unit-test failure.
652+
# than folded into ts-fast: ts-fast carries the mutation gate (diff-scoped since
653+
# 2026-11; the full pass took up to ~45 min and flaked under runner contention), and a
654+
# mutation flake must not mask a unit-test failure.
640655
# Measured ~5s total, so the independent signal is nearly free.
641656
# NOTE: per-package `bun test` only — a bare root `bun test` walks java/python/
642657
# csharp/fixtures and takes many minutes.
@@ -783,7 +798,87 @@ gate_java_reactor() {
783798
# themselves (two concurrent `mvn install`s clobber the shared module jars). When the reactor
784799
# step is not selected — `--integration-only` — this installs it without its tests.
785800
gate_java_install() { ( cd server/java && mvn -q -DskipTests -Djacoco.skip=true install ); }
786-
gate_completeness() { ( cd server/typescript/packages/metadata && bun run conformance:mutation ); }
801+
# ── The mutation (completeness) gate: two scopes over ONE file list ───────────
802+
# Stryker mutates the files listed in `mutate` in server/typescript/packages/metadata/
803+
# stryker.conf.mjs — the ONLY copy of that list; both scopes read it from there.
804+
#
805+
# FULL (gate_completeness): every listed file. ~10–45 min of CPU on a contended host.
806+
# Runs in the flagless full run's `mutation` lane, in `--only mutation` (release prep
807+
# and on demand), and wherever MO_CI_MUTATION_SCOPE=full is set — conformance.yml does,
808+
# so its tag (`v*`, the release gate), nightly and dispatch runs mutate everything.
809+
# DIFF (gate_completeness_diff): only the listed files this change touches. This is
810+
# the step in ts-fast / --quick / --only ts, and so in the no-mistakes gate's Test
811+
# step (.no-mistakes.yaml). It used to run the full pass on every gate round, which
812+
# was most of that step's wall time and most of the machine's load.
813+
#
814+
# "Touches" = differs between the merge base with the base branch and the working tree.
815+
# The base is MO_CI_MUTATION_BASE when set, else origin/$GITHUB_BASE_REF on a GitHub PR,
816+
# else whichever of origin/main and origin/release/* HEAD is fewest commits ahead of (a
817+
# branch cut from release/1.2 diffs against release/1.2, not main). When no base resolves
818+
# (a shallow checkout, no remote refs) the step says so and runs the FULL pass — it never
819+
# guesses a smaller scope. A change to stryker.conf.mjs itself also runs the full pass.
820+
# When none of the listed files changed it prints a SKIP banner naming the base and the
821+
# reason, and passes: there is nothing new to mutate, and the full lane still covers the
822+
# whole list. Thresholds and concurrency come from stryker.conf.mjs unchanged.
823+
#
824+
# Why --mutate on the diff rather than Stryker's incremental mode: incremental keeps a
825+
# results file between runs, and a cache shared across branches and worktrees can report
826+
# a mutant killed by tests another branch has. The diff scope keeps no state at all.
827+
#
828+
# Sandboxes: when DEV_SCRATCH names a directory (a RAM scratch area), Stryker's sandbox
829+
# goes there instead of the package's .stryker-tmp/, and is removed when the step ends.
830+
MUTATION_PKG="server/typescript/packages/metadata"
831+
stryker_run() { # stryker_run [stryker args...] — in the metadata package, sandbox in DEV_SCRATCH when set
832+
local tmp="" rc
833+
if [ -n "${DEV_SCRATCH:-}" ] && [ -d "${DEV_SCRATCH}" ]; then
834+
tmp="${DEV_SCRATCH%/}/stryker/metaobjects-$(printf '%s' "$ROOT" | cksum | cut -d' ' -f1)-$$"
835+
mkdir -p "$tmp" || return 1
836+
fi
837+
( cd "$MUTATION_PKG" && bun run conformance:mutation ${tmp:+--tempDirName "$tmp"} "$@" ); rc=$?
838+
[ -n "$tmp" ] && rm -rf "$tmp"
839+
return "$rc"
840+
}
841+
gate_completeness() { bun_install || return 1; stryker_run; }
842+
mutation_base() { # prints the base ref the diff scope compares against; empty when none resolves
843+
local ref best="" best_n="" mb n
844+
if [ -n "${MO_CI_MUTATION_BASE:-}" ]; then echo "$MO_CI_MUTATION_BASE"; return 0; fi
845+
if [ -n "${GITHUB_BASE_REF:-}" ]; then echo "origin/$GITHUB_BASE_REF"; return 0; fi
846+
for ref in origin/main $(git for-each-ref --format='%(refname:short)' 'refs/remotes/origin/release/*'); do
847+
git rev-parse --verify -q "$ref^{commit}" >/dev/null 2>&1 || continue
848+
mb="$(git merge-base HEAD "$ref" 2>/dev/null)" || continue
849+
n="$(git rev-list --count "$mb..HEAD")"
850+
if [ -z "$best_n" ] || [ "$n" -lt "$best_n" ]; then best="$ref"; best_n="$n"; fi
851+
done
852+
echo "$best"
853+
}
854+
gate_completeness_diff() {
855+
if [ "${MO_CI_MUTATION_SCOPE:-diff}" = "full" ]; then
856+
echo " mutation scope: FULL (MO_CI_MUTATION_SCOPE=full)"; gate_completeness; return
857+
fi
858+
local base mb files changed
859+
base="$(mutation_base)"
860+
if [ -z "$base" ] || ! mb="$(git merge-base HEAD "$base" 2>/dev/null)"; then
861+
echo " mutation scope: FULL — no merge base with '${base:-origin/main or origin/release/*}' resolves here (shallow checkout or no remote refs), so the changed files cannot be computed"
862+
gate_completeness; return
863+
fi
864+
files="$(cd "$MUTATION_PKG" && bun -e 'const c = (await import("./stryker.conf.mjs")).default; console.log(c.mutate.join("\n"))')" || {
865+
echo " ✖ could not read the mutate list from $MUTATION_PKG/stryker.conf.mjs" >&2; return 1; }
866+
if ! git diff --quiet "$mb" -- "$MUTATION_PKG/stryker.conf.mjs"; then
867+
echo " mutation scope: FULL — stryker.conf.mjs changed since $base (merge base ${mb:0:12})"
868+
gate_completeness; return
869+
fi
870+
# shellcheck disable=SC2086 # the mutate entries are plain paths, one per line
871+
changed="$(cd "$MUTATION_PKG" && git diff --name-only --relative "$mb" -- $files)"
872+
if [ -z "$changed" ]; then
873+
echo ""
874+
echo " ⊘ SKIP mutation: none of the $(echo "$files" | wc -l | tr -d ' ') files in stryker.conf.mjs 'mutate' changed since $base (merge base ${mb:0:12})."
875+
echo " The full pass runs in the flagless run's mutation lane, in --only mutation, and in conformance.yml."
876+
return 0
877+
fi
878+
echo " mutation scope: changed files only, vs $base (merge base ${mb:0:12}):"
879+
echo "$changed" | sed 's/^/ /'
880+
stryker_run --mutate "$(echo "$changed" | paste -sd, -)"
881+
}
787882
gate_doc_template_drift() {
788883
bash scripts/sync-doc-templates.sh \
789884
&& git diff --exit-code -- \
@@ -1185,7 +1280,8 @@ lane_ts() {
11851280
if want ts; then step_if bun "conformance: typescript" gate_conf_ts; fi
11861281
if want ts; then step_if bun "ts unit suites" gate_ts_unit; fi
11871282
}
1188-
lane_mutation() { if want ts; then step_if bun "completeness-gate (mutation)" gate_completeness; fi; }
1283+
# The full run's mutation lane is the FULL pass over every file stryker.conf.mjs lists.
1284+
lane_mutation() { if want ts; then step_if bun "completeness-gate (mutation, full)" gate_completeness; fi; }
11891285
# ts-slow's half: the two real-PG suites plus the TS integration suite.
11901286
lane_ts_integ() {
11911287
integ_ready || return 0
@@ -1284,13 +1380,19 @@ if [ "$PARALLEL" -eq 0 ]; then
12841380
run_gates_bun
12851381
# The ts port is split into two lanes so CI can run them as separate jobs (see
12861382
# .github/workflows/local-ci.yml): a FAST lane (build+typecheck, conformance,
1287-
# completeness — the pure-code correctness signal, no docker) and a SLOW lane
1383+
# completeness on the changed files — the pure-code correctness signal, no docker) and a SLOW lane
12881384
# (docker/sidecar integration). Isolating integration means a docker/sidecar
12891385
# hiccup no longer fails the code signal, and vice versa. `--only ts` still runs both.
12901386
if want_any ts ts-fast; then step_if bun "ts build + typecheck" gate_ts_build_typecheck; fi
12911387
if want_any ts ts-fast; then step_if bun "conformance: typescript" gate_conf_ts; fi
12921388
if want_any ts ts-unit; then step_if bun "ts unit suites" gate_ts_unit; fi
1293-
if want_any ts ts-fast; then step_if bun "completeness-gate (mutation)" gate_completeness; fi
1389+
# The DIFF-scoped mutation step (see gate_completeness_diff); `--only mutation` asks for
1390+
# the full pass instead, and gets it in place of this step rather than as well.
1391+
if [ -n "$ONLY" ] && want mutation; then
1392+
step_if bun "completeness-gate (mutation, full)" gate_completeness
1393+
elif want_any ts ts-fast; then
1394+
step_if bun "completeness-gate (mutation, changed files)" gate_completeness_diff
1395+
fi
12941396
run_drift_gates
12951397

12961398
if [ "$QUICK" -eq 1 ]; then

0 commit comments

Comments
 (0)