Skip to content

Commit 1bb5ea3

Browse files
authored
chore: PR #759 — CI acceleration, SPEC-009, and the repairs of #756 and #757 (#759)
* PR CI acceleration (SPEC-009 baked in): one build of mcpp per host, a single stage-of-work per host workflow, single-writer caches, four timed Linux shards, coverage check + every-e2e-test-runs-somewhere gate. * Repairs of #757 (engine field in .build_cache, replay declines a record whose engine is not the running one) and #756 (each path-dependency root is recorded and swept by the extension table of its own package). * Release at 2026.10.3.1 (this PR's version, post the engine-binding bump to 2026.10.2.1). The engine field is a breaking change for old caches. * Closes #757, #756.
1 parent 4d81d06 commit 1bb5ea3

52 files changed

Lines changed: 5190 additions & 901 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md‎

Lines changed: 1039 additions & 0 deletions
Large diffs are not rendered by default.

‎.agents/docs/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded
1818
---
1919
```
2020

21-
323 records.
21+
324 records.
2222

2323
## By subject
2424

@@ -30,6 +30,7 @@ Records that declare one. Everything else is listed by date below.
3030

3131
### design
3232

33+
- [PR CI acceleration and the toolchain specification (#756, #757, #669)](2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md) — active
3334
- [工具与工具链的来源:声明、编程决定、可观察](2026-10-01-tool-and-toolchain-sources-design.md) — landed
3435
- [A pack's build reported as a build, and a unit's compile independent of the member selection: triage and design (#753, #751)](2026-10-01-pack-drive-and-selection-independent-compile-design.md) — landed
3536
- [Member selection, build programs prepared once, a pack over several members, and the output streams of `mcpp run`: the plan for the release after 2026.9.30.2 (#748, #749, #750)](2026-09-30-member-selection-and-build-program-cost-plan.md) — landed
@@ -114,6 +115,7 @@ Records that declare one. Everything else is listed by date below.
114115

115116
### 2026-10
116117

118+
- [PR CI acceleration and the toolchain specification (#756, #757, #669)](2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md) — active
117119
- [工具与工具链的来源:声明、编程决定、可观察](2026-10-01-tool-and-toolchain-sources-design.md) — landed
118120
- [A pack's build reported as a build, and a unit's compile independent of the member selection: triage and design (#753, #751)](2026-10-01-pack-drive-and-selection-independent-compile-design.md) — landed
119121
### 2026-09

‎.agents/skills/mcpp-contributing/SKILL.md‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -182,14 +182,16 @@ gh pr checks <pr-number> # 查看状态
182182
gh run view <run-id> --log-failed # 查看失败日志
183183
```
184184

185-
CI 由分平台的基础构建/单元集成检查与独立 E2E 检查组成:
186-
| Workflow | 平台 | 内容 |
187-
|----------|------|------|
188-
| `ci-linux` / `ci-linux-e2e` | Linux x86_64 | 自举构建、unit/integration / 分片 E2E |
189-
| `ci-macos` / `ci-macos-e2e` | macOS ARM64 | 自举构建、unit/integration / E2E |
190-
| `ci-windows` / `ci-windows-e2e` | Windows x86_64 | 自举构建、toolchain 回归 / E2E |
191-
| `cross-build-test` | Linux/Windows cross targets | 交叉构建、产物运行与 MinGW/Wine 检查 |
192-
| `ci-aarch64-fresh-install` | Linux ARM64 native | path-filtered fresh install、原生自举与 musl `build.mcpp` host-helper 回归 |
185+
一次提交的 CI 是 `ci.yml` 的一次运行,分段执行(设计见 `.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md` 第二部分):
186+
| 阶段 | 内容 |
187+
|------|------|
188+
| `changes` | 按改动路径分类;只改了没有任何脚本、测试或源码读取的文档时,只跑 `docs` |
189+
| `docs` | 不需要二进制的检查(版本钉、文档风格与结构、工作流断言等) |
190+
| `build-*` | 每个宿主构建一次 mcpp(`build.yml`),上传为 `mcpp-built-<host>` |
191+
| `linux` / `linux-e2e`、`macos` / `macos-e2e` / `macos-ios`、`windows` / `windows-e2e` / `windows-msvc-xlings`、`cross`、`target-matrix`、`openkal` | 各领域的可复用工作流(原来的 `ci-*.yml`),通过 `.github/actions/use-built-mcpp` 使用上面那次构建,不再各自构建 |
192+
| `e2e-coverage` | 每个 e2e 测试都在某个宿主上运行、由专门 job 运行,或在 `tests/e2e/coverage-exceptions.tsv` 中写明原因 |
193+
194+
`ci-aarch64-fresh-install`、`measure-windows-tool-crt` 与 `pypi-publish` 仍是按路径触发的独立工作流。缓存只在 main 上由一个 job 保存;PR 只恢复。
193195

194196
**以 PR 实际 required checks 为准,所有未跳过的 required checks 必须通过。** 如果某个平台失败:
195197
1. 下载日志分析原因

‎.agents/skills/mcpp-release/SKILL.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,9 @@ gh run list --branch main --limit 3
7373
```
7474

7575
以分支保护和 `gh pr checks <pr-number>` 显示的 actual required checks 为准。
76-
在 main 上监控当前运行时,检查 `ci-linux`、`ci-linux-e2e`、`ci-macos`、
77-
`ci-macos-e2e`、`ci-windows`、`ci-windows-e2e` 与 `cross-build-test` 的结果;
78-
跳过或非 required 的 workflow 不是合入 gate。不要在 required CI 红的时候发版。
76+
在 main 上监控当前运行时,检查 `ci` 这一个工作流的运行(它包含各平台、e2e 分片、
77+
交叉构建与 `e2e-coverage`);known-red 的腿(名字里带 issue 号)允许失败。跳过或
78+
非 required 的 workflow 不是合入 gate。不要在 `ci` 红的时候发版。
7979

8080
### 2. bump 版本号(第一组两处,单个 commit,走 PR)
8181

‎.github/actions/bootstrap-mcpp/action.yml‎

Lines changed: 101 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,21 @@
11
name: bootstrap-mcpp
22
description: >
3-
Restore the shared CI cache lineage (mcpp sandbox + xlings + target/) and
4-
bootstrap a released mcpp via xlings. Exports MCPP and XLINGS_BIN.
3+
Restore the shared CI cache lineage (mcpp sandbox and xlings) and bootstrap
4+
a released mcpp via xlings. Exports MCPP and XLINGS_BIN.
55
6-
Extracted so the split CI jobs (build / toolchain legs / e2e shards /
7-
integration) share ONE definition instead of copy-pasting a 40-line
8-
preamble per job. Every job that uses it lands on the same cache keys,
9-
which is what makes splitting cheap: each job restores a warm sandbox
10-
and only pays one incremental `mcpp build`.
6+
The caches are RESTORED here and never saved. One job per host writes them,
7+
the build job of .github/workflows/build.yml, and only on a push to main
8+
(rule R3 of .agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-
9+
specification-design.md). Before that rule every job that used this action
10+
saved the same key on success: thirty-four to forty-two saves per pull
11+
request against a 10 GB repository limit, parallel saves of one key racing
12+
each other, and the main lineage evicted within forty minutes (measured
13+
2026-10-01). The keys are outputs so that the one writer saves exactly what
14+
was restored.
15+
16+
`target/` is no longer cached. A restored `target/` made no build
17+
incremental: on an exact hit ninja still ran 830 of 830 edges, while the
18+
caches themselves were up to 3.3 GB each.
1119
1220
inputs:
1321
xlings-version:
@@ -26,10 +34,20 @@ inputs:
2634
# depended on the machine, which is why CI failed on `compat:lua` on
2735
# Windows and `mcpplibs.capi:lua` on Linux. Never pin below that.
2836
default: '2026.9.30.1'
29-
cache-target:
30-
description: also restore/save target/ (build artifacts + BMIs)
31-
required: false
32-
default: 'true'
37+
38+
outputs:
39+
sandbox-key:
40+
description: the exact key of the mcpp sandbox cache
41+
value: ${{ steps.sandbox.outputs.cache-primary-key }}
42+
sandbox-hit:
43+
description: "'true' when the sandbox was restored by its exact key"
44+
value: ${{ steps.sandbox.outputs.cache-hit }}
45+
xlings-key:
46+
description: the exact key of the xlings cache
47+
value: ${{ steps.xlings.outputs.cache-primary-key }}
48+
xlings-hit:
49+
description: "'true' when xlings was restored by its exact key"
50+
value: ${{ steps.xlings.outputs.cache-hit }}
3351

3452
runs:
3553
using: composite
@@ -38,8 +56,9 @@ runs:
3856
# "-release-" caches. A bare "mcpp-sandbox-<os>-" restore prefix used to
3957
# match the release sandbox too, silently swapping in a differently
4058
# populated registry (issue #120).
41-
- name: Cache mcpp sandbox
42-
uses: actions/cache@v4
59+
- name: Restore the mcpp sandbox
60+
id: sandbox
61+
uses: actions/cache/restore@v4
4362
with:
4463
path: ~/.mcpp
4564
# `runner.arch` IS PART OF EVERY KEY, AND WAS NOT.
@@ -66,12 +85,16 @@ runs:
6685
# sandbox — which is what actually resolves dependencies — would
6786
# silently stay behind (observed: a 0.4.30 sandbox surviving under a
6887
# 0.4.69 bootstrap for weeks).
69-
key: mcpp-sandbox-${{ runner.os }}-${{ runner.arch }}-ci-xl${{ inputs.xlings-version }}-${{ hashFiles('mcpp.toml', '.xlings.json') }}
88+
# ci.yml is part of the key because it names the toolchains the build
89+
# job installs before it saves (`prewarm`): a sandbox saved under an
90+
# unchanged key would never gain one added there.
91+
key: mcpp-sandbox-${{ runner.os }}-${{ runner.arch }}-ci-xl${{ inputs.xlings-version }}-${{ hashFiles('mcpp.toml', '.xlings.json', '.github/workflows/ci.yml') }}
7092
restore-keys: |
7193
mcpp-sandbox-${{ runner.os }}-${{ runner.arch }}-ci-xl${{ inputs.xlings-version }}-
7294
73-
- name: Cache xlings
74-
uses: actions/cache@v4
95+
- name: Restore xlings
96+
id: xlings
97+
uses: actions/cache/restore@v4
7598
with:
7699
path: ~/.xlings
77100
key: xlings-${{ runner.os }}-${{ runner.arch }}-v2-xl${{ inputs.xlings-version }}-${{ hashFiles('.xlings.json') }}
@@ -109,6 +132,37 @@ runs:
109132
esac
110133
tarball="xlings-${XLINGS_VERSION}-linux-${xa}.tar.gz" ;;
111134
esac
135+
# FAST PATH: the xlings cache already holds the pinned version, so the
136+
# tarball fetch + extract + `self install` are skipped. Without this
137+
# guard every job of every CI run paid the download and the extract,
138+
# measured at 5 to 30 seconds per job on Linux and ~30 seconds on
139+
# Windows, across thirty jobs per run — about half the bootstrap-mcpp
140+
# step on Windows, more the 7 seconds the unix leg pays. The cache key
141+
# is `xl$VER` already; the check is the one case this guard would
142+
# otherwise miss: a stale `xlings` cache from BEFORE the pin was
143+
# bumped (a partial restore-key match hands the same OS/ARCH cache
144+
# back, but with the previous release), or a binary that no longer runs
145+
# because its dynamic loader is gone.
146+
XL_BIN_PATH="$HOME/.xlings/subos/default/bin/xlings"
147+
if [ -x "$XL_BIN_PATH" ]; then
148+
xl_ver="$("$XL_BIN_PATH" --version 2>/dev/null | head -1 || true)"
149+
if [ -n "$xl_ver" ] && echo "$xl_ver" | grep -qF "$XLINGS_VERSION"; then
150+
export PATH="$HOME/.xlings/subos/default/bin:$PATH"
151+
echo "$HOME/.xlings/subos/default/bin" >> "$GITHUB_PATH"
152+
"$XL_BIN_PATH" --version
153+
MCPP=$(bash "$REPO_DIR/.github/tools/install_pinned_mcpp.sh" "$REPO_DIR")
154+
echo "system xlings: $("$XL_BIN_PATH" --version 2>/dev/null | head -1)"
155+
if [ -x "$HOME/.mcpp/registry/bin/xlings" ]; then
156+
echo "sandbox xlings: $("$HOME/.mcpp/registry/bin/xlings" --version 2>/dev/null | head -1)"
157+
else
158+
echo "sandbox xlings: (not initialised yet)"
159+
fi
160+
echo "MCPP=$MCPP" >> "$GITHUB_ENV"
161+
echo "XLINGS_BIN=$XL_BIN_PATH" >> "$GITHUB_ENV"
162+
exit 0
163+
fi
164+
fi
165+
112166
WORK=$(mktemp -d)
113167
# Retried and verified — see .github/tools/fetch_release.sh. A bare curl
114168
# here was the single largest source of unexplained CI red on this repo
@@ -179,6 +233,37 @@ runs:
179233
XLINGS_VERSION: ${{ inputs.xlings-version }}
180234
run: |
181235
REPO_DIR="$(pwd)"
236+
# FAST PATH: see the unix leg for the reasoning. The cost on Windows is
237+
# larger (the zip is bigger and the runner's network path to
238+
# github.com is slower), measured at ~30 s per job.
239+
#
240+
# The fast path addresses xlings by ABSOLUTE PATH, not by bare
241+
# `xlings.exe`. The cold path runs `xlings self install` first, which
242+
# writes the dir into Windows PATH via `[Environment]::SetEnvironmentVariable`;
243+
# without that step a bare `xlings.exe` call depends on the bash
244+
# export PATH, which Git Bash re-derives from Windows on every child
245+
# shell and drops the mixed-separator entry. install_pinned_mcpp.sh
246+
# carries the same note.
247+
XL_BIN_PATH="$USERPROFILE/.xlings/subos/default/bin/xlings.exe"
248+
if [ -x "$XL_BIN_PATH" ]; then
249+
xl_ver="$("$XL_BIN_PATH" --version 2>/dev/null | head -1 || true)"
250+
if [ -n "$xl_ver" ] && echo "$xl_ver" | grep -qF "$XLINGS_VERSION"; then
251+
export PATH="$USERPROFILE/.xlings/subos/default/bin:$PATH"
252+
echo "$USERPROFILE/.xlings/subos/default/bin" >> "$GITHUB_PATH"
253+
"$XL_BIN_PATH" --version
254+
MCPP=$(bash "$REPO_DIR/.github/tools/install_pinned_mcpp.sh" "$REPO_DIR")
255+
echo "system xlings: $("$XL_BIN_PATH" --version 2>/dev/null | head -1)"
256+
if [ -x "$USERPROFILE/.mcpp/registry/bin/xlings.exe" ]; then
257+
echo "sandbox xlings: $("$USERPROFILE/.mcpp/registry/bin/xlings.exe" --version 2>/dev/null | head -1)"
258+
else
259+
echo "sandbox xlings: (not initialised yet)"
260+
fi
261+
echo "MCPP=$MCPP" >> "$GITHUB_ENV"
262+
echo "XLINGS_BIN=$(cygpath -w "$XL_BIN_PATH")" >> "$GITHUB_ENV"
263+
exit 0
264+
fi
265+
fi
266+
182267
WORK=$(mktemp -d)
183268
zipfile="xlings-${XLINGS_VERSION}-windows-x86_64.zip"
184269
# Same helper as the unix leg. This is the leg that kept failing, and a
@@ -207,17 +292,3 @@ runs:
207292
# Precise key on src/ + manifest so a no-source-change run lands on a full
208293
# hit; layered restore-keys let partial hits keep BMI/dyndep state for a
209294
# proper incremental build.
210-
- name: Cache target/ (build artifacts + BMIs)
211-
if: inputs.cache-target == 'true'
212-
uses: actions/cache@v4
213-
with:
214-
path: target
215-
# `modules/**` belongs here as much as `src/**` does. mcpp's own
216-
# source lives in both since the subsystem split, and a key that hashed
217-
# only one of them would restore a target/ built from different sources
218-
# and report success — the failure mode a cache key exists to prevent,
219-
# arriving silently.
220-
key: mcpp-target-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ hashFiles('src/**', 'modules/**', 'tests/**', 'mcpp.toml', 'mcpp.lock') }}
221-
restore-keys: |
222-
mcpp-target-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-
223-
mcpp-target-${{ runner.os }}-${{ runner.arch }}-

‎.github/actions/setup-macos-llvm/action.yml‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,22 @@ inputs:
2424
required: false
2525
default: 'macos-15'
2626

27+
outputs:
28+
xlings-key:
29+
description: the exact key of the xlings cache
30+
value: ${{ steps.xlings.outputs.cache-primary-key }}
31+
xlings-hit:
32+
description: "'true' when xlings was restored by its exact key"
33+
value: ${{ steps.xlings.outputs.cache-hit }}
34+
2735
runs:
2836
using: composite
2937
steps:
30-
- name: Cache xlings
31-
uses: actions/cache@v4
38+
# Restored, never saved here: the macOS build job of
39+
# .github/workflows/build.yml is the one writer, on a push to main (rule R3).
40+
- name: Restore xlings
41+
id: xlings
42+
uses: actions/cache/restore@v4
3243
with:
3344
path: ~/.xlings
3445
key: xlings-${{ inputs.image }}-arm64-v3-xl${{ inputs.xlings-version }}-${{ hashFiles('.xlings.json') }}
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
name: use-built-mcpp
2+
description: >
3+
Put this commit's mcpp, built once per host by .github/workflows/build.yml,
4+
in place of a build of the job's own (rule R1 of
5+
.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md).
6+
7+
Run it after bootstrap-mcpp or setup-macos-llvm, which restore the sandbox
8+
and install the released bootstrap. It exports MCPP_BOOT (that bootstrap),
9+
MCPP and MCPP_FRESH (this commit's binary, at an absolute path), and
10+
MCPP_VENDORED_XLINGS, and sets the given mirror on xlings and on the binary.
11+
12+
The binary is the one the build job produced, not a repackaging of it, so
13+
every consumer tests what a self-host build makes. On Linux that binary's
14+
interpreter and runtime libraries live in payloads of the sandbox (glibc and
15+
the GCC runtime of the toolchain mcpp.toml names). A restored sandbox holds
16+
them; when it does not, the bootstrap installs that toolchain and the binary
17+
is run again. A binary that still does not run fails this step.
18+
19+
The toolchain mcpp.toml names for the host is then installed with the
20+
binary, as a job that built mcpp used to install it as a side effect.
21+
22+
inputs:
23+
host:
24+
description: >
25+
The host the artifact was built on: linux-x86_64, linux-aarch64,
26+
macos-arm64 or windows-x86_64.
27+
required: true
28+
mirror:
29+
description: The mirror xlings and mcpp use in this job.
30+
required: false
31+
default: GLOBAL
32+
33+
runs:
34+
using: composite
35+
steps:
36+
- name: Download this commit's mcpp (${{ inputs.host }})
37+
uses: actions/download-artifact@v4
38+
with:
39+
name: mcpp-built-${{ inputs.host }}
40+
path: ${{ runner.temp }}/mcpp-built
41+
42+
- name: Use this commit's mcpp
43+
shell: bash
44+
run: bash "$GITHUB_ACTION_PATH/use.sh" "${{ inputs.host }}" "${{ inputs.mirror }}"
Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
#!/usr/bin/env bash
2+
# The body of the use-built-mcpp action; see action.yml for what it provides.
3+
#
4+
# Usage: use.sh <host> <mirror>
5+
set -euo pipefail
6+
7+
host="$1"
8+
mirror="$2"
9+
10+
dir="$RUNNER_TEMP/mcpp-built"
11+
case "$host" in
12+
windows-*) exe=mcpp.exe; dir="$(cygpath -u "$dir")" ;;
13+
*) exe=mcpp ;;
14+
esac
15+
bin="$dir/$exe"
16+
if [ ! -f "$bin" ]; then
17+
echo "::error::the artifact mcpp-built-$host holds no $exe"
18+
ls -la "$dir" || true
19+
exit 1
20+
fi
21+
chmod +x "$bin"
22+
23+
boot="${MCPP:-}"
24+
if [ -z "$boot" ]; then
25+
echo "::error::MCPP is unset: run bootstrap-mcpp or setup-macos-llvm before use-built-mcpp"
26+
exit 1
27+
fi
28+
29+
# The toolchain mcpp.toml names for this host is the one the build used, so it
30+
# is the one whose payloads hold the binary's runtime.
31+
manifest_toolchain() {
32+
local key
33+
case "$host" in
34+
macos-*) key=macos ;;
35+
windows-*) key=windows ;;
36+
*) key=default ;;
37+
esac
38+
awk -v k="$key" '
39+
/^\[/ { in_tc = ($0 == "[toolchain]") ; next }
40+
in_tc && $1 == k { gsub(/"/, "", $3); print $3; exit }
41+
' mcpp.toml
42+
}
43+
44+
if ! out=$("$bin" --version 2>&1); then
45+
tc="$(manifest_toolchain)"
46+
echo "this commit's mcpp does not run yet ($out); installing ${tc:-the default toolchain} with the bootstrap"
47+
if [ -n "$tc" ]; then
48+
"$boot" toolchain install "${tc%@*}" "${tc#*@}"
49+
fi
50+
if ! out=$("$bin" --version 2>&1); then
51+
echo "::error::this commit's mcpp does not run on this runner: $out"
52+
exit 1
53+
fi
54+
fi
55+
echo "this commit's mcpp: $out ($bin)"
56+
57+
# The mirror first: the runners are outside CN, and the install below and
58+
# every later download read it.
59+
if [ -n "${XLINGS_BIN:-}" ]; then
60+
"$XLINGS_BIN" config --mirror "$mirror" 2>/dev/null || true
61+
fi
62+
MCPP_VENDORED_XLINGS="${XLINGS_BIN:-}" "$bin" self config --mirror "$mirror"
63+
64+
# THE TOOLCHAIN THE BUILD USED IS INSTALLED, AS IT WAS WHEN EVERY JOB BUILT.
65+
# A job that built mcpp itself installed this toolchain as a side effect, and
66+
# the steps after the build relied on it without saying so: measured on the
67+
# first run of this action, the aarch64 leg of the target matrix restored no
68+
# sandbox, its binary ran without any payload, and the invariants that list the
69+
# host's toolchains found none ("gcc is not installed here"). Installing it here
70+
# keeps every consumer's environment what it was. It is a lookup when the
71+
# toolchain is present.
72+
tc="$(manifest_toolchain)"
73+
if [ -n "$tc" ]; then
74+
MCPP_VENDORED_XLINGS="${XLINGS_BIN:-}" "$bin" toolchain install "${tc%@*}" "${tc#*@}"
75+
fi
76+
77+
{
78+
echo "MCPP_BOOT=$boot"
79+
echo "MCPP=$bin"
80+
echo "MCPP_FRESH=$bin"
81+
if [ -n "${XLINGS_BIN:-}" ]; then echo "MCPP_VENDORED_XLINGS=$XLINGS_BIN"; fi
82+
} >> "$GITHUB_ENV"

0 commit comments

Comments
 (0)