Skip to content

Commit ccb3b86

Browse files
committed
feat(build): restore the schedule foundation — the crash was in the graph split, not here
**修正一个错误归因。** 上一个提交把 `schedule/` 整批退回,理由是「改动后的 mcpp 段错误」。 重新施加后逐条复现:**基础层本身 rc=0**(mcpp 冷构建自身 82.27s,e2e 230/231/232 全过, policy 单测 8/8)。那两次崩溃用的二进制**都包含当时未提交的图拆分发射**—— 崩的是那部分,不是这里。回退整批是过度反应。 恢复的内容: * `schedule/policy.cppm` —— 纯函数决策表(clang→two-phase,gcc→detach-codegen, msvc→none),每条决策都带 reason;`requested_switch` / `resolve_jobs` 是唯一读开关 与并发的地方。 * `schedule/detach_codegen.cppm` —— gcc 运行期(阶段一/监督/阶段二), 实测阶段一在 **14%** 处返回、产物正确。 * 决策在 prepare 求值一次并落到 `BuildPlan`;图头记 `schedule=<tag>`; `--verbose` 打印理由;失效靠指纹而非守卫。 **`auto` 现在是 off**,`on` 才选择拆分形状。理由写进了 policy:调度改错是**静默**的 ——漏掉一条头文件依赖不会报错,只会不再重编——所以它不该凭一台机器的结果成为默认。 单测两侧钉住这一点(`auto` 必须 None,`on` 必须非 None), 否则「永远不启用」的实现也能通过。 图的拆分发射**不在本提交内**;它需要先按实测把 depfile 接到 P1689 扫描的产出上 (编译器自己的 depfile 写在 BMI 之后,挂哪边都会静默丢掉头文件跟踪), 并定位上一轮那次段错误。
1 parent e61f9ef commit ccb3b86

13 files changed

Lines changed: 934 additions & 47 deletions

‎src/build/execute.cppm‎

Lines changed: 6 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -377,46 +377,11 @@ compute_subos_env(const mcpp::build::BuildPlan& plan) {
377377
// so the latter doesn't call prepare_build twice (and re-print the toolchain
378378
// resolution banner).
379379
// How many compiles to run at once.
380-
//
381-
// Precedence: `--jobs` (arriving as MCPP_JOBS, same channel --offline uses and
382-
// for the same reason — the consumers span subsystems) > `[build] jobs` >
383-
// 0, which means "say nothing" and leaves ninja's own default (nproc + 2).
384-
// The default is deliberately unchanged: altering everyone's concurrency is a
385-
// behaviour change, and this lands as an opt-in first.
386-
//
387-
// `auto` is resolved HERE, against the machine doing the build, never frozen
388-
// into a manifest. Measured on this repository: the cold self-build takes
389-
// 81.0s at -j8 and 79.9s at -j32 — 4x the workers for 1.4%, because the build
390-
// is latency-bound. Meanwhile a single module compile peaks at 0.5-1.0 GB, so
391-
// the extra jobs are pure memory pressure; on a high-core, modest-RAM machine
392-
// ninja's default swaps.
393-
std::size_t resolve_parallel_jobs(const mcpp::build::BuildPlan& plan) {
394-
auto from_text = [&](std::string_view v) -> std::optional<std::size_t> {
395-
if (v.empty()) return std::nullopt;
396-
if (v == "auto") {
397-
const auto cap = mcpp::platform::capacity::host_capacity();
398-
return static_cast<std::size_t>(
399-
mcpp::platform::capacity::recommended_jobs(cap));
400-
}
401-
std::size_t n = 0;
402-
const auto* first = v.data();
403-
const auto* last = v.data() + v.size();
404-
if (auto [p, ec] = std::from_chars(first, last, n);
405-
ec == std::errc{} && p == last && n > 0)
406-
return n;
407-
// A malformed value must not silently become "use the default" — that
408-
// is how a typo turns into a build that is mysteriously slower.
409-
mcpp::ui::warning(std::format(
410-
"ignoring invalid job count '{}' (expected a positive number or 'auto')", v));
411-
return std::nullopt;
412-
};
413-
414-
if (const char* e = std::getenv("MCPP_JOBS"))
415-
if (auto n = from_text(e)) return *n;
416-
if (auto n = from_text(plan.manifest.buildConfig.jobs)) return *n;
417-
return 0;
418-
}
419-
380+
// Concurrency and the module-edge schedule are resolved together in
381+
// mcpp.build.schedule.policy and stamped onto the plan, so this reads one value
382+
// instead of re-deriving it. `scheduleNinjaJobs` is NOT the compiler cap under
383+
// detach-codegen: a detached compiler stops holding a ninja slot, so ninja is
384+
// handed a larger number on purpose.
420385
export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache,
421386
std::string_view targetOverride = "") {
422387
// `--cache=off` means a cold build: no global cache, and target/ cleared —
@@ -488,7 +453,7 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache,
488453

489454
mcpp::build::BuildOptions opts;
490455
opts.verbose = verbose;
491-
opts.parallelJobs = resolve_parallel_jobs(ctx.plan);
456+
opts.parallelJobs = static_cast<std::size_t>(ctx.plan.scheduleNinjaJobs);
492457
auto r = be->build(ctx.plan, opts);
493458
if (!r) {
494459
std::fflush(stdout);

‎src/build/graph_shape.cppm‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,15 @@ std::string_view to_string(GraphShape shape) {
4747

4848
// The marker line, without its newline. A ninja comment, so it costs nothing
4949
// and older ninja versions do not care.
50-
std::string header_line(GraphShape shape) {
51-
return std::format("# mcpp:graph={}", to_string(shape));
50+
// `scheduleTag` names the SHAPE OF THE MODULE EDGES (see
51+
// mcpp.build.schedule.policy): "none", "two-phase", "detach-codegen". It rides
52+
// the same line for the same reason the shape does — build.ninja is shared
53+
// mutable state and the fast path replays it, so a graph built under one
54+
// schedule must not be replayed under another. Flipping the switch has to
55+
// invalidate the graph, and the only way that cannot be forgotten is if the
56+
// graph says which schedule produced it.
57+
std::string header_line(GraphShape shape, std::string_view scheduleTag) {
58+
return std::format("# mcpp:graph={};schedule={}", to_string(shape), scheduleTag);
5259
}
5360

5461
// Read the shape back. `nullopt` means "this file does not say" — a build.ninja
@@ -68,6 +75,10 @@ std::optional<GraphShape> read_shape(const std::filesystem::path& ninjaPath) {
6875
auto value = std::string_view(line).substr(prefix.size());
6976
while (!value.empty() && (value.back() == '\r' || value.back() == ' '))
7077
value.remove_suffix(1);
78+
// `graph=<shape>[;schedule=<tag>]`. Split before comparing, so adding
79+
// the schedule field does not turn every existing graph into "unknown".
80+
if (const auto semi = value.find(';'); semi != std::string_view::npos)
81+
value = value.substr(0, semi);
7182
if (value == "normal") return GraphShape::Normal;
7283
if (value == "test") return GraphShape::WithTests;
7384
// A shape this binary does not know is not `Normal`. An older mcpp
@@ -77,7 +88,40 @@ std::optional<GraphShape> read_shape(const std::filesystem::path& ninjaPath) {
7788
return std::nullopt;
7889
}
7990

91+
// The schedule tag this graph was written with. Empty means the file predates
92+
// the field — which is NOT the same as "none": an unlabelled graph is exactly
93+
// the case that must not be replayed blind, so callers compare and miss.
94+
std::string read_schedule(const std::filesystem::path& ninjaPath) {
95+
std::ifstream input(ninjaPath);
96+
if (!input) return {};
97+
std::string line;
98+
for (int i = 0; i < 8 && std::getline(input, line); ++i) {
99+
constexpr std::string_view prefix = "# mcpp:graph=";
100+
if (!line.starts_with(prefix)) continue;
101+
auto value = std::string_view(line).substr(prefix.size());
102+
while (!value.empty() && (value.back() == '\r' || value.back() == ' '))
103+
value.remove_suffix(1);
104+
const auto semi = value.find(';');
105+
if (semi == std::string_view::npos) return {};
106+
auto rest = value.substr(semi + 1);
107+
constexpr std::string_view schedPrefix = "schedule=";
108+
if (!rest.starts_with(schedPrefix)) return {};
109+
return std::string(rest.substr(schedPrefix.size()));
110+
}
111+
return {};
112+
}
113+
80114
// The one question every fast path asks.
115+
//
116+
// It deliberately does NOT compare the schedule tag. The fast paths run BEFORE
117+
// a plan exists, so they have no toolchain to derive the expected schedule
118+
// from — and passing one in would mean deriving the same decision a second
119+
// time, in a place that cannot see the compiler.
120+
//
121+
// Instead the schedule SWITCH is part of the toolchain fingerprint, so flipping
122+
// it lands in a different build directory: a graph written under one schedule
123+
// is structurally unreachable from a build configured with another. The tag on
124+
// the line is then for humans and for `mcpp explain`, not for invalidation.
81125
bool is_plain_build_graph(const std::filesystem::path& ninjaPath) {
82126
return read_shape(ninjaPath) == GraphShape::Normal;
83127
}

‎src/build/ninja_backend.cppm‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -407,7 +407,7 @@ std::string emit_ninja_string(const BuildPlan& plan) {
407407
// #407: the graph declares which mode produced it, because three modes
408408
// write this one file and the fast path has to know what it is about to
409409
// replay. Must stay within the first few lines — see read_shape.
410-
append(mcpp::build::header_line(plan.graphShape) + "\n");
410+
append(mcpp::build::header_line(plan.graphShape, plan.scheduleTag) + "\n");
411411
append("ninja_required_version = 1.11\n\n");
412412

413413
// All compile/link flags are computed once via flags.cppm.

‎src/build/plan.cppm‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,18 @@ struct BuildPlan {
145145
// share an output directory and overwrite each other's graph; this is what
146146
// lets a fast path tell them apart (mcpp#407, mcpp.build.graph_shape).
147147
GraphShape graphShape = GraphShape::Normal;
148+
// The module-edge schedule this plan will emit, resolved ONCE (see
149+
// mcpp.build.schedule.policy). The backend writes the graph in this shape,
150+
// the graph records the tag, and the fast path compares against it — three
151+
// readers, one derivation. Deriving it separately in the backend and in the
152+
// executor is how the BMI-equivalence check and the job count drifted into
153+
// disagreeing about what a module edge is.
154+
std::string scheduleTag = "none";
155+
// What to hand ninja. Under detach-codegen a compiler stops holding a slot
156+
// when it publishes, so this must exceed the real compiler cap or the ready
157+
// frontier starves — see the hazard note in schedule/detach_codegen.
158+
int scheduleNinjaJobs = 0;
159+
int scheduleCompilerCap = 0;
148160
// One immutable snapshot selected before workspace member substitution.
149161
// Build/run/test and cache fast paths consume this value; none may re-read
150162
// xlings active/current state.

‎src/build/prepare.cppm‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ import mcpp.toolchain.post_install;
3333
import mcpp.toolchain.abi;
3434
import mcpp.toolchain.triple;
3535
import mcpp.build.plan;
36+
import mcpp.build.schedule.policy;
3637
import mcpp.build.graph_shape; // #407: the graph says which mode wrote it
3738
import mcpp.build.runtime_validation; // declared artifact -> identity verdict
3839
import mcpp.build.cache_key;
@@ -5226,6 +5227,17 @@ prepare_build(bool print_fingerprint,
52265227
fpi.cppStandard = m->package.standard;
52275228
fpi.compileFlags = canonical_compile_flags(*m)
52285229
+ canonical_package_build_metadata(packages);
5230+
// The module-edge schedule changes the SHAPE of build.ninja, and the fast
5231+
// path replays that file without a plan to compare against. Folding the
5232+
// switch into the fingerprint puts a differently-scheduled build in a
5233+
// different directory, which makes replaying the wrong shape structurally
5234+
// impossible instead of merely guarded. Only appended when non-default, so
5235+
// existing build directories keep their identity.
5236+
if (const auto sched = mcpp::build::schedule::requested_switch(*m);
5237+
sched != "auto") {
5238+
fpi.compileFlags += " #schedule=";
5239+
fpi.compileFlags += sched;
5240+
}
52295241
if (m->cppStandard.experimental) {
52305242
// c++fly gate flags are derived (not manifest-declared): fold them in
52315243
// so a cppfly table change across mcpp versions re-fingerprints.
@@ -5311,6 +5323,24 @@ prepare_build(bool print_fingerprint,
53115323
ctx.plan.graphShape = (includeDevDeps || !extraTargets.empty())
53125324
? mcpp::build::GraphShape::WithTests
53135325
: mcpp::build::GraphShape::Normal;
5326+
// Resolve the module-edge schedule ONCE, here, where both the toolchain and
5327+
// the manifest are in hand. The backend writes the graph in this shape, the
5328+
// graph records the tag, and `mcpp build --verbose` prints the reason — all
5329+
// three read this, none of them re-derives it.
5330+
{
5331+
const auto decision = mcpp::build::schedule::decide(
5332+
ctx.plan.toolchain,
5333+
mcpp::build::schedule::requested_switch(*m),
5334+
mcpp::build::schedule::resolve_jobs(*m, [](std::string_view bad) {
5335+
mcpp::ui::warning(std::format(
5336+
"ignoring invalid job count '{}' (expected a positive number or 'auto')", bad));
5337+
}));
5338+
ctx.plan.scheduleTag = std::string(mcpp::build::schedule::to_string(decision.strategy));
5339+
ctx.plan.scheduleNinjaJobs = decision.ninjaJobs;
5340+
ctx.plan.scheduleCompilerCap = decision.compilerCap;
5341+
mcpp::log::verbose("build", std::format("schedule: {} — {}",
5342+
ctx.plan.scheduleTag, decision.reason));
5343+
}
53145344
ctx.plan.runtimeBinding = runtimeBindingSnapshot;
53155345
mcpp::build::merge_runtime_binding_contract(
53165346
ctx.plan, runtimeBindingSnapshot);

0 commit comments

Comments
 (0)