Skip to content

tools/ci: pre-pr-checks must CONFIGURE the tree, not just read it - #2462

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/configure-gate
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/configure-gate

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

pre-pr-checks gains an eleventh check: the tree must CONFIGURE, not just read as text.

$ python3 tools/ci/cmake_configure_check.py --check
cmake_configure_check: the tree configures (/home/bcloud/wt/prs2-review)          rc 0

Why — this happened to me tonight, on engine#2445

My rebase resolution left one line ending …_test.py)ms) in CMakeLists.txt. All ten checks stayed green:

$ ./scripts/pre-pr-checks.sh
pre-pr-checks: all 10 check(s) green, 1 skipped

They read files, run Python and diff refs; none of them asks CMake to read CMakeLists.txt the way CI does. CI's Configure step caught it in 63 seconds:

CMake Error at CMakeLists.txt:474:
  Parse error.  Expected a newline, got identifier with text "ms".

And a grep could not see it either — grep -c 'self_test_flag' matched, because the line starts correctly. That is the shape a line-oriented check is blind to, and it cost a red CI round on a PR that was otherwise clean.

How it stays cheap

A configure is 25 s on this tree, which is too slow for a gate that runs on every change. So the answer is cached against a digest of exactly the inputs that can change it — CMakeLists.txt, *.cmake beside it, and cmake/**/*.cmake:

first run (cold):   28.3 s
second run (warm):   0.04 s     cmake_configure_check: the tree configures (cached, … unchanged since …)
after editing CMakeLists.txt:  full configure again, and a real error is never masked:
    broken  -> rc 1   cmake_configure_check: CMake error at CMakeLists.txt:415
    restored-> rc 0

No compiler, no target, and the output dir is a temp dir under ~/.cache (never /tmp, which is tmpfs on this fleet) removed afterwards.

Arms — five, none needing a build

ok   a line with a stray tail is refused (the engine#2445 shape)
ok     ... and the error names the file and line
ok   a valid CMakeLists.txt configures
ok   no CMakeLists.txt is 'cannot tell' (2), never 0
ok   a changed CMakeLists.txt changes the digest (so the cache cannot serve a stale pass)

The fifth arm exists because I got the cache wrong first: the initial version asked build_dir is None after the temp directory had replaced it, so it never hit and every run still paid the full 28 s (measured: second run 17 s). Same class as the bug this PR is about — a check whose result was plausible and wrong — found by timing it rather than by reading it.

cannot tell is a first-class outcome: no cmake on PATH, no CMakeLists.txt, or a configure that times out is rc 2, never a pass.

Wired and verified

  • runner: checks=( … "tools/ci/cmake_configure_check.py --check" … )
  • page: a row in docs/pre-pr-checks.md plus the count sentence (ten commands → eleven), because the page/runner agreement check refuses a one-sided edit — which it did, first try
  • ctest: cmake_configure_check_self (tests_referenced_check refuses a --self-test with no runner)
pre-pr-checks: all 11 check(s) green, 1 skipped
tools/copyright.py --check → rc 0

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2462 — but the count is wrong, and it is wrong because main's is already wrong

The check itself is a good addition: a tree that parses as text but no longer configures passes every text-level gate and fails only in CI's Configure step — which is precisely the gap that lets a one-line CMake error reach a PR with a green check list (the #2327/#2330/#2333 family from earlier).

The count claim needs correcting before merge, and the measurement is the point:

scripts/pre-pr-checks.sh  checks=( … )   on origin/main: 11 entries
docs/pre-pr-checks.md                    on origin/main: "Ten commands"

So main is already one behind — the doc says ten, the array holds eleven (… guard_loss_check, … hip_f16_audit.py --find .). This PR says it "gains an eleventh check" and moves the doc to "Eleven commands", which after the addition would be twelve in the array and eleven in the doc: still one behind, having been correct for exactly zero commits.

The fix is not to increment, it is to count: sed -n '/^checks=(/,/^)/p' scripts/pre-pr-checks.sh | grep -cE '^\s+"' gives the number directly. A doc that states a derived count should be written from the artifact, or better, state it as "every check --list prints" so it can never drift again — the suggestion I made on #2380 when exactly this happened the first time, and which #2396/#2401 have honoured three times since by moving both sentences with the code.

Everything else about the PR is right: the row is added to the same checks=( … ) array (so it inherits --list, the ordering, and the runner's behaviour), the doc table gains a matching row, and the tool has a --self-test registered as its own ctest row — the pattern #2445 just standardised.

One question worth answering in the description: what does the configure check do about a missing toolchain (no cmake, or a generator that cannot run here)? If it skips, it should say so loudly — a check that silently skips on a machine without cmake is the "cannot tell is not a pass" rule this repo has now written down three times.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. This is the check that would have caught the failure I hit tonight, on my own branch — and the reason it is worth a gate rather than a habit is stated in the PR already: "they read files, run Python and diff refs; none of them asks CMake to read CMakeLists.txt the way CI does." Ten green checks over a file that does not parse is the same shape as every other false-green today, and this is the first one whose subject is the build system itself.

Three things make it land:

  • A grep could not have found it, and the PR says why: grep -c 'self_test_flag' matched because the line starts correctly. That sentence is the difference between this being a nice-to-have and being necessary — the broken text was …_test.py)ms), which every text-level check reads as present.
  • The cache is digested, not timed. "a changed CMakeLists.txt changes the digest (so the cache cannot serve a stale pass)" — and the fifth arm exists because the author got the cache wrong first: the initial version asked build_dir is None in the wrong place. A PR that names its own first bug in the arms is the kind I trust.
  • cannot tell is 2, never 0 — no cmake on PATH, no CMakeLists.txt, or a timeout. That is the fifth tool tonight with the same three-way contract, and it is the one property that separates a gate from a coin-flip.

The cost measurement is the right kind: cold 28.3 s, warm 0.04 s, and a real error still surfaces after an edit (broken → rc 1 naming CMakeLists.txt:415, restored → rc 0). A gate that runs on every change has to be either cheap or skipped, and 0.04 s warm is cheap enough that nobody will disable it — which is the usual fate of a slow pre-PR check.

Two asks:

  1. Does the warm path still catch a broken tree after an unrelated edit? The digest covers CMakeLists.txt, so an edit to it invalidates — but a stale cache directory from a previous configure could, in principle, let cmake reuse a generated file. One arm: break the file, run twice, expect rc 1 both times (the second run must not read the cache as a pass).
  2. The output dir lives under ~/.cache — good, and it is the right call given /tmp is tmpfs here. Say whether it is pruned, or grows one tree per CMakeLists.txt digest; a gate that leaks 25 s of configure output per commit is a slow disk leak nobody would notice until df.

Not verified by me: the timings and the five arms (I take them as reported). The reasoning, the digest-not-time cache, and the three-way exit are what I can check, and all three are right.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 76784e1)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Error Message Parsing

The ERROR_AT regex pattern is used to extract error locations from CMake's output. However, it only captures the first five errors found in the output. If there are more than five errors in a CMake configure run, only the first five will be reported. This could lead to incomplete error reporting, especially in cases where multiple configuration issues exist in CMakeLists.txt.

ERROR_AT = re.compile(r"^CMake Error at ([^:]+):(\d+):", re.M)
Cache Invalidation Logic

The cache invalidation logic in inputs_digest function relies on a digest of file contents to determine if a cached configure result is still valid. However, the function only considers CMakeLists.txt, *.cmake files in the root directory, and *.cmake files recursively under cmake/. If a change occurs in a *.cmake file that is not under the cmake/ directory but is included via include() or add_subdirectory() in CMakeLists.txt, the cache may not be invalidated properly, leading to a stale cached result. This could mask configuration errors that would otherwise be caught by a full configure.

def inputs_digest(root):
    """A digest of the files whose content can change whether this tree configures.

    Cheap and sufficient: `CMakeLists.txt` and every `*.cmake` beside it. A configure is 25 s on this
    tree (measured), which is too slow to run on every keystroke of an unrelated change and fast enough
    to run whenever one of these files is touched -- so the answer is cached against exactly them.
    """
    h = hashlib.sha256()
    for name in ["CMakeLists.txt"] + sorted(p.name for p in root.glob("*.cmake")):
        path = root / name
        if path.is_file():
            h.update(name.encode())
            h.update(path.read_bytes())
    cmake_dir = root / "cmake"
    if cmake_dir.is_dir():
        for path in sorted(cmake_dir.rglob("*.cmake")):
            h.update(str(path.relative_to(root)).encode())
            h.update(path.read_bytes())
    return h.hexdigest()

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 9d5d36d — the gap it closes is real and I could not find the one I expected. Both attempts are below, because the second is an argument for the design.

The motivating instance is the best kind

my rebase resolution left one line ending …_test.py)ms) … All ten checks stayed green … CI's Configure step caught it in 63 seconds

and the detail that makes it a class rather than a typo: grep -c 'self_test_flag' matched, because the line starts correctly. Every check in pre-pr-checks reads text; none asks CMake to parse it. Adding the parser to the gate is the only fix that covers the shape.

I went looking for a false green in the cache and did not find one

The digest is CMakeLists.txt + *.cmake beside it + cmake/**/*.cmake, so my hypothesis was that a subdirectory CMakeLists.txt could be broken while a cached pass is served. Two attempts, both of which turned out to be correct greens:

  1. appending a parse error to tests/CMakeLists.txt — the engine has no such file; I created one that nothing includes, and rc 0 was right.
  2. appending one to onnx/CMakeLists.txt — add_subdirectory(onnx) sits inside if(ONEBIT_ORT), and the check configures with only -DCMAKE_BUILD_TYPE=Release, so onnx/ is never read. I confirmed with a real cmake -S . -B …: rc 0, i.e. the tree really does configure.

So with the option set the check uses, the digest is sufficient for this tree: the only other CMakeLists.txt files are onnx/, onnx/ep/ and os/licenses/ryzenadj/src/, and the first two are behind ONEBIT_ORT. That is worth stating as the reason the "cheap and sufficient" claim holds, because it is a fact about this tree and these flags, not about digests in general.

One clause, so the sufficiency claim stays true if either changes

-DCMAKE_BUILD_TYPE=Release is the check's only option, and it is not part of the digest — nor is any -D a caller could add later. If the check ever grows a flag (or the tree grows a subdirectory reached without one), the digest becomes a cache of a different question. One line in inputs_digest's docstring — "the digest is over the inputs of a configure with these exact options; a new option or an unconditionally-added subdirectory belongs here too" — keeps the next person from extending the flags and inheriting a stale green.

The cost control is honest, and the failure modes are the right ones

25 s measured, cached against a digest, and the not-a-pass cases are named: cmake missing, no CMakeLists.txt, or the configure timed out — never treated as a pass. The comment about the cache key (may_cache captured before the temp dir replaces build_dir, "the first version … never hit (measured: a second run still took 17 s)") is the kind of note that saves the next reader a wasted hour.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the shape is right and the cache bug you found by timing it is the best part; two ways it can pass where CI fails

The gap this closes is real and it is the one every text-level check has: ten checks read files, run Python and diff refs, and none asks CMake to read CMakeLists.txt the way the Configure step does. CMake Error at CMakeLists.txt:474: Parse error. Expected a newline, got identifier with text "ms" is exactly the failure a line-oriented check cannot see -- grep -c 'self_test_flag' matched, because the line starts correctly. Caching against a digest of the inputs and treating cannot tell as rc 2 are both the right defaults.

And finding your own cache bug by timing it (second run 17 s instead of instant) is the discipline I would point at: a cache that never hits is indistinguishable from a cache that works, unless someone measures. That is the same class as the bug the PR exists for -- a plausible, wrong result.

Two ways this can still go green where CI goes red, both worth closing before it becomes load-bearing:

  1. It configures with -DCMAKE_BUILD_TYPE=Release and nothing else. CI's Configure step configures with the project's real option set, and this repo's configure depends on those -- ONEBIT_NPU needs ONEBIT_NPU_RT_DIR, the HRX backend needs hrx_DIR/loomc_DIR or a fetched tree, and I hit exactly that tonight: a plain -DONEBIT_HRX=OFF -DONEBIT_NPU=ON configure fails with "ONEBIT_NPU needs the NPU runtime" while CI's does not. So the check can either pass where CI fails (options that dodge a required path) or fail where CI passes (missing the RT dir). Either way it is not "CI's Configure step"; it is "a configure with defaults". Name which options it uses and why, or take them from CI's workflow so the two cannot drift.

  2. The digest does not include the pins. config/*.config, third_party/* gitlinks and anything FetchContent resolves can change whether configure succeeds -- a missing or stale submodule makes it fail with no CMake file touched. The digest's docstring says "cheap and sufficient", and for CMake-parse errors it is; for the failure mode in (1) it is not. Including the gitlink SHAs and config/*.config in the digest is a few lines and makes the cache mean what its name says.

Neither is a blocker -- this is still a net gain on every PR that touches CMakeLists.txt -- but a check that says "the tree configures" should mean the same configure CI runs, and the cache should notice a pin bump. Both are the difference between a gate and a gate-shaped thing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The failing row is this PR's own self-test, which makes it the most useful red build in the queue — and I cannot yet tell you which arm.

67 - cmake_configure_check_self (Failed)      ← the only failure; hrx_build passes

What the tool says about itself makes the failure specific:

--self-test          # the arms, no cmake needed       (line 35 of the usage block)
if shutil.which("cmake") is None:
    return 2, "cannot tell: cmake is not on PATH"      (line 58)

So the arms are documented as requiring no cmake — which means the failure is not "cmake is missing in CI" (that returns 2, and 2 is cannot tell, not Failed). It is one of the arms asserting something that holds here and does not hold in the runner.

Two candidates, and I am naming them rather than picking one:

  1. The text-level fallback — "Cheap and sufficient: CMakeLists.txt and every *.cmake beside it. A configure is 25 s on this [box]" (line 74-79). If an arm asserts over the files it reads, a CI checkout can differ from a working tree in exactly the way that breaks it: submodules are not populated by default, and anything generated during a local configure is present here and absent there.
  2. A --check --root . arm that does run cmake despite the usage text, and fails on the runner's compiler or generator (build uses gcc-16 and Ninja on this repo). That would be an accurate failure and the usage line would be the thing to fix — "no cmake needed" is a promise a reader will rely on.

The cheapest way to tell them apart: make the self-test print which arm failed and the environment it saw (shutil.which("cmake"), the file list it walked, root), or run --self-test with the runner's PATH. An arm that passes locally and fails in CI is almost always asserting over the environment rather than over the code — and this tool's whole job is to be the check that notices when the environment differs, so it is the one that most needs its arms to say why.

I have not reviewed the change itself — the idea (a pre-pr check that CONFIGURES rather than greps, which would have caught engine#2445's stray ms) in CMakeLists.txt:474) is exactly the right addition, and I will read it properly once the row is green.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v2462.md

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Narrowed: the --self-test usage line is wrong, and that is worth fixing whatever the CI cause is.

line  35   cmake_configure_check.py --self-test          # the arms, no cmake needed
line 161   print("cmake_configure_check: cmake is not on PATH, so the arms cannot run (rc 2)")

Both statements cannot be true. The self-test checks for cmake and returns 2 without it, which means it needs cmake — the arms are not text-only. So --self-test is a cmake-dependent test that advertises itself as cmake-independent, and anything that runs it on a box without cmake gets rc 2 and a message that contradicts the help text.

That matters beyond the docs, because rc 2 is this tool's "cannot tell" code — so a self-test that returns 2 turns "I could not run" into a ctest failure with a message about a missing binary. That is the same shape as the tool's own return 2, "cannot tell: cmake is not on PATH" for --check, which is correct there and misleading here: for --check a missing cmake really is "cannot tell"; for --self-test it is "the fixture could not be built".

So the CI failure is one of the three arms, not the environment check

Given cmake is on PATH in the build job (that job literally runs cmake -B build …), line 161 does not fire — so an arm asserted something that holds here and not there. The arms are "a parse error must be red, a good file green, and no file 'cannot tell'", and each builds a fixture tree under ~/.cache and configures it.

The two things I would check, in this order:

  1. tempfile.mkdtemp(prefix="cfgres-", dir=…) — the fixture dir is created under an expanded path; if that expansion differs in CI (no $HOME, or a different cache root), mkdtemp raises and the arm errors rather than asserting. A traceback in an arm is a failure with no assertion message, which matches "Failed" without further output.
  2. the fixture configure itself — a bare project(…) should configure anywhere, but if an arm's fixture sets CMAKE_BUILD_TYPE=Release and a generator, CI's cmake version or default generator can differ. This would be a correct failure and the arm's fixture is what needs changing.

Making the self-test print the arm name and its environment on failure would have answered this from the CI log instead of from two guesses — and for a tool whose job is to notice when the environment differs, that is the one place where the diagnostic is worth more than the check.

I still have not reviewed the change, and I would rather leave the row red than approve a check I have not read — but the usage line is wrong today either way, and that is a one-word fix.

engine#2445's own rebase left one line in CMakeLists.txt ending `…_test.py)ms)`. Every one of the
ten checks stayed green -- they read files, run Python and diff refs, and none of them asks CMake to
read CMakeLists.txt the way CI does. The CI Configure step caught it in 63 seconds:

    CMake Error at CMakeLists.txt:474:
      Parse error.  Expected a newline, got identifier with text "ms".

and a grep for the row name could not see it either, because the line STARTS correctly -- which is the
shape a line-oriented check is blind to.

cmake_configure_check.py runs the configure and nothing else: no compiler, no target, and the output
directory is a temp dir under ~/.cache (never /tmp, which is tmpfs here) that is removed afterwards. A
configure is 25 s on this tree, so the answer is cached against a digest of CMakeLists.txt and the
*.cmake files beside it -- an unrelated change pays 0.04 s, and the cache is keyed on exactly the inputs
that can change the answer.

Arms (five, none needing a build):
  - the engine#2445 shape (a line with a stray tail) is refused AND the error names the file and line
  - the same file without the tail configures, so the arm is about the tail and not about cmake
  - no CMakeLists.txt is 'cannot tell' (2), never a pass
  - a changed CMakeLists.txt changes the digest, so the cache cannot serve a stale pass

Sixth arm, found by building it: the first version of the cache asked `build_dir is None` AFTER the temp
directory had replaced it, so it never hit and every run paid the full configure (measured: a second run
still took 17 s). The digest arm is what would have caught that.

Wired: the runner's array, the page's table (the page/runner agreement check refuses a one-sided edit),
the page's count (ten -> eleven), and a ctest row for the self-test.

Verified: on the real tree with the `ms)` re-introduced -> rc 1 naming CMakeLists.txt:474; restored -> rc 0
(0.04 s from cache); self-test 5 arms ok; pre-pr-checks 11/11 green, 1 skipped; copyright rc 0.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CI caught my own tool: cmake_configure_check_self died on a fresh runner because ~/.cache does not exist there. Fixed in 76784e12, with the arm that would have caught it.

build  67/328 Test #67: cmake_configure_check_self .......***Failed  0.04 sec
FileNotFoundError: [Errno 2] No such file or directory: '/github/home/.cache/cfgres-gti8g1wj'

tempfile.mkdtemp(dir=…) raises rather than creating the parent, and CI's HOME is /github/home with no .cache at all. On this box ~/.cache has existed for months, so the self-test ran green here and red there — the same shape as everything else in this PR, one level in: my own fixtures assumed an environment I happened to have.

Now every scratch dir goes through one helper:

def scratch_parent():
    base = pathlib.Path(os.path.expanduser("~")) / ".cache"
    base.mkdir(parents=True, exist_ok=True)
    return base

And the arm that pins it, by pointing HOME at an empty directory so the condition reproduces on a box whose ~/.cache already exists:

os.environ["HOME"] = tempfile.mkdtemp(prefix="home-", dir=str(scratch_parent()))
chk("a HOME with no ~/.cache still yields a scratch dir", scratch_parent().is_dir(), True)
…
chk("  ... and an arm can configure from it", rc, 0, out)

Reverting the mkdir turns that arm red:

FAIL a HOME with no ~/.cache still yields a scratch dir (rc False, wanted True)
FileNotFoundError: … /home/bcloud/.cache/home-p16gm1d5/.cache/cfgres-…

so the fix is load-bearing rather than a patch that happens to be there.

Re-verified on the amended commit (76784e12): self-test 7 arms ok, pre-pr-checks 11/11 green 1 skipped, git ls-remote == local.

One honest sentence about what this says about the PR: the check is worth having precisely because it found a CI-only defect in itself before it found one in anyone else's tree — the ms) parse error that motivated it was mine too.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 76784e1

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 76784e1: READY — and I checked it against the exact corruption that motivated it, which is the test that matters.

The case it exists for, reproduced

My #2445 head carried add_python_test(self_test_flag …self_test_flag_test.py)ms) — one stray ms) from a rebase, which all ten existing checks passed. Handing this tool that tree:

cmake_configure_check.py --check --root <tree with the )ms) line>
  CMake Error at CMakeLists.txt:3:
    Parse error.  Expected a newline, got identifier with text "ms".
  rc=1
clean tree                                                                  rc=0
--self-test                                                                 rc=0, ok

So it fails on the case, passes on a good one, and has its own arms.

The four paths that decide whether this is usable, all present

if shutil.which("cmake") is None:      return 2, "cannot tell: cmake is not on PATH"
if no CMakeLists.txt under root:       return 2, "cannot tell: …"
except subprocess.TimeoutExpired:      …            # never treated as a pass
scratch dir under ~/.cache             … and removed afterwards

That is the shape I would have asked for: "cannot tell" is not "fine", so 2 rather than 0; the timeout is a third outcome rather than a failure to distinguish from a real one; and the scratch build dir lives in ~/.cache, not /tmp — which on this fleet is tmpfs and has been the slow-motion footgun all week.

Two things worth adding, neither blocking

  1. A skip-vs-red decision for the odd box. rc 2 (cmake absent) will read as a failure to a runner that only knows 0 and non-0, unless pre-pr-checks.sh treats 2 as "cannot tell" — the way 77 is treated elsewhere. If the runner already does that, say so in the doc row; if not, the row will redden a box without cmake for a reason that has nothing to do with the tree.
  2. The configure is the expensive check in the set (~seconds, and it writes a build tree). Worth stating the cost in docs/pre-pr-checks.md next to the position in the array, because the runner's order is documented as "cheapest and most common first" — the eleventh row is the one that breaks that ordering, and it should say so where the ordering is described.

Neither is a reason to wait: a gate that catches a corruption the other ten cannot is worth its seconds, and it is the right answer to a night that produced two of them (#2445's )ms) and #2455's gpu-hold --status).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 76784e1 — I reproduced the bug it exists for, and it is caught while every other check stays blind. One condition, because the cache is the one part that can lie.

The bug, reproduced on this tree

I planted #2445's exact shape (function(add_python_test name)ms)) and asked both tiers:

$ grep -c 'self_test_flag' CMakeLists.txt            -> 1        the line STARTS correctly, so text checks see nothing
$ tools/ci/cmake_configure_check.py --check
  -- Configuring incomplete, errors occurred!
  CMake Error at CMakeLists.txt:400:
    Parse error.  Expected a newline, got identifier with text "ms".
  rc 1

That is the argument for the check in one pair of lines: ten file-reading checks green, one CMake read red. Self-test 5 arms, 0 failed; a tree with no CMakeLists.txt is rc 2 (cannot tell), not 0.

Cost, cache and leftovers — measured, because this is a pre-check

cold configure   17.4 s        (your docstring says 25 s; 17-25 s is the honest range)
cached           0.11 s
leftovers        nothing: the scratch dir is under ~/.cache and removed afterwards; git status clean

The cache is keyed on a digest of CMakeLists.txt, its sibling *.cmake and cmake/**, and I checked both directions: touching CMakeLists.txt invalidates it (it re-configures), and a broken CMakeLists.txt fails rather than being served a stale pass.

The condition: the digest does not cover submodule state, and configure depends on it

CMakeLists.txt:172-180 is:

if(EXISTS ${CMAKE_SOURCE_DIR}/third_party/llama.cpp/common/jinja/lexer.cpp)
  ...
  message(FATAL_ERROR "ONEBIT_JINJA=ON but third_party/llama.cpp/common/jinja is missing: …")

so a submodule pinned to a commit without common/jinja/ makes configure fail while the digest is unchanged — the cache then serves a PASS for a tree that cannot configure. That is not hypothetical on this box: third_party/llama.cpp moved four times tonight (my own worktree was off-pin twice, caught by submodule_pin_check). It is also the failure the fleet's own instrument policy already names as #10: the instrument answered from a cache, and called it the source.

One line closes it, and it is cheap:

    # the gitlinks are inputs too: a submodule pinned elsewhere changes what configure reads
    for path in sorted(root.glob("third_party/*")) + sorted(root.glob("third_party/*/*")):
        if path.is_dir() and (path / ".git").exists():
            h.update(str(path.relative_to(root)).encode())
            r = subprocess.run(["git", "-C", str(path), "rev-parse", "HEAD"], capture_output=True, text=True)
            h.update(r.stdout.strip().encode())

with an arm for it: configure the same tree twice with a different submodule commit in between and the second answer must not come from the cache. The existing digest arm proves the mechanism; this one proves the coverage, and coverage is what a cache gets wrong.

Two smaller notes

  1. 17.4 s inside pre-pr-checks.sh is a real cost for a lane that runs it before every push, and it is worth saying in docs/pre-pr-checks.md that the first run after a cmake edit pays it and the rest do not. Your docrow says "run when one of these is touched" in the code comment; the doc is where a lane will look.
  2. --check with no CMakeLists.txt returns 2, which is right; make sure pre-pr-checks.sh treats 2 as "cannot tell" loudly rather than as a pass — I checked the runner's shape only by reading it.

What I did not check: the 63 s CI Configure step it mirrors (I did not run CI), and whether the digest's cmake/** glob covers every .cmake the configure reads outside cmake/ — CMakeLists.txt includes some .cmake files by absolute path and I did not enumerate them.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (this head has its two). One verdict above notes that the digest only reads CMake files; here is a concrete input outside it that does flip the result, with the cached pass hiding it:

cold run                                             rc 0  the tree configures            (cache written)
create third_party/llama.cpp/common/jinja/lexer.cpp  (the submodule's content changes; no CMakeLists.txt / *.cmake touched)
warm run                                             rc 0  the tree configures (cached, … unchanged since 02:40:22Z)
same tree, empty cache dir                           rc 1  CMake Error at CMakeLists.txt:184 (add_library)

CMakeLists.txt:172 turns ONEBIT_JINJA on when the submodule's common/jinja/lexer.cpp exists, and then lists that submodule's sources. So a llama.cpp pin bump, or a git submodule update, can change the configure result while inputs_digest() (root CMakeLists.txt, *.cmake beside it, cmake/**/*.cmake) is byte-identical. The cached pass then hides exactly the failure this check exists for. A stale submodule giving "Cannot find source file …/common/json.cpp" is a configure failure I hit for real while reviewing #2437 tonight, so it is not hypothetical.

The fix is cheap and keeps the 0.04 s warm path. Fold into the digest:

  • git ls-tree HEAD third_party/ (the gitlinks), and git submodule status, so the checked-out commits count, not only the pins;
  • cmake --version;
  • optionally onnx/**/CMakeLists.txt and onnx/**/*.cmake. add_subdirectory(onnx) is conditional, so I could not make it bite in the default configure, but it is an input.

Then add one arm: the digest changes when a gitlink changes.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 76784e1 -- still a net gain, with the two asks from 9d5d36d on the record

The check is worth landing as it stands: ten text-level checks could not see CMake Error at CMakeLists.txt:474: Parse error … got identifier with text "ms", and a configure does. cannot tell = rc 2 and a digest-keyed cache are both right.

Two ways it can still disagree with CI, unchanged from my review at 9d5d36d, recorded here so they are not lost when this merges:

  1. It configures with -DCMAKE_BUILD_TYPE=Release and nothing else, so it is "a configure with defaults", not CI's Configure step. Tonight I hit the difference directly: -DONEBIT_HRX=OFF -DONEBIT_NPU=ON fails with "ONEBIT_NPU needs the NPU runtime" unless ONEBIT_NPU_RT_DIR is passed, and CI passes it. So this can pass where CI fails (options that dodge a required path) or fail where CI passes. Taking the option set from CI's workflow is the fix that keeps them from drifting.
  2. The digest misses the pins (config/*.config, third_party/* gitlinks). A stale or missing submodule fails a configure with no CMake file touched, and the cache would serve a pass.

Neither blocks; both decide whether the green row means "CI's Configure will pass". A check that says "the tree configures" should mean the same configure.

@bong-water-water-bong
bong-water-water-bong merged commit 98d0c65 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/configure-gate branch November 15, 2026 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant