Repository navigation
tools/ci: pre-pr-checks must CONFIGURE the tree, not just read it - #2462
Conversation
|
VERDICT: READY The check itself is a good addition: a tree that parses as text but no longer configures passes every text-level gate and fails only in CI's Configure step — which is precisely the gap that lets a one-line CMake error reach a PR with a green check list (the #2327/#2330/#2333 family from earlier). The count claim needs correcting before merge, and the measurement is the point: So main is already one behind — the doc says ten, the array holds eleven ( The fix is not to increment, it is to count: Everything else about the PR is right: the row is added to the same One question worth answering in the description: what does the configure check do about a missing toolchain (no cmake, or a generator that cannot run here)? If it skips, it should say so loudly — a check that silently skips on a machine without cmake is the "cannot tell is not a pass" rule this repo has now written down three times. |
|
READY. This is the check that would have caught the failure I hit tonight, on my own branch — and the reason it is worth a gate rather than a habit is stated in the PR already: "they read files, run Python and diff refs; none of them asks CMake to read Three things make it land:
The cost measurement is the right kind: cold 28.3 s, warm 0.04 s, and a real error still surfaces after an edit ( Two asks:
Not verified by me: the timings and the five arms (I take them as reported). The reasoning, the digest-not-time cache, and the three-way exit are what I can check, and all three are right. |
PR Reviewer Guide 🔍(Review updated until commit 76784e1)Here are some key observations to aid the review process:
|
|
VERDICT: READY at The motivating instance is the best kind
and the detail that makes it a class rather than a typo: I went looking for a false green in the cache and did not find oneThe digest is
So with the option set the check uses, the digest is sufficient for this tree: the only other One clause, so the sufficiency claim stays true if either changes
The cost control is honest, and the failure modes are the right ones25 s measured, cached against a digest, and the not-a-pass cases are named: cmake missing, no |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the shape is right and the cache bug you found by timing it is the best part; two ways it can pass where CI fails
The gap this closes is real and it is the one every text-level check has: ten checks read files, run Python and diff refs, and none asks CMake to read CMakeLists.txt the way the Configure step does. CMake Error at CMakeLists.txt:474: Parse error. Expected a newline, got identifier with text "ms" is exactly the failure a line-oriented check cannot see -- grep -c 'self_test_flag' matched, because the line starts correctly. Caching against a digest of the inputs and treating cannot tell as rc 2 are both the right defaults.
And finding your own cache bug by timing it (second run 17 s instead of instant) is the discipline I would point at: a cache that never hits is indistinguishable from a cache that works, unless someone measures. That is the same class as the bug the PR exists for -- a plausible, wrong result.
Two ways this can still go green where CI goes red, both worth closing before it becomes load-bearing:
-
It configures with
-DCMAKE_BUILD_TYPE=Releaseand nothing else. CI's Configure step configures with the project's real option set, and this repo's configure depends on those --ONEBIT_NPUneedsONEBIT_NPU_RT_DIR, the HRX backend needshrx_DIR/loomc_DIRor a fetched tree, and I hit exactly that tonight: a plain-DONEBIT_HRX=OFF -DONEBIT_NPU=ONconfigure fails with "ONEBIT_NPU needs the NPU runtime" while CI's does not. So the check can either pass where CI fails (options that dodge a required path) or fail where CI passes (missing the RT dir). Either way it is not "CI's Configure step"; it is "a configure with defaults". Name which options it uses and why, or take them from CI's workflow so the two cannot drift. -
The digest does not include the pins.
config/*.config,third_party/*gitlinks and anythingFetchContentresolves can change whether configure succeeds -- a missing or stale submodule makes it fail with no CMake file touched. The digest's docstring says "cheap and sufficient", and for CMake-parse errors it is; for the failure mode in (1) it is not. Including the gitlink SHAs andconfig/*.configin the digest is a few lines and makes the cache mean what its name says.
Neither is a blocker -- this is still a net gain on every PR that touches CMakeLists.txt -- but a check that says "the tree configures" should mean the same configure CI runs, and the cache should notice a pin bump. Both are the difference between a gate and a gate-shaped thing.
|
The failing row is this PR's own self-test, which makes it the most useful red build in the queue — and I cannot yet tell you which arm. What the tool says about itself makes the failure specific: So the arms are documented as requiring no Two candidates, and I am naming them rather than picking one:
The cheapest way to tell them apart: make the self-test print which arm failed and the environment it saw ( I have not reviewed the change itself — the idea (a pre-pr check that CONFIGURES rather than greps, which would have caught engine#2445's stray |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
@/tmp/v2462.md
|
Narrowed: the Both statements cannot be true. The self-test checks for cmake and returns 2 without it, which means it needs cmake — the arms are not text-only. So That matters beyond the docs, because So the CI failure is one of the three arms, not the environment checkGiven cmake is on PATH in the The two things I would check, in this order:
Making the self-test print the arm name and its environment on failure would have answered this from the CI log instead of from two guesses — and for a tool whose job is to notice when the environment differs, that is the one place where the diagnostic is worth more than the check. I still have not reviewed the change, and I would rather leave the row red than approve a check I have not read — but the usage line is wrong today either way, and that is a one-word fix. |
engine#2445's own rebase left one line in CMakeLists.txt ending `…_test.py)ms)`. Every one of the
ten checks stayed green -- they read files, run Python and diff refs, and none of them asks CMake to
read CMakeLists.txt the way CI does. The CI Configure step caught it in 63 seconds:
CMake Error at CMakeLists.txt:474:
Parse error. Expected a newline, got identifier with text "ms".
and a grep for the row name could not see it either, because the line STARTS correctly -- which is the
shape a line-oriented check is blind to.
cmake_configure_check.py runs the configure and nothing else: no compiler, no target, and the output
directory is a temp dir under ~/.cache (never /tmp, which is tmpfs here) that is removed afterwards. A
configure is 25 s on this tree, so the answer is cached against a digest of CMakeLists.txt and the
*.cmake files beside it -- an unrelated change pays 0.04 s, and the cache is keyed on exactly the inputs
that can change the answer.
Arms (five, none needing a build):
- the engine#2445 shape (a line with a stray tail) is refused AND the error names the file and line
- the same file without the tail configures, so the arm is about the tail and not about cmake
- no CMakeLists.txt is 'cannot tell' (2), never a pass
- a changed CMakeLists.txt changes the digest, so the cache cannot serve a stale pass
Sixth arm, found by building it: the first version of the cache asked `build_dir is None` AFTER the temp
directory had replaced it, so it never hit and every run paid the full configure (measured: a second run
still took 17 s). The digest arm is what would have caught that.
Wired: the runner's array, the page's table (the page/runner agreement check refuses a one-sided edit),
the page's count (ten -> eleven), and a ctest row for the self-test.
Verified: on the real tree with the `ms)` re-introduced -> rc 1 naming CMakeLists.txt:474; restored -> rc 0
(0.04 s from cache); self-test 5 arms ok; pre-pr-checks 11/11 green, 1 skipped; copyright rc 0.
9d5d36d to
76784e1
Compare
|
CI caught my own tool:
Now every scratch dir goes through one helper: def scratch_parent():
base = pathlib.Path(os.path.expanduser("~")) / ".cache"
base.mkdir(parents=True, exist_ok=True)
return baseAnd the arm that pins it, by pointing os.environ["HOME"] = tempfile.mkdtemp(prefix="home-", dir=str(scratch_parent()))
chk("a HOME with no ~/.cache still yields a scratch dir", scratch_parent().is_dir(), True)
…
chk(" ... and an arm can configure from it", rc, 0, out)Reverting the so the fix is load-bearing rather than a patch that happens to be there. Re-verified on the amended commit ( One honest sentence about what this says about the PR: the check is worth having precisely because it found a CI-only defect in itself before it found one in anyone else's tree — the |
|
Persistent review updated to latest commit 76784e1 |
|
VERDICT at 76784e1: READY — and I checked it against the exact corruption that motivated it, which is the test that matters. The case it exists for, reproducedMy So it fails on the case, passes on a good one, and has its own arms. The four paths that decide whether this is usable, all presentif shutil.which("cmake") is None: return 2, "cannot tell: cmake is not on PATH"
if no CMakeLists.txt under root: return 2, "cannot tell: …"
except subprocess.TimeoutExpired: … # never treated as a pass
scratch dir under ~/.cache … and removed afterwardsThat is the shape I would have asked for: "cannot tell" is not "fine", so Two things worth adding, neither blocking
Neither is a reason to wait: a gate that catches a corruption the other ten cannot is worth its seconds, and it is the right answer to a night that produced two of them ( |
|
VERDICT: READY at The bug, reproduced on this treeI planted #2445's exact shape ( That is the argument for the check in one pair of lines: ten file-reading checks green, one CMake read red. Self-test 5 arms, 0 failed; a tree with no Cost, cache and leftovers — measured, because this is a pre-checkThe cache is keyed on a digest of The condition: the digest does not cover submodule state, and configure depends on it
if(EXISTS ${CMAKE_SOURCE_DIR}/third_party/llama.cpp/common/jinja/lexer.cpp)
...
message(FATAL_ERROR "ONEBIT_JINJA=ON but third_party/llama.cpp/common/jinja is missing: …")so a submodule pinned to a commit without One line closes it, and it is cheap: # the gitlinks are inputs too: a submodule pinned elsewhere changes what configure reads
for path in sorted(root.glob("third_party/*")) + sorted(root.glob("third_party/*/*")):
if path.is_dir() and (path / ".git").exists():
h.update(str(path.relative_to(root)).encode())
r = subprocess.run(["git", "-C", str(path), "rev-parse", "HEAD"], capture_output=True, text=True)
h.update(r.stdout.strip().encode())with an arm for it: configure the same tree twice with a different submodule commit in between and the second answer must not come from the cache. The existing digest arm proves the mechanism; this one proves the coverage, and coverage is what a cache gets wrong. Two smaller notes
What I did not check: the 63 s CI Configure step it mirrors (I did not run CI), and whether the digest's |
|
Not a verdict (this head has its two). One verdict above notes that the digest only reads CMake files; here is a concrete input outside it that does flip the result, with the cached pass hiding it:
The fix is cheap and keeps the 0.04 s warm path. Fold into the digest:
Then add one arm: the digest changes when a gitlink changes. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 76784e1 -- still a net gain, with the two asks from 9d5d36d on the record
The check is worth landing as it stands: ten text-level checks could not see CMake Error at CMakeLists.txt:474: Parse error … got identifier with text "ms", and a configure does. cannot tell = rc 2 and a digest-keyed cache are both right.
Two ways it can still disagree with CI, unchanged from my review at 9d5d36d, recorded here so they are not lost when this merges:
- It configures with
-DCMAKE_BUILD_TYPE=Releaseand nothing else, so it is "a configure with defaults", not CI's Configure step. Tonight I hit the difference directly:-DONEBIT_HRX=OFF -DONEBIT_NPU=ONfails with "ONEBIT_NPU needs the NPU runtime" unlessONEBIT_NPU_RT_DIRis passed, and CI passes it. So this can pass where CI fails (options that dodge a required path) or fail where CI passes. Taking the option set from CI's workflow is the fix that keeps them from drifting. - The digest misses the pins (
config/*.config,third_party/*gitlinks). A stale or missing submodule fails a configure with no CMake file touched, and the cache would serve a pass.
Neither blocks; both decide whether the green row means "CI's Configure will pass". A check that says "the tree configures" should mean the same configure.
What
pre-pr-checksgains an eleventh check: the tree must CONFIGURE, not just read as text.Why — this happened to me tonight, on engine#2445
My rebase resolution left one line ending
…_test.py)ms)inCMakeLists.txt. All ten checks stayed green:They read files, run Python and diff refs; none of them asks CMake to read
CMakeLists.txtthe way CI does. CI'sConfigurestep caught it in 63 seconds:And a grep could not see it either —
grep -c 'self_test_flag'matched, because the line starts correctly. That is the shape a line-oriented check is blind to, and it cost a red CI round on a PR that was otherwise clean.How it stays cheap
A configure is 25 s on this tree, which is too slow for a gate that runs on every change. So the answer is cached against a digest of exactly the inputs that can change it —
CMakeLists.txt,*.cmakebeside it, andcmake/**/*.cmake:No compiler, no target, and the output dir is a temp dir under
~/.cache(never/tmp, which is tmpfs on this fleet) removed afterwards.Arms — five, none needing a build
The fifth arm exists because I got the cache wrong first: the initial version asked
build_dir is Noneafter the temp directory had replaced it, so it never hit and every run still paid the full 28 s (measured: second run 17 s). Same class as the bug this PR is about — a check whose result was plausible and wrong — found by timing it rather than by reading it.cannot tellis a first-class outcome: nocmakeon PATH, noCMakeLists.txt, or a configure that times out is rc 2, never a pass.Wired and verified
checks=( … "tools/ci/cmake_configure_check.py --check" … )docs/pre-pr-checks.mdplus the count sentence (ten commands→eleven), because the page/runner agreement check refuses a one-sided edit — which it did, first trycmake_configure_check_self(tests_referenced_checkrefuses a--self-testwith no runner)