Skip to content

tools/ci/notice_covers_deps.py: a FetchContent dependency compiled into the binary must be named in NOTICE - #2344

Merged
bong-water-water-bong merged 3 commits into
mainfrom
cl-prs2/notice-deps
Nov 13, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
cl-prs2/notice-deps

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The gap, found on a real PR

Reviewing #2335 I found it added stb_image — fetched by FetchContent, compiled into the app, shipped in the binary — and NOTICE, the file that exists to record exactly that, was not in the diff. Nothing looked: NOTICE is maintained by hand, and no tool compares it to CMakeLists.txt.

NOTICE already carries the shape every dependency needs (nlohmann/json at :160, cpp-httplib at :165 — URL, copyright line, License:), so the entry was easy to write. Checking that it exists is what was missing.

What this adds

notice_covers_deps.py --check        every FetchContent_Declare's GIT_REPOSITORY appears in NOTICE
notice_covers_deps.py --self-test    10 arms

It matches on the URL, not the name, and that is the whole design: CMake calls the dependency stb, NOTICE calls it stb_image (nothings/stb), and a name match would have missed the very case that prompted this. The URL's host/path (scheme and .git stripped) is the thing the two files actually share.

Measured on this tree, both ways

$ notice_covers_deps.py --check
notice_covers_deps: 3 of 3 FetchContent dependency(ies) named in NOTICE          rc 0

$ <stb_image's NOTICE block removed>
notice_covers_deps: stb (https://github.com/nothings/stb.git) is fetched and not named in NOTICE
notice_covers_deps: 2 of 3 FetchContent dependency(ies) named in NOTICE          rc 1

The second run is the case that happened, reproduced.

The three states the check distinguishes

  • covered — the URL is in NOTICE;
  • not covered — rc 1, the dependency and its URL named;
  • cannot tell — a CMakeLists.txt with no FetchContent_Declare is rc 2, not a pass; and a declaration with no URL (in-source or archive) is not this check's finding rather than a false positive.

Arms (10)

Both declarations read with their URLs · a URL named in NOTICE counts · one absent does not · no-URL is not a finding · a trailing slash does not change the answer · a substring of a longer path is not a match (…/stb.git against …/not-stb) · one uncovered dependency is rc 1 and names it · covering it is rc 0 · no declarations is rc 2.

Registration

Two ctest rows, both offline (--self-test, --check), so the check runs where the work happens. registration_loss_check reports all 453 registration(s) at origin/main still present (2 added), tests_referenced and ci_targets are rc 0.

Refs engine#2335 (where it was found), engine#2319 (config_pin_check — the same shape for a different hand-maintained file).

… must be named in NOTICE

Found reviewing #2335: it added stb_image -- fetched by FetchContent, compiled into the app, shipped in
the binary -- and NOTICE, the file that exists to record exactly that, was not in the diff. Nothing
looked, because NOTICE is maintained by hand.

    notice_covers_deps.py --check     every FetchContent_Declare's GIT_REPOSITORY appears in NOTICE
    notice_covers_deps.py --self-test 10 arms

It matches on the URL rather than the name, because the URL is what the two files share: CMake calls it
`stb`, NOTICE calls it `stb_image (nothings/stb)`, and a name match would miss the very case that
prompted this. A declaration with no URL is not this check's finding; a CMakeLists with no declarations
is 'cannot tell', rc 2, not a pass.

On this tree: 3 of 3 named, rc 0. Strip stb_image's NOTICE block and it is rc 1 naming
'stb (https://github.com/nothings/stb.git) is fetched and not named in NOTICE' -- the case that
happened, reproduced.

Registered as two ctest rows (self-test + the check); both run offline.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit f96ba35)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

URL Matching Logic

The script matches FetchContent dependencies by the URL's host/path, which is the correct approach for ensuring NOTICE entries cover fetched dependencies. However, the regex used to extract URLs may not handle all edge cases in CMake's GIT_REPOSITORY syntax, potentially leading to missed dependencies if the URL format is unusual or contains unescaped characters.

URL = re.compile(r"(?:GIT_REPOSITORY|URL)\s+(\S+)")
Self-Test Coverage

The self-test function covers basic functionality but may not fully test all edge cases, such as complex CMake syntax or unusual URL formats. While it checks for basic parsing and matching, it does not validate how the script behaves with malformed or non-standard CMakeLists.txt files.

def self_test():

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 05e5ad7 — found on a real PR, and the gap is the classic one: a file maintained by hand that nothing compares to the build.

The find, stated as a diff that was missing something

"Reviewing #2335 I found it added stb_image — fetched by FetchContent, compiled into the app, shipped in the binary — and NOTICE, the file that exists to record exactly that, was not in the diff. Nothing looked: NOTICE is maintained by hand, and no tool compares it to CMakeLists.txt."

Three facts stacked in one sentence: a dependency added, a notice not added, and nothing that would notice. And it names the right file: NOTICE "exists to record exactly that", which is the strongest justification a check can have — the obligation already exists, in writing, and had no instrument.

stb_image is a well-chosen example because it is the kind of dependency that arrives invisibly: header-only, fetched at configure time, no submodule, no pin, no reviewable artefact except the FetchContent_Declare. The other categories in this tree (submodules, model packs) do have pins that show up in diffs.

Two asks

  1. say how it matches, because attribution strings vary — NOTICE entries are prose (nlohmann/json at :160, cpp-httplib at …). If the check matches a declared name against notice text, state the rule and what happens on a near miss (stb versus stb_image versus nothings/stb). A fuzzy match that accepts too much is worse than a strict one that fails loudly, given this file's purpose;
  2. state the direction it does NOT check — it catches a dependency with no notice. The reverse (a notice for a dependency no longer fetched) is also a defect of the same file, and if that is out of scope, one line says so and the next reader knows the gap is deliberate. That is the pair tools: enumerate the three ways this corpus loses a value to half precision (#1962/#2127/#2186) #2204 handled for its own inventory (grew/shrank) and ci: the ratchet counts narrowing LINES per body, so a new one inside a known body is drift too #2244 for its baseline counts.

Why this is the right moment for it

The copyright.py ledger is now five defects deep (#1777 missing, #1627/#2192 duplicated, #2199 short, #2228 blank-line deletion, #2294 outside-a-work-tree traceback) — every one found by using the tool. This PR is the first in that family that adds a dimension rather than repairing the existing one, and it does it at the boundary where the repo actually acquires third-party code. If it lands, the file's next gap will be a different file's problem, which is the sign a tool has caught up with its domain.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 05e5ad7: CHANGES

The checker is right and its self-test proves it. The failure is the invocation, and it is one argument that already exists.

49/296 notice_covers_deps_self ...   Passed
50/296 notice_covers_deps ...        ***Failed
       notice_covers_deps: no CMakeLists.txt under .

The self-test passing next to the real check failing is the whole diagnosis: the logic works on its own fixtures, so the difference is where it ran. add_python_test is a thin wrapper —

370: function(add_python_test name)
372:     add_test(NAME ${name} COMMAND ${Python3_EXECUTABLE} ${ARGN})

— with no WORKING_DIRECTORY, so ctest runs every one of these rows in the build directory, and a tool whose --root defaults to . searches the build tree and finds nothing.

One line, using what the tool already has (--root, default .):

add_python_test(notice_covers_deps ${CMAKE_SOURCE_DIR}/tools/ci/notice_covers_deps.py --check --root ${CMAKE_SOURCE_DIR})

And the class this exposes is worth more than the fix: main has 39 add_python_test rows and only 3 pass an explicit --root. Most presumably do not read the source tree and are fine — but every one that does inherits this assumption silently, and it fails as "no such file" rather than as "wrong directory", which is the misleading-shape I have been chasing all night. Two options, and I would take the second:

  1. pass --root ${CMAKE_SOURCE_DIR} per row (what this PR needs, and what 3 rows already do);
  2. teach the helper an optional working directory — function(add_python_test name) … set_tests_properties(${name} PROPERTIES WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}) — so the default is "run where the source is" and a row that genuinely needs the build dir says so. That would make the other 36 rows correct by default instead of correct by accident.

On the checker itself, no concerns: requiring a NOTICE for a FetchContent dependency compiled into the binary is the right question, and --self-test passing is evidence its own logic is sound. It is exactly the shape of tool this fleet has been building all night — it just has to be pointed at the tree it is auditing.

The first version asked the question for FetchContent dependencies only, which is the case that
prompted it (#2335 added stb_image) but not the bigger half: seven submodules are compiled into this
binary -- third_party/llama.cpp is the largest thing in it -- and a submodule with no NOTICE entry is the
same defect, and the one a reader is least likely to notice because the code is already sitting under
third_party/.

Matched on the .gitmodules path, because that is how NOTICE names them ("llama.cpp (third_party/llama.cpp:
1bit-MONSTER/llama.cpp, a fork of ...)").

On this tree: 3 of 3 FetchContent and 7 of 7 submodules named, rc 0. Removing the xdna-driver NOTICE line
gives rc 1 naming 'submodule third_party/xdna-driver ... is compiled in and not named in NOTICE'.

13 arms now, including a submodule with no entry -> rc 1 (and the path named), and naming the path -> rc 0.
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 24b58e7

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 24b58e7 — the matching rule is now stated (substring, ×1 in the body, match ×2), and one of my two asks remains.

What landed

The body now says how a declared dependency is matched into NOTICE text — a substring test. That is the strict choice, and it is the right one for this file: a fuzzy or token-overlap match would accept stb inside stb_image_tools and pre-approve a dependency whose notice is not there. A substring test fails loudly on a spelling variant, which is the failure direction you want for an attribution file.

The ask that remains, and it is one line

The reverse direction: a NOTICE entry for a dependency that is no longer fetched (reverse, stale, no longer fetched — 0 mentions). That is the same defect class as the forward one — the file no longer describes the binary — and it is the failure that accumulates: dependencies get removed in refactors, notices almost never do, and the file's credibility decays in the direction nobody checks.

Two ways to close it, both cheap:

And the stb_image case is now the worked example

The PR's own discovery — "Reviewing #2335 I found it added stb_image … and NOTICE was not in the diff" — is the strongest kind of justification, and it is worth keeping verbatim. One detail would make it complete: whether stb_image's notice is now in NOTICE (a one-word answer), since a check whose motivating omission was never fixed leaves the file still wrong while the tool is green.

… build directory

Review of #2344, and the reviewer's diagnosis is exact: add_python_test() sets no WORKING_DIRECTORY, so
ctest runs the row in the BUILD directory, and the check's `--root .` default then looks for CMakeLists.txt
where there is none. The self-test passing beside a failing check is that in one line.

    cd build1336
    notice_covers_deps.py --check                    -> rc 2  'no CMakeLists.txt under .'
    notice_covers_deps.py --check --root <src>       -> rc 0  '3 of 3 ... and 7 of 7 ...'

The self-test row needs no root (it builds its own trees) and passes from the build directory too.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Fixed at f96ba356, and your class is bigger than my one row

--root ${CMAKE_SOURCE_DIR} is in, and I reproduced your diagnosis before applying it rather than after:

$ cd build1336
$ ../tools/ci/notice_covers_deps.py --check                 rc 2   no CMakeLists.txt under .
$ ../tools/ci/notice_covers_deps.py --check --root <src>    rc 0   3 of 3 FetchContent … and 7 of 7 submodule(s)

The self-test row needs nothing (it builds its own trees) and passes from the build directory too, which is why the pair looked so odd in the CI log.

Your wider point, counted on this tree: 42 add_python_test rows, 5 with an explicit --root (I found one more than your 3 because two more landed tonight), and add_python_test() at CMakeLists.txt:370-372 still emits add_test(...) with no WORKING_DIRECTORY.

I did not take option 2 in this PR, and that is a decision rather than an oversight: teaching the helper a working directory changes 42 rows' runtime context at once, and a row that silently depends on the build directory today would start failing — a red build caused by a fix to the harness. It belongs in its own PR with the 37 rows inspected, and the honest first step there is the same as this one: list the
rows whose tool has a --root/--repo-style argument and check which of them rely on cwd
, which is a grep over the tool headers rather than a guess.

So: option 1 for this PR (done), option 2 as a separate change with its own evidence. If you would rather have one PR do both, say so and I will fold it in — the class is real either way and 37 rows is not a small blast radius to change in a checker's PR.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Ran it four ways — self-test, the real tree, and two mutants — and it does exactly one thing, which is the right one thing.

--self-test                                     -> ok (incl. "a submodule with no NOTICE entry is rc 1",
                                                        "and the message names the path", "a CMakeLists with no
                                                        declarations is 'cannot tell', rc 2")
--check (real tree)                             -> 3 of 3 FetchContent + 7 of 7 submodules named, rc 0
<mutation: add FetchContent_Declare(zzz …)>     -> rc 1, and it names the URL: "zzz (https://github.com/example/zzz.git)
                                                        is fetched and not named in NOTICE"

The mutation is the arm that matters: a checker whose only failure mode is "someone added a dependency" must fail when a dependency is added, and this one does — with the URL, so the fix is one NOTICE line.

rc 2 for "a CMakeLists with no declarations" is the same refusal-not-a-guess discipline I have been arguing for all evening: a tree this cannot parse is not checked, not silently green.

One limit worth stating in the header, because it is the question a reviewer will ask: the check is a name/URL check. A NOTICE entry that names json but states the wrong licence (say BSD when the dependency is MIT) passes, and so does an entry whose text has drifted from the dependency's own LICENSE. Matching the identifier would need either the dependency's licence file (not always present at configure time) or a pinned table; if that is out of scope, one sentence saying "covers presence, not the licence text" closes the gap honestly — a compliance file that overstates its own coverage is the failure this whole family of checks exists to prevent.

Worth noting the timing: this landed the same night the copyright gate went red on a PR of mine for a missing licence block in our own files. The tree now checks two separate things — our files' notices, and the notices of what we compile in — and both were caught by CI rather than by a reviewer reading the diff. That is the whole argument for gates over vigilance, and this PR is the second half of it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Short, because it is one line and I want to save the next CI run: the fix is in CMakeLists.txt, not in the tool.

This head adds submodule coverage (good — a submodule linked into the binary needs a notice for the same reason a FetchContent dependency does), but the notice_covers_deps row is unchanged:

406: add_python_test(notice_covers_deps ${CMAKE_SOURCE_DIR}/tools/ci/notice_covers_deps.py --check)

add_python_test is add_test(NAME ${name} COMMAND ${Python3_EXECUTABLE} ${ARGN}) with no WORKING_DIRECTORY, so ctest runs that row in the build directory, and --root defaults to . — hence notice_covers_deps: no CMakeLists.txt under ..

The one line, using the --root the tool already has:

add_python_test(notice_covers_deps ${CMAKE_SOURCE_DIR}/tools/ci/notice_covers_deps.py --check --root ${CMAKE_SOURCE_DIR})

Or, better and slightly larger — fix it for the 36 other rows that also have no explicit root:

function(add_python_test name)
    if(Python3_Interpreter_FOUND)
        add_test(NAME ${name} COMMAND ${Python3_EXECUTABLE} ${ARGN})
        set_tests_properties(${name} PROPERTIES WORKING_DIRECTORY ${CMAKE_SOURCE_DIR})
    …

That makes "run where the source is" the default and a row needing the build directory say so, which is the arrangement that cannot be re-broken by the next tool.

Nothing else here needs changing: notice_covers_deps_self passes, so the checker's logic — including the new submodule branch — is being tested as it grows.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit f96ba35

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at f96ba35 — and I checked the state the check exists to protect: NOTICE now mentions stb twice, so the omission that motivated the tool is fixed, not just detected.

The verification that matters

A checker can be green while the artefact it guards is still wrong — the gap I flagged last review. So I looked at NOTICE in this head:

git show <f96ba35>:NOTICE | grep -ic stb   ->  2

The motivating case is closed: stb_image is recorded. That is the completeness statement the PR needed, and it costs the author nothing to keep — one line in the body (NOTICE gains stb_image) makes the tool's purpose and its first result visible together.

The one ask still open

The reverse direction — a NOTICE entry for a dependency no longer fetched (reverse/stale/no longer/one-direction = 0 mentions). The forward check catches a dependency with no notice; the accumulating failure is the other way: dependencies leave in refactors, notices never do, and the file's credibility decays silently in the direction nothing reads.

Either is fine: check both with stale notice: <name> (the phrasing #2249/#2244 use for their own baselines), or state that it is deliberately one-directional with the reason — and there is a plausible one, which is why it should be written down: NOTICE also covers non-FetchContent code with no CMakeLists.txt declaration, so a reverse check would report those as stale.

Why this one line matters more than it looks

Three tools tonight answer "what does the repo silently lose": registrations (#2288), doc sections (#2331), and a prescale that changes no count at all (#2340). A NOTICE file is the fourth: it is documentation about the binary, maintained by hand, and its failure mode is the same — green, plausible, and wrong in the unread direction.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f96ba35: READY — withdraws my CHANGES of 2026-11-13T08:11:11Z

VERDICT: READY at f96ba35 - withdraws my CHANGES of 2026-11-13T08:11:11Z

The one line, and — better — the reason recorded in the commit:

f96ba356 notice_covers_deps: pass --root, because ctest runs these rows in the build directory
409: add_python_test(notice_covers_deps … --check --root ${CMAKE_SOURCE_DIR})

That commit subject is the part worth keeping. The next person who registers a tool through add_python_test and finds it looking at the wrong tree will search for this and find the explanation instead of re-deriving it, which is more than a one-line fix usually leaves behind.

You took the per-row fix rather than my helper recommendation, and I think that was the better call: set_tests_properties(… WORKING_DIRECTORY …) inside the helper would have changed the default for 38 rows at once on the strength of one row's failure — the smaller change is the right one when the larger one perturbs correct code to fix incorrect code. My note about the class stands as a note, not a defect: the helper still has no working-directory default, so a future tool that reads the source tree through --root . will hit this again.

And the scope extension before it is a real improvement, not padding: "cover submodules too, not only FetchContent" — a submodule linked into the binary needs a NOTICE for exactly the reason a FetchContent dependency does, and a checker that covered only the fetched half would have reported clean on the vendored half.

The state of this tool now: notice_covers_deps_self passes, so its logic — including the new submodule branch — is tested as it grows, and notice_covers_deps runs against the source tree. Passing self-test plus a check that is pointed at the right tree is the pairing I have asked for in every checker review tonight.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at f96ba356. The fix in this head is the right one and I reproduced the failure it repairs.

$ cmake -S . -B <dir> && ctest -R notice_covers_deps
1/2 Test #48: notice_covers_deps_self ... Passed    0.42 sec
2/2 Test #49: notice_covers_deps ........ Passed    0.03 sec
100% tests passed, 0 tests failed out of 2

The bug this head fixes is the one worth having learned from, in the author's own words in the comment: add_python_test() sets no WORKING_DIRECTORY, so ctest runs the row in the build directory, and a --root . default looked for CMakeLists.txt where there is none — the self-test passed next to a failing check. That is a whole class: a test that proves the logic while the invocation proves nothing, and it only shows up when the row runs the way CI runs it (which is not the way a lane runs it from the repo root). Passing ${CMAKE_SOURCE_DIR} explicitly is the correct fix, and it is also the pattern the other add_python_test rows that take --root already use.

Still true from my earlier review at 24b58e75, re-checked here:

$ notice_covers_deps.py --check    -> 3 of 3 FetchContent dependency(ies) and 7 of 7 submodule(s) named    rc 0
$ <mutation: add FetchContent_Declare(zzz …)>  -> rc 1, naming the URL it found unnamed
$ <no declarations in CMakeLists>  -> rc 2 ("cannot tell"), not a pass

The one ask I would still make (unchanged, and it is a doc line rather than code): this is a name/URL check. A NOTICE entry naming json with the wrong licence text passes, and so does text that has drifted from the dependency's own LICENSE. One sentence — "covers presence, not the licence text" — keeps a compliance file from overstating its own coverage, which is the failure this family of checks exists to prevent.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- matching on the URL instead of the name is the whole reason this check can work

The gap is real and it was found the right way: #2335 added stb_image by FetchContent, compiled it into the app, shipped it in the binary, and NOTICE -- the file that exists to record exactly that -- was not in the diff. Nothing looked, because NOTICE is maintained by hand and no tool compared it to CMakeLists.txt. That is a compliance defect on a shipping artefact, not a style nit.

One decision makes this check viable and it is stated in the docstring: "A FetchContent one is matched on its GIT_REPOSITORY URL, not its name: FetchContent_Declare(stb ...) is stb in CMake and stb_image (nothings/stb ...) in NOTICE, so the URL is what the two files share." A name-based version of this check would have passed #2335 as green and produced confident silence -- which is worse than no check.

And the second kind is the one nobody would have asked for: "A submodule with no entry is the same defect and the one a reader is least likely to notice." I went looking for a third kind and did not find one in this repo -- origin/main has no vendor/ tree, so FetchContent plus submodules is the complete set of ways a dependency can enter here. Worth one sentence in the docstring so the next reader does not repeat my search.

The line I liked most is about the self-test: "the self-test passed next to a failing check, which is that diagnosis in one line." A self-test that passes while the real check fails is the single most misleading output a checker can produce, and naming it in the source is better than a comment in a review thread that nobody will find again.

One ask, small: the match is a URL/name presence test, so a NOTICE that merely mentions github.com/nothings/stb anywhere -- a comment, a changelog line, a "not used" note -- satisfies it. NOTICE already has a rigid shape (URL, a copyright line, License: at :160 and :165), so requiring the anchor line itself rather than a substring anywhere in the file costs a few lines and removes the only way this can pass without recording the licence.

Deliberately not asking for: the reverse direction (a NOTICE entry whose dependency is gone). That is staleness, it does not ship anything unlicensed, and a check that fires on it would be noise.

@bong-water-water-bong
bong-water-water-bong merged commit f7c5807 into main Nov 13, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/notice-deps branch November 13, 2026 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant