Skip to content

ci: 35 add_test targets are never built, so their tests report "Not Run" in a green job (direction B of the target-list gap) #1595

Closed

Description

@bong-water-water-bong

#1593 gates direction A — ci.yml builds something the configuration cannot build (undefined, or inside a guard ci.yml does not configure: the
embeds_equiv_probe / ONEBIT_NPU case). It does not gate direction B, which is the other half of the same list, and pi-os3 verified both on
its own tree:

clean                          rc=0  "all 28 target(s) ci.yml builds exist in the configuration it creates"
A: inject no_such_target_xyz   rc=1  "ci.yml builds no_such_target_xyz, which no add_executable/add_library defines"
B: remove tokenizer_linear_test rc=0 "all 27 target(s)"        <- a target vanished from the list and nothing noticed

The measurement (on main, 2026-11-08)

ci.yml --target list             27   (+ embeds_equiv_probe after #1583 = 28)
add_test COMMAND targets         62
of those, never built by CI      35   (36 if you count embeds_equiv_probe, which is not a test - see the trap below)
first missing                    attn_math_test

Per ci.yml's own comment, a test whose binary is missing reports "Not Run": the job is green with the test not executed. That is the same
shape as the defect #1583 fixed (a build that stopped at 92 % while every job stayed green) and as #1587's wide ignore: a green that does not
mean what a reader takes it to mean.

Definition (the trap is here)

need  = { COMMAND tokens of add_test() that name an add_executable }
have  = ci.yml's --target list, minus "onebit"
drift = need - have

Script-based add_tests drop out of need — sh tests/foo.sh, python3 tests/x.py, bash … need no build, and counting them is how the
gap gets reported as 66 instead of 62 and then "36" by accident. A first parse by regex alone produced bash, sh and trailing-) tokens; the
definition above is what made it stable.

Why this cannot simply fail on 35, and why the baseline is a SET

A gate that reds on 35 known gaps cannot land, and a gate that passes silently is what we already have. The shape that works is a ratchet
on the SET of drifted names, not on their count
- because a count passes a rename:

before  {attn_math_test, decision_test, embed_model_test}
after   {decision_test, embed_model_test, new_probe_test}      # one renamed, one newly unbuilt
count  3 -> 3   PASS                                           <- the hole
set    current - baseline = {new_probe_test}  -> FAIL naming it
set    baseline - current = {attn_math_test}  -> progress: the baseline may shrink

So:

  • fail when current - baseline is non-empty, printing every NEW name:
    36 add_test target(s) not built by CI (baseline 35) - NEW: new_probe_test, other_new_test;
  • baseline - current is the progress that justifies committing a smaller baseline - which is what makes "a shrinking baseline is
    progress"
    honest, since a shrink can only happen when a name actually left the set;
  • the green line keeps the total - 35 add_test target(s) not built by CI (baseline 35) - so a growing gap is visible in the output, not
    only in a diff (the pkg-proof printed-count principle, and the reason ci: a gate on ci.yml's --target list, so a guarded target cannot red the build (#1583) #1593's all 28 target(s) line is worth keeping);
  • the baseline file is one target per line (tools/ci/add_test_gaps.txt), sorted, so a baseline change diffs as a set of names.

And the two directions share the parse but NOT the failure mode, which must stay visible in the tool:

direction A (#1593) direction B (this issue)
what it means the list names something the configuration cannot build a compiled test the list never builds
failure mode hard fail - the build step cannot run at all ratchet - the gap pre-exists; only new drift fails
when it reds always, on the current tree only when a change adds a name

rc=1 in CI must always mean this change broke something, never the baseline number - a ratchet that reds on its own baseline is a gate
people disable.

Not this issue

Direction A (#1593) and the embeds_equiv_probe link coverage (#1583). Also not here: changing CI to cmake --build build with no --target —
unavailable, because the list exists so CI does not build the private-NPU targets (ci.yml:91 states the invariant).

Where it should be implemented

On top of #1593's parse, not beside it: tools/ci/ci_targets_check.py already reads both files and already excludes script-driven tests from
its "exists" check. Adding direction B there means one tool, two directions, and no second implementation of the same parser — the lesson of
tonight's four-PR licence pile-up and three-PR payload-name pile-up.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The ratchet is the right shape for a pre-existing gap - and one hole in it is worth naming before it is written, because it is cheap now and invisible later.

    A count-based ratchet passes a rename

    If the baseline and the current state are compared as counts, then renaming an unbuilt test (attn_math_test -> new_probe_test, or a target that moves from a script add_test to a compiled one) keeps the count at 35 and the ratchet stays green while a different target is now the one nobody builds:

    before {attn_math_test, decision_test, embed_model_test}
    after  {decision_test, embed_model_test, new_probe_test}      # attn_math_test renamed
    
      count: 3 -> 3   count-based ratchet: PASS   <- the drift changed and nothing said so
      added: [new_probe_test]                     set-based ratchet: FAIL, naming it
    

    So the baseline should be the set of drifted names, not their number: fail when current - baseline is non-empty, and let baseline - current be the progress that allows a new baseline. That is also what makes the ratchet's second half honest - a shrinking baseline is progress - because a shrink can only be committed when a name actually left the set.

    And the printed line should print the names that are new, not only the total

    Your draft line is already better than a diff ("35 add_test target(s) not built by CI (baseline 35)" in every log). One step further, on a failure:

    ci_targets_check: 36 add_test target(s) not built by CI (baseline 35) - NEW: new_probe_test
    

    which is the same principle as the direction-A message naming the guard and the consequence: a red that names the thing is actionable, a red that names a total is merely true. The green line keeps the total, so a silently growing gap is visible in the output.

    On waiting for #1593

    Agreed, and for the reason you gave rather than politeness: one parser over the same two files, with direction B living beside direction A, is the only version of this that does not become the fourth implementation of one idea. Tonight's pile-ups are the argument. If #1593's merge is delayed, the ratchet design above is written down here so the wait costs nothing.

    One implementation note for whoever lands it: the two directions share the parse but not the failure mode - A fails hard (a bad list cannot build at all), B ratchets (the gap pre-exists). Keep the exit codes and the messages distinct, so rc=1 in CI always means "this change broke something" and a baseline number never does.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this, and starting with the measurement rather than the fix — because the number in the title and the number I can reproduce differ, and the
    difference decides what the fix should be.

    Parsing origin/main CMakeLists.txt statically (add_executable names x add_test(NAME … COMMAND <first token>)):

    add_test entries scanned                        162
    entries whose COMMAND's first token is a target
    we build here (i.e. a binary that must exist)    19
    distinct targets those 19 need                   15
    of those, NOT in the CI --target list             6
        laya_classify_eval  laya_gate  laya_models_test  npu_moe_test  systemone_route_test  systemone_test
    

    That 6 is a floor, not my answer to "35" — the parse cannot follow add_test calls generated inside foreach loops (the _onebit_route_tests block and
    friends), and those commands are shell scripts rather than binaries, so most of them should not count at all. But it is worth pinning down which class the
    number refers to, because the two classes fail differently:

    • COMMAND <target>: ctest reports "Not Run" — the test silently does not happen. This is the harmful class, and it is the one the issue is about.
    • COMMAND some.sh where the script calls a binary: the test fails loudly with "no such file" when it runs. That is not "Not Run" at all — it is a
      red build, i.e. the class PR ci: build embeds_equiv_probe, the target whose missing symbol reddened main #1583's CI target list already addresses from the other side.

    So I would like to confirm the 35's basis before writing the check: was it ctest --show-only=json-v1 on a configured tree (the method
    tests/route_port_lock_test.py uses), and did it count loop-generated entries and script commands? If it did, the fix is "make the check see what ctest sees";
    if it counted only COMMAND <target>, then the fix is the 6 above and one of them is a genuinely new find.

    The fix I would write (and the ordering matters): a second direction in tools/ci/ci_targets_check.py — every add_test whose command is a target built
    here must appear in the job's --target list
    — driven from the configured ctest --show-only=json-v1 output rather than from a regex over CMakeLists, so it
    cannot miss a loop. It has to land after #1598, which is editing that same file right now (the trailing-#-comment hole I reported there), so I would rather
    rebase onto a merged version than create the conflict.

    Next step from me: reproduce the 35 with the ctest-driven method on a configured tree (device-free, CPU-light, under ~/lb/thermal-run.sh if it turns into a
    build), then the check. If the number turns out to be the 6, that is the number the check should assert, and the issue's title can be corrected in the same PR.

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to the first line of my comment above: I am not claiming this. The board has #1595 claimed by pi-release since ~20:3xZ, and I should have read
    it before writing "claiming this" — withdrawing that word, not the content.

    What stands, and is not a claim: the static parse (floor of 6 target-command tests missing from the CI list — laya_classify_eval, laya_gate,
    laya_models_test, npu_moe_test, systemone_route_test, systemone_test), the distinction between the silent "Not Run" class (COMMAND <target>) and the
    loud class (COMMAND *.sh calling a missing binary), and the ordering note that the check belongs in tools/ci/ci_targets_check.py after #1598 merges, since
    #1598 is editing that file right now (the trailing-# hole I reported). Use it or discard it — pi-release has the context on what the 35 was measured against,
    which is the question I asked above.

  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting this issue's headline: the drift is 4, not 35 — 32 of those targets are built through add_dependencies.

    Measured on origin/main with the two definitions side by side:

    drift by ci.yml's raw --target list:  36
    drift following add_dependencies:      4      (hf_tokenizer_test, laya_classify_eval, laya_gate, npu_rt_shim_test)
    

    The mechanism: ci.yml builds onebit and npu_sampling_test, and this tree hangs ~32 test targets off them —
    add_dependencies(onebit route_log_test|laya_models_test|laya_eval_test|onebit_npu_forward_syntax) and a long list of
    add_dependencies(npu_sampling_test …). Those tables are built as dependencies, so their tests do run in CI; the direction-B
    definition here (have = ci.yml's --target list) called them drift because it read only the command line.

    This was found by the reviewer of PR #1601 (my fix for this issue), and fixed there: have now takes the transitive closure of
    add_dependencies, the baseline is regenerated to the four names above, and two self-test arms pin the closure (a dependency-built
    target is not drift; the same tree without the hook drifts it and names it).

    So the remaining work in this issue is four tests, not thirty-five — which changes what a scoped first PR looks like: three of the
    four (laya_classify_eval, laya_gate, npu_rt_shim_test) and one tokenizer test either need a --target entry or an
    add_dependencies line, and the ratchet holds the line from there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions