tests/lint/ fixtures are inside clang-tidy-diff's scope, and nothing tells the author that until CI is red

[Yaraslaut]

tests/lint/ fixtures are inside clang-tidy-diff's scope, and nothing tells the author that until CI is red

What happened

PR #644 added a seventh fixture directory under tests/lint/
(tests/lint/bidi_controls/, for #628's raw-bidi-control gate). Its three
.cpp fixtures turned clang-tidy-diff red with 14 findings, every one of
them inside the new fixture files — job 106163759717, run 35542939709:

11  clang-diagnostic-unused-const-variable
 3  misc-misleading-bidirectional

These files are text fixtures. Their own headers say so — "Nothing here is
compiled; the file exists to be read by the checker"
(tests/lint/catch_test_names/valid/ordinary_names.cpp), "Not compiled -- it
is scanned as text" (tests/lint/nolint_directives/valid/effective_directives.hpp).
None of them is in compile_commands.json. They are nevertheless analysed,
because:

• clang-tidy-diff.py analyses every changed C/C++ line in the diff, and every
  line of a new file is a changed line;
• the clang-tidy job passes no -regex and no -exclude — deliberately,
  and the step's own comment explains at length why (morph#481: an exclusion
  there silently dropped 276 of the database's 690 entries);
• there is no .clang-tidy anywhere under tests/lint/ — find . -name .clang-tidy returns the repository root, tests/, and the ten
  examples/*/tests/ copies, and nothing else;
• clang tooling does not skip a file with no compile command; it interpolates
  one from a neighbouring entry.

Why the first six directories did not hit this

They are green by accident of content, not by design. The 30 .cpp and 12
.hpp fixtures in nolint_directives, automoc_includes, catch_test_names,
deprecated_markers, journal_stamps and test_type_names happen to declare
nothing clang-tidy objects to, and the checks that would have objected
(misc-use-internal-linkage, readability-identifier-length) are among the
thirteen tests/.clang-tidy subtracts — a file whose stated subject is Catch2
and raw-syscall idiom, which reaches these fixtures only because clang-tidy
resolves configuration from the translation unit's path.

So the convention that makes a tests/lint/ fixture safe — write it so
clang-tidy has nothing to say — is real, load-bearing, and written down
nowhere. The seventh directory is the first one to find out, and the eighth
will find out the same way.

Verification status

Reproduced, locally, on the PR branch, with clang-tidy 22.1.8 (CI's pinned
CLANG_VERSION: "22"), using the warning set cmake/compiler_options.cmake
builds for Clang — -Weverything minus every suppression in its Clang branch.
That flag set reproduces CI's deduplicated list exactly: same 14 findings, same
files, same lines, same columns.

$ clang-tidy $(find tests/lint/bidi_controls -type f \( -name '*.cpp' -o -name '*.hpp' \) | sort) \
      --quiet -- $(cat flags.txt)
tests/lint/bidi_controls/invalid/arabic_letter_mark/alm_in_a_comment.cpp:9:23: error: unused variable 'kSign' [clang-diagnostic-unused-const-variable,-warnings-as-errors]
tests/lint/bidi_controls/invalid/embeddings_and_overrides/reordering.cpp:8:23: error: unused variable 'kEmbed' [clang-diagnostic-unused-const-variable,-warnings-as-errors]
tests/lint/bidi_controls/invalid/embeddings_and_overrides/reordering.cpp:9:23: error: unused variable 'kEmbedRtl' [clang-diagnostic-unused-const-variable,-warnings-as-errors]
tests/lint/bidi_controls/invalid/embeddings_and_overrides/reordering.cpp:10:23: error: unused variable 'kOverride' [clang-diagnostic-unused-const-variable,-warnings-as-errors]
tests/lint/bidi_controls/invalid/embeddings_and_overrides/reordering.cpp:10:35: error: string literal contains misleading bidirectional Unicode characters [misc-misleading-bidirectional,-warnings-as-errors]
tests/lint/bidi_controls/valid/escaped_literals.cpp:8:23: error: unused variable 'kArabicLetterMark' [clang-diagnostic-unused-const-variable,-warnings-as-errors]
... 8 more ...

The absence of an exclusion is read from .github/workflows/ci.yml and
from find . -name .clang-tidy, not inferred.

Not verified: whether any of the six pre-existing fixture directories would
go red if their files were re-added to a diff today. I measured only the
bidi_controls set. A cheap way to check would be to re-run
clang-tidy-diff.py over git diff of a synthetic commit that touches one line
in each of the 42 existing fixtures.

Options

Not a proposal, a list — whoever picks this up should argue for one:

1. Write the convention down, in docs/spec/testing_strategy.md (which
   already documents tests/lint/test_type_names/) and/or a short
   tests/lint/README.md: a fixture is scanned by clang-tidy-diff on the PR
   that adds it, so it must be clang-tidy-clean; inline your namespace-scope
   constants; suppress per line with the reason above the directive.
2. A gate that proves it, of the kind this repository prefers: run
   clang-tidy over every file under tests/lint/ on every PR (not just changed
   lines), so the seventh directory's failure mode is caught at the point the
   fixture is written rather than at the point CI runs. Cost: it needs the
   clang-tidy job's full Qt + ladder configure, which already exists in that
   job.
3. A tests/lint/.clang-tidy. Cheapest, and the one PR ci: reject raw bidi control characters in first-party files (fixes #628) #644 deliberately
   did not take: it puts a new suppression file into the area morph#632
   landed scripts/check_tidy_suppression_scope.sh to audit, and a
   directory-wide -misc-misleading-bidirectional would have discarded real
   coverage. If someone takes this route it should be argued check by check,
   the way tests/.clang-tidy is.
4. Do nothing, and accept that each new fixture directory costs one red CI
   round. Defensible — it has cost one so far — but it should then be a written
   decision rather than a gap.

What would change the verdict

Close this if either (1) the convention is documented where a fixture author
will read it and the documentation is cited by something that fails when it
goes stale, or (2) a gate makes the failure impossible to reach CI. Re-open if a
new tests/lint/ directory turns clang-tidy-diff red again.

Related

• No lint keeps raw bidi controls out of first-party sources, and the authoring path that introduces them was hit twice while fixing #610 #628 / ci: reject raw bidi control characters in first-party files (fixes #628) #644 — the directory that surfaced this.
• tests/.clang-tidy's suppressions also silence include/morph headers, so a header gate driven by test TUs under-reports #632 — scripts/check_tidy_suppression_scope.sh, the reason option 3 is not
  free.
• clang-tidy-diff's configure omits the application ladder, so no rung is in the compile database it lints against #481 — why the clang-tidy job carries no path exclusion, deliberately.

[Yaraslaut]

A third instance, found hours after this was filed — and the worst-shaped one

This issue says the tests/lint/ fixture directories sit inside clang-tidy-diff's scope with nothing telling the author until CI is red. PR #649 has just hit the same rule from a completely different direction, which suggests the finding generalises beyond fixtures.

#649 is a documentation PR — #570 turned out to be prose-only, because nothing under examples/ ever called the verbs it was supposed to migrate. It edits six files in comments only. That was enough to turn clang-tidy-diff red, because a changed comment line is a changed line:

examples/bookmarks/gui_wasm/main_wasm.cpp:58:10: error: 'QGuiApplication' file not found
examples/common/wasm_spike/main_wasm.cpp:47:29:  error: expected ')'
examples/pastebin/gui_wasm/main_wasm.cpp:50:10:  error: 'QGuiApplication' file not found
examples/polls/gui_wasm/main_wasm.cpp:178:10:    error: 'emscripten/emscripten.h' file not found
include/morph/core/bridge.hpp:1610 / :2667       (instantiated from
    tests/compile_checks/client_only_facade_no_model_header.cpp:53, a TU that
    deliberately omits model.hpp, so the type is incomplete by design)

Four WASM sources needing a toolchain this job does not configure, and one TU whose whole purpose is to be incomplete.

So there are now three instances of one rule, none of which shares a fix site with the others as filed:

	what was touched	how it was discovered
#624 (closed by #647)	two sources self-including their .moc	a PR editing one of them went red
#650 (this)	tests/lint/ fixtures	#644 added a seventh fixture directory
#649	four WASM examples + one compile-check TU	a comment-only documentation edit

The rule, unwritten anywhere: editing any line — including a comment — in a file the clang-tidy configure cannot build turns that file red, with no signal until CI says so.

The third instance sharpens this issue's argument in a way worth keeping. The option list here weighs "document the convention" against "gate it", on the assumption the affected set is fixture directories a contributor could learn about. It is not: it includes every WASM example and every deliberately-incomplete compile-check TU, and the triggering edit can be a comment. Documentation that says "do not add a fixture directory without checking" does not help someone fixing a typo in a WASM example's comment.

I am not re-scoping or closing anything on this basis — that is a triage verdict, and per AGENTS.md it is not mine to self-assign. Recording the evidence here so whoever triages this has all three instances, rather than filing a fourth issue that would fragment them.

Not measured: whether the six pre-existing fixture directories would go red if re-added to a diff today; that remains this issue's open question, unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

PR #649 hit the third instance of this and changed the clang-tidy job, so
this issue's option list needs updating rather than answering as written.

What #649 does. The job now filters the diff before clang-tidy-diff.py
sees it: a changed source with no entry in compile_commands.json is
dropped, a changed header never is (dropping headers is what
-only-check-in-db does, and is #479's defect one directory over), and every
drop is printed as a ::warning:: naming the file. The filter refuses to run
unless the database is still the wide one the Configure step builds — at least
600 in-workspace sources, at least 200 under examples/ — because a filter
consulting a narrowed database would quietly stop analysing everything the
narrowing dropped.

What that means here, measured. On 3c6dd24f with the job's own configure
flags and clang 22.1.8, all 30 tests/lint/** sources are absent from
compile_commands.json, so a PR adding a seventh (or eighth) fixture
directory no longer costs a red CI round — the fixtures are skipped and named
in the log instead.

What it does not do. Option 1 of this issue is untouched: the convention
that a tests/lint/ fixture is scanned as text and is not a translation unit
is still written down nowhere, and an author still learns it by finding their
fixture listed in a ::warning:: rather than by reading anything. Option 2 (a
gate that analyses every tests/lint/ file deliberately, rather than by
accident of the diff) is also untouched, and is now more clearly a separate
decision, because the accidental route is gone. Option 3 (a
tests/lint/.clang-tidy) is no longer needed for this failure mode.

So the remaining question on this issue is narrower than it was: is the
convention worth documenting, and is a deliberate gate over tests/lint/
worth its configure? The "each new fixture directory costs one red CI round"
cost that option 4 accepted is gone either way.

Verification status: the 30-file count and the absence from the database are
reproduced locally with the job's configure flags. I did not re-run
clang-tidy-diff.py over a synthetic commit touching the 42 pre-existing
fixtures — this issue's own "not verified" note still stands, though the
filter makes the answer moot for the gate's purposes.

Sizing of the wider hole this sits inside — 60 of 470 tracked C/C++ sources,
split by cause — is on #651.

[Yaraslaut]

Triage: rescope — #649 landed and removed the symptom this was filed for

The red-CI half of this issue is gone. #649's clang-tidy-diff filter, now on master (0067b5bf), drops a changed source with no entry in compile_commands.json and names it in a ::warning file=:

.github/workflows/ci.yml:2501  SOURCE_SUFFIXES = {".c", ".cc", ".cpp", …}
.github/workflows/ci.yml:2569  if path.suffix.lower() in SOURCE_SUFFIXES and path.resolve() not in compiled:

A tests/lint/ fixture .cpp is exactly such a source, so the eighth fixture directory no longer costs a red round — it produces a named warning instead. That was this issue's concrete cost, and it is paid.

What survives is the documentation half, and it is now the whole ticket: nothing still tells a fixture author what the rule is, and the rule has changed rather than disappeared — it is no longer "your fixture will turn CI red" but "your fixture will be silently skipped by clang-tidy, and that is intended." A contributor who does not know that could reasonably believe a new fixture is being linted when it is not.

So the option list here needs rewriting rather than picking from:

• Option 2 (a deliberate gate) and option 3 (tests/lint/.clang-tidy) are moot — the filter supersedes both, and ci: reject raw bidi control characters in first-party files (fixes #628) #644 separately showed option 3 would collide with tests/.clang-tidy's suppressions also silence include/morph headers, so a header gate driven by test TUs under-reports #632's check_tidy_suppression_scope.sh.
• Option 1 (document the convention) is the remainder, but what it documents has inverted.
• The issue's own close condition — "documented where a fixture author will read it and the documentation is cited by something that fails when it goes stale" — is still the right bar and is unaffected.

triage: rescope, documentation, area: ci.

Still unmeasured, and worth carrying into the successor: whether any of the six pre-existing fixture directories would go red if re-added to a diff today. That was this issue's open question; the filter makes it less urgent but no more answered, since a fixture that is skipped is not a fixture that was checked.

Related: #651 measures the wider set the filter now skips (60 of 470 tracked sources, 11 of them recoverable by passing -DMORPH_BUILD_BANK_GUI=ON). The successor to this issue and #651 may well be one ticket — they share a fix site now, which they did not when both were filed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Not done, and two measurements for whoever does it

Dispatched to a gates lane owning .github/workflows/** and scripts/**. The
half of this issue that survives morph#649 is documentation, and its home —
tests/lint/README.md or a section of docs/spec/testing_strategy.md reached
from it — is held by the tests/** lane this sweep. Reporting rather than
crossing the boundary.

Two things measured on c6f6d953 that the issue does not have, both of which
narrow it further.

1. Option 2's gate would be vacuous on arrival

The issue's option 2 is "a gate that analyses every file under tests/lint/
deliberately". The natural cheap version of that — every fixture directory
must be driven by some checker's self-test, or it is dead weight nobody runs —
has nothing to find:

automoc_includes     files= 6   scripts/test_check_automoc_includes.sh
bidi_controls        files= 7   scripts/test_check_bidi_controls.sh, scripts/check_bidi_controls.py
catch_test_names     files=13   scripts/test_check_catch_test_names.sh, scripts/check_catch_test_names.sh
deprecated_markers   files= 5   scripts/test_check_deprecated_markers.sh
journal_stamps       files= 6   scripts/test_check_journal_stamps.sh
nolint_directives    files= 4   scripts/test_check_nolint_directives.sh
test_type_names      files=10   scripts/check_test_type_names.sh, scripts/test_check_test_type_names.sh

Seven directories, 51 files (33 .cpp, 14 .hpp, 2 .qml, 2 .moc), every
one referenced by exactly its own self-test. No orphans. A gate keyed on that
relation would ship green with nothing to say — this repository's named failure
mode — so if option 2 is taken it has to be the expensive version (run
clang-tidy over the fixtures under the clang-tidy job's full Qt + ladder
configure), and that is a real cost to argue rather than a cheap one to assume.

Note also that three of the seven directories are named inside the checker
as well as its self-test (check_bidi_controls.py, check_catch_test_names.sh,
check_test_type_names.sh), because a checker that scans the tree finds its own
fixtures. That coupling is a second unwritten convention in the same area, and
it belongs in whatever gets documented.

2. The documentation's close condition can be met without a new gate

The issue requires the documentation to be "cited by something that fails when
it goes stale". scripts/check_spec_citations.sh already has that shape: its
dangling-reference check (check 3) fails on any docs/spec/*.md path in the
tree that does not resolve. A tests/lint/README.md that names each fixture
directory, plus one directory-to-doc correspondence check, is a much smaller
build than option 2 — but it still starts with the README, which is the file
this lane cannot write.

Verification status

Measured on c6f6d953: the directory/self-test map, the file counts, the
extension split. Not measured: anything about compile_commands.json or the
morph#649 filter — the 30-file absence in the comment above stands unrepeated,
and the issue's own open question (whether the six pre-existing fixture
directories would go red if re-added to a diff) is still open and still
unanswered.

What would change the verdict

Unchanged. Close when the convention is documented where a fixture author will
read it and something fails when that documentation goes stale. The measurement
above only argues that the gate half should not be the cheap version of option
2, because the cheap version has nothing to check.

[Yaraslaut]

Not done, deliberately — and the rescope verdict needs one more correction before anyone builds against it

Dispatched to a CI/scripts lane owning .github/workflows/**, scripts/*.py,
scripts/*.sh and CMakePresets.json, with tests/** held by another PR
this sweep. The half of this issue that survives morph#649 is documentation,
and its home -- tests/lint/README.md, where a fixture author will actually
look -- is in that held tree. Reporting rather than crossing the boundary, as
the previous lane did.

Two things measured on d03c66f3 that sharpen the remaining ticket. Both make
it smaller.

1. The coupling is four checkers, not three

The previous comment recorded that three fixture directories are named inside
the checker as well as its self-test. It is four, and the fourth is a
different shape:

$ grep -n "tests/lint" scripts/check_*.sh scripts/check_*.py
scripts/check_test_type_names.sh:47:    prune=(-path "tests/lint" -prune -o)
scripts/check_catch_test_names.sh:123:   prune=(-path "tests/lint" -prune -o)
scripts/check_nolint_directives.sh:87:        -type d -path '*tests/lint' -prune -o \
scripts/check_bidi_controls.py:149:FIXTURE_ROOT = "tests/lint/bidi_controls"

Three of them prune tests/lint out of a tree-wide scan -- a fixture that
deliberately carries the rejected shape would otherwise fail the checker that
owns it. The fourth names its fixture root to scan it on purpose.

So the unwritten convention is not one rule but two, in opposite directions:

• a fixture is not a translation unit, and clang-tidy will skip it (morph#649's
  filter, and this issue as filed);
• a fixture is also not ordinary test source, and its own checker prunes it --
  except bidi_controls, whose checker seeks it out.

An author adding an eighth fixture directory has to know which of those applies
to their checker, and nothing tells them either. That is a stronger case for
documentation than the issue currently makes, and a weaker one for a gate:
the two rules have no single mechanical statement to check.

2. One sentence of the documentation already exists, and covers one of seven

$ grep -rn "tests/lint" docs/ --include="*.md"
docs/spec/testing_strategy.md:479:`tests/lint/test_type_names/` before the real scan runs, following the same

docs/spec/testing_strategy.md is therefore the natural second home, and it
already satisfies half of this issue's close condition by construction:
scripts/check_spec_citations.sh's dangling-reference check fails on any
docs/spec path in the tree that does not resolve. What it does not do is
notice a fixture directory that no documentation names -- which is the
direction that rots, since directories are added and prose is not.

So the smallest honest version of the remaining ticket is:

1. tests/lint/README.md naming all seven directories, their checker, their
   self-test, and which of the two pruning conventions each falls under;
2. a directory-to-README correspondence check -- both ways, because "every
   directory named in the README exists" alone passes a README that names
   nothing, and this repository's named failure mode is exactly that.

Item 2 is scripts/-shaped and this lane could write it. It is worthless
without item 1, which this lane cannot.

Verification status

Measured on d03c66f3: the four checker references above, the single
docs/ reference, seven directories and 51 files under tests/lint/. Not
measured, and still this issue's own open question: whether any of the
pre-existing fixture directories would go red if re-added to a diff today. The
morph#649 filter makes that less urgent and no more answered.

What would change the verdict

Unchanged. Dispatch to a lane owning tests/** (for tests/lint/README.md)
and scripts/** (for the correspondence check) together. The rescope verdict
stands; the option list should record that the convention is two rules, not one.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Closed — the gate this describes no longer exists

morph#755 removed the meta-gates: the checks that policed other checks. drift-guard.yml (11 jobs), the mutation campaign, spec-sync.yml and seven ci.yml lint jobs are gone, together with the 24 checker scripts they drove and the data files only they read.

This issue described a defect in one of those, so there is nothing left to fix. It is closed as not planned rather than resolved — the report was accurate, its subject was deleted.

No compile, test, sanitizer, coverage or platform coverage was removed. Every build leg, all four sanitizer legs, Valgrind, the coverage floor, both Windows legs and both WASM legs still run. What was given up is drift detection, deliberately: 57 checks per PR had become more maintenance than signal, and more than half the open backlog was CI maintaining itself.

Re-open if the gate is ever reinstated — the analysis here would apply again unchanged.