tests/ci: a race regression test that detects its race, and a Catch2 version that is checked rather than asserted (fixes #668, fixes #666, refs #667)

.github/workflows/ci.yml

@@ -85,6 +85,27 @@ env:
   # the cache itself -- jobs still read each other's entries.
   FASTCACHE_PREFETCH_GROUP: "${{ github.run_id }}-${{ github.job }}"
   CLANG_VERSION: "22"
+  # The Catch2 the clang-tidy-diff job analyses against. Not a version this
+  # workflow installs -- `apt-get install -y catch2` takes whatever
+  # ubuntu-24.04 ships -- but a record of what that has been measured to be,
+  # which the "Assert the Catch2 this job analyses against" step below reads
+  # back off the runner and fails on if it has moved.
+  #
+  # It is worth recording because it is load-bearing and invisible. Which
+  # Catch2 is on the include path decides whether
+  # readability-function-cognitive-complexity findings on a TEST_CASE body
+  # reach this job at all, so a workstation with a different one can run the
+  # same clang-tidy over the same diff with the same flags and exit 0 where
+  # this job exits 1 -- silently, reporting nothing rather than reporting less
+  # (morph#666). An unpinned package that decides a gate's outcome and moves
+  # without notice is the shape this repository keeps getting caught by.
+  #
+  # ubuntu-24.04's package is 3.4.0-1build1 (Launchpad, noble Release pocket),
+  # and the clang-tidy-report artifact of run 35581623269 prints
+  # `/usr/include/catch2/internal/catch_test_registry.hpp:121` with
+  # `INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ) )`
+  # and no `, __VA_ARGS__` -- v3.4.0's text exactly, and not v3.5.3's.
+  CATCH2_VERSION: "3.4.0"
   # MORPH_BUILD_FORMS_QML needs Qt 6.5+; ubuntu-24.04 apt still ships 6.4.2.
   QT_VERSION: "6.8.1"
 
@@ -2193,6 +2214,20 @@ jobs:
       - name: Self-test the clang-tidy suppression-scope checker
         run: bash scripts/test_check_tidy_suppression_scope.sh clang-tidy-${{ env.CLANG_VERSION }}
 
+      # The Catch2 on this runner's include path decides whether a
+      # readability-function-cognitive-complexity finding on a TEST_CASE body
+      # reaches this job, and `apt-get install -y catch2` above pins nothing.
+      # This step reads the version out of the headers the step above just
+      # installed and fails if it is not the one CATCH2_VERSION records, so a
+      # move in the runner image is a red job rather than a quiet change of
+      # what this gate measures (morph#666). Its self-test runs first, for the
+      # same reason the suppression-scope checker's does.
+      - name: Self-test the catch2-pin checker
+        run: bash scripts/test_check_catch2_pin.sh
+
+      - name: Assert the Catch2 this job analyses against
+        run: bash scripts/check_catch2_pin.sh . --strict
+
       # Catches morph#632's bug class: tests/.clang-tidy's thirteen
       # suppressions are argued as Catch2 and raw-syscall idiom, which is true
       # of test sources and says nothing about include/morph/** -- yet

examples/bank/tests/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a

examples/bookmarks/tests/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a

examples/common/testkit/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a

examples/crm/tests/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a

examples/kanban/tests/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a

examples/ledger/tests/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a

examples/lims/tests/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a

examples/pastebin/tests/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a

examples/polls/tests/.clang-tidy

@@ -11,9 +11,12 @@
 # logical operator -- would defeat the expression decomposition that makes a
 # failing assertion print its two operands. Catch2 says so itself; the macro
 # definition carries `/* NOLINT(bugprone-chained-comparison) */` on that very
-# line in 3.15.3. The clang-tidy-diff job installs ubuntu-24.04's catch2,
-# 3.4.0, which predates that comment -- which is why the finding reaches CI
-# here and not on a workstation with a current Catch2.
+# line in 3.15.3. CI pins catch2 3.4.0 -- ubuntu-24.04's package, which
+# predates that comment -- which is why the finding reaches CI here and not on
+# a workstation with a current Catch2. That number is checked rather than
+# asserted: scripts/check_catch2_pin.sh reads the runner's installed version
+# back off the include path and fails this line and the clang-tidy-diff job
+# together if the package moves (morph#666).
 #
 # Directory-scoped, and no wider. clang-tidy resolves configuration by walking
 # up from the file it is analysing and offers no finer granularity than a