ci+mutation: the campaign could not report a failure at all -- three compounding defects, a pipefail gate, and TSan's second stack (fixes #730, #731, #732, #736)

.github/workflows/ci.yml

@@ -598,7 +598,22 @@ jobs:
           #
           # Harmless on the clang-asan and clang-ubsan legs of this matrix:
           # neither runtime reads TSAN_OPTIONS.
-          TSAN_OPTIONS: suppressions=${{ github.workspace }}/cmake/tsan.supp
+          #
+          # `second_deadlock_stack=1` is colon-separated *into this value*, not
+          # a second `TSAN_OPTIONS:` key -- a second key silently replaces the
+          # first and drops the suppressions file, which is morph#688's failure
+          # one spelling over. It makes TSan print, for a lock-order inversion,
+          # the stacks where the *already-held* mutexes were taken. Measured on
+          # a two-mutex inversion under clang 22.1.8: 36 lines -> 55, the extra
+          # 19 being two `Mutex Mn previously acquired by the same thread here:`
+          # stacks that name the acquiring function and line. Without it TSan
+          # prints only `Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get
+          # more informative warning message` -- advice nobody can take after
+          # the fact, because morph#578 and morph#717 are intermittent and the
+          # run that fires is the only evidence that will ever exist. No
+          # measurable runtime cost: 2M lock acquisitions under TSan took a
+          # median 0.275s without and 0.273s with (morph#736).
+          TSAN_OPTIONS: suppressions=${{ github.workspace }}/cmake/tsan.supp:second_deadlock_stack=1
         run: |
           # morph::testkit::OomInjector (tests/oom_injector.cpp) overrides
           # the process-wide operator new/delete to force std::bad_alloc on
@@ -1027,7 +1042,13 @@ jobs:
           # Without it this leg is a coin toss on a false positive that the
           # other TSan leg already knows is one — and the failure mode of a
           # known false positive is that the next real one is waved past.
-          TSAN_OPTIONS: suppressions=${{ github.workspace }}/cmake/tsan.supp
+          #
+          # `second_deadlock_stack=1` for the reason the linux-sanitizers Test
+          # step above gives at length (morph#736), and appended to the same
+          # value rather than added as a second `TSAN_OPTIONS:` key, which
+          # would silently drop the suppressions file this leg was given in the
+          # first place.
+          TSAN_OPTIONS: suppressions=${{ github.workspace }}/cmake/tsan.supp:second_deadlock_stack=1
         run: ctest --preset clang-tsan -L ladder-kanban -R ThreadSanitizer --output-on-failure
 
       # Cumulative hit/miss for this leg. Without it the cache is
@@ -1412,6 +1433,12 @@ jobs:
       - name: Determine whether the ladder needs to run
         id: filter
         run: |
+          # Not because `echo | grep` can lose a status today -- `echo` does
+          # not fail -- but because this block decides whether a whole job
+          # runs, and the next edit to it is the one that adds a pipeline
+          # that can (morph#730). scripts/check_workflow_pipefail.py enforces
+          # the same rule on every other `run:` block.
+          set -o pipefail
           if [ "${{ github.event_name }}" = "pull_request" ]; then
             base="${{ github.event.pull_request.base.sha }}"
           else
@@ -1883,6 +1910,12 @@ jobs:
       - name: Determine whether the ladder needs to run
         id: filter
         run: |
+          # Not because `echo | grep` can lose a status today -- `echo` does
+          # not fail -- but because this block decides whether a whole job
+          # runs, and the next edit to it is the one that adds a pipeline
+          # that can (morph#730). scripts/check_workflow_pipefail.py enforces
+          # the same rule on every other `run:` block.
+          set -o pipefail
           if [ "${{ github.event_name }}" = "pull_request" ]; then
             base="${{ github.event.pull_request.base.sha }}"
           else
@@ -2623,6 +2656,11 @@ jobs:
 
       - name: Check every tracked C++ file against .clang-format
         run: |
+          # Without pipefail the count below is `wc`'s status, not `tr`'s: a
+          # missing or unreadable cpp-files.z yields COUNT=0 rather than an
+          # error (morph#730). The `-lt 100` guard catches that particular
+          # case; pipefail is what keeps the *next* pipeline here honest.
+          set -o pipefail
           git ls-files -z '*.hpp' '*.cpp' > cpp-files.z
           COUNT=$(tr -cd '\0' < cpp-files.z | wc -c)
           # An empty or truncated file list would make this job pass while
@@ -2961,8 +2999,14 @@ jobs:
           else
             BASE_SHA="${{ github.event.before }}"
           fi
+          # The left-hand side of the pipeline below is *expected* to fail:
+          # `find` exits non-zero when one of its two search roots is absent,
+          # and `head -1` closing the pipe SIGPIPEs it. The `-z` test is what
+          # reads the result. This is why the `set -o pipefail` further down is
+          # placed after this line rather than at the top of the step, and why
+          # the line carries the marker scripts/check_workflow_pipefail.py reads.
           CLANG_TIDY_DIFF="$(find /usr/lib/llvm-${{ env.CLANG_VERSION }}/share/clang /usr/share/clang \
-            -name 'clang-tidy-diff.py' 2>/dev/null | head -1)"
+            -name 'clang-tidy-diff.py' 2>/dev/null | head -1)"  # pipefail-ok: find's roots may be absent; head SIGPIPEs it; the -z test below reads the result
           if [ -z "$CLANG_TIDY_DIFF" ]; then
             echo "::error::clang-tidy-diff.py not found"
             exit 1
@@ -3372,6 +3416,47 @@ jobs:
       - name: Check every job named in a workflow comment exists
         run: python3 scripts/check_workflow_job_references.py .
 
+  # ── A workflow pipeline must read its left-hand side's status ─────────
+  #
+  # Its own job for the same reason banner-lint and job-reference-lint above
+  # are: it reads workflow text and builds nothing, so it answers in seconds
+  # rather than riding on a leg that takes 26-74 minutes to say the same thing.
+  #
+  # A `run:` block with no `shell:` key runs under `bash -e {0}` -- `-e` but
+  # not `-o pipefail` -- so `cmd | tee log` exits with `tee`'s status and a
+  # failing `cmd` is green. That has now cost this repository two gates:
+  #
+  #   * morph#479: `clang-tidy-diff.py … | tee` could not fail on anything it
+  #     found, for as long as it took someone to notice;
+  #   * morph#730: `scripts/mutation.sh … | tee` reported success for a
+  #     campaign that exited 1 having produced no report, so the mutation score
+  #     went unmeasured for two weeks and the workflow said nothing.
+  #
+  # Between those, three steps set `pipefail` by hand and one carries a comment
+  # about this exact hazard. The guard was known and applied inconsistently,
+  # which is what a gate is for. `shell: bash` -- the explicit spelling -- is
+  # `bash --noprofile --norc -eo pipefail {0}` and does carry the guard; only
+  # the absent key does not, and that asymmetry is most of why this recurs.
+  #
+  # Not vacuous on arrival: it found six unguarded pipelines on the tree it
+  # landed against, one of which was morph#730 itself.
+  pipefail-lint:
+    name: Workflow pipelines read their exit status
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+
+      # See scripts/test_check_workflow_pipefail.sh's own comment: this gate
+      # repairs the tree it guards in the same commit that adds it, so it
+      # passes on day one whether it parses a single `run:` block or none at
+      # all. The cases that decide whether it measures anything are the two
+      # vacuity ones, not the defects.
+      - name: Self-test the pipefail checker
+        run: bash scripts/test_check_workflow_pipefail.sh
+
+      - name: Check every workflow pipeline reads its left-hand side's status
+        run: python3 scripts/check_workflow_pipefail.py .
+
   # ── Install / export: find_package(morph CONFIG) must work ─────────────
   #
   # Its own job rather than a step on an existing leg. It configures, installs

.github/workflows/drift-guard.yml

@@ -474,6 +474,21 @@ jobs:
       - name: Self-test the mutation-survivor citation gate
         run: python3 scripts/check_mutation_survivors.py --self-test
 
+      # scripts/report_mutation_failure.sh is the campaign's reporting half,
+      # and it runs only on a failed scheduled run -- i.e. a few times a year,
+      # on the day its output matters most. It is self-tested here, per-PR,
+      # against a stub `gh`, because the defect it fixes (morph#731) is the
+      # *absence* of a record: the old step skipped when any issue with the
+      # scope's title was open, so the second, different failure in a scope
+      # produced no issue, no comment and no label -- only a line in a run log.
+      # There is nothing to inspect afterwards, so it has to be driven.
+      #
+      # The self-test is morph#731's acceptance condition executed rather than
+      # argued: two different failures in one scope, both recorded, with the
+      # second record carrying the second failure's own evidence.
+      - name: Self-test the mutation-failure reporter
+        run: bash scripts/test_report_mutation_failure.sh
+
   # ── Every line-cited allowlist's citations, resolved in one place ──────
   # Three files in this repository cite source lines by `{file, line, source}`:
   # scripts/branch_partial_allowlist.json, scripts/error_path_allowlist.json

.github/workflows/mutation.yml

@@ -115,15 +115,41 @@ jobs:
         env:
           CXX: clang++-${{ env.CLANG_VERSION }}
         run: |
-          bash scripts/mutation.sh "${{ inputs.scope || 'core-forms' }}" \
-            | tee "mutation-${{ inputs.scope || 'core-forms' }}.log"
+          # `set -o pipefail`, and it is the whole point of this step
+          # (morph#730). A `run:` with no `shell:` key is `bash -e {0}` --
+          # `-e` but NOT `-o pipefail` -- so this pipeline used to exit with
+          # `tee`'s status, which is 0 unless the disk fills.
+          # scripts/mutation.sh takes deliberate care to exit 1 when
+          # mull-runner writes no report, and that 1 was discarded: the step
+          # went green and the failure surfaced one step later as
+          # check_mutation_regression.py failing to *open* a file. Two weeks
+          # of scheduled runs measured nothing and nothing said so. The same
+          # trap cost morph#479 the clang-tidy gate; scripts/check_workflow_pipefail.py
+          # now fails on an unguarded pipeline in any workflow.
+          #
+          # `tee` stays: the Upload step below publishes the log it writes.
+          set -o pipefail
+          scope="${{ inputs.scope || 'core-forms' }}"
+          bash scripts/mutation.sh "$scope" | tee "mutation-${scope}.log"
+          # An empty log means the campaign produced no output at all, which
+          # `set -o pipefail` cannot see -- a script that prints nothing and
+          # exits 0 is a successful pipeline. Same reason the install steps
+          # above `test -s` what curl wrote (morph#681).
+          test -s "mutation-${scope}.log"
 
       - name: Check for a regression against the recorded baseline
         id: regression
         run: |
+          # Teed to a file for the same reason the campaign step is: the
+          # failure report below quotes the checker's own verdict rather than
+          # paraphrasing it, and a verdict that exists only in the run log is
+          # the state morph#731 describes. `set -o pipefail` first -- this is
+          # the pipeline morph#730 was about.
+          set -o pipefail
           scope="${{ inputs.scope || 'core-forms' }}"
           python3 scripts/check_mutation_regression.py \
-            "build/mutation-${scope}/mutation-${scope}.txt" "$scope"
+            "build/mutation-${scope}/mutation-${scope}.txt" "$scope" \
+            2>&1 | tee "regression-${scope}.log"
 
       # scripts/check_mutation_regression.py updates scripts/mutation_baseline.json
       # in place on a first run or an improvement (never on a regression --
@@ -150,23 +176,27 @@ jobs:
       # have to be noticed by someone comparing by hand" gap
       # docs/spec/testing_charter.md names, closed: something now notices and
       # files it, instead of the campaign's own log being the only record.
-      - name: Open an issue on failure
+      #
+      # The body of this step is scripts/report_mutation_failure.sh, and that
+      # is morph#731's fix rather than a tidy-up. What used to be here built a
+      # fixed title per scope and *skipped* when any issue with that title was
+      # open, so the first failure in a scope silenced the report of every
+      # later, different one -- which is how morph#732 stayed invisible behind
+      # morph#517 for two weeks. The script comments on the open issue instead,
+      # and classifies the cause so the thread reads as distinct events. It
+      # lives in scripts/ because morph#731's acceptance condition is "force
+      # two different failures in one scope and confirm both are recorded",
+      # and a dozen lines inside a `run:` block cannot be driven at all --
+      # scripts/test_report_mutation_failure.sh drives exactly that.
+      - name: Report the failure
         if: failure()
         env:
           GH_TOKEN: ${{ github.token }}
           RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
         run: |
           scope="${{ inputs.scope || 'core-forms' }}"
-          title="Mutation campaign failed or regressed (scope: ${scope})"
-          existing="$(gh issue list --state open --search "in:title \"${title}\"" --json number --jq 'length')"
-          if [ "$existing" != "0" ]; then
-            echo "An open issue already names this failure; not filing a duplicate."
-            exit 0
-          fi
-          body="The scheduled mutation campaign (.github/workflows/mutation.yml) failed or found a regression for scope \`${scope}\`."
-          body="${body}\n\nRun: ${RUN_URL}"
-          body="${body}\n\nSee \`scripts/check_mutation_regression.py\`'s own doc comment for what this compares, \`scripts/mutation_baseline.json\` for the recorded baseline, and \`scripts/mutation_survivors.json\` for the triage process for a genuine new survivor."
-          gh issue create --title "$title" --label "area: ci" --body "$(printf '%b' "$body")"
+          bash scripts/report_mutation_failure.sh "$scope" "$RUN_URL" \
+            "mutation-${scope}.log" "regression-${scope}.log"
 
       - name: Upload the mutation report
         if: always()
@@ -175,5 +205,8 @@ jobs:
           name: mutation-report-${{ inputs.scope || 'core-forms' }}
           path: |
             mutation-*.log
+            regression-*.log
             build/mutation-*/mutation-*.txt
+            build/mutation-*/baseline-*.log
+            build/mutation-*/mull.yml
           retention-days: 30

.github/workflows/wasm-ladder.yml

@@ -206,4 +206,10 @@ jobs:
       # Informational: the build steps above are the gate. Listed rather than
       # asserted by path, since where Qt drops a wasm bundle is Qt's business.
       - name: Show the produced artifacts
-        run: find build-wasm-ladder -name '*.wasm' -o -name '*.html' | sort
+        run: |
+          # pipefail because `find` over a build tree the previous step was
+          # supposed to fill is a claim worth failing on: without it this step
+          # prints nothing and exits 0 whether the directory is empty or absent
+          # (morph#730).
+          set -o pipefail
+          find build-wasm-ladder -name '*.wasm' -o -name '*.html' | sort

CMakePresets.json

@@ -251,9 +251,9 @@
       "name": "clang-tsan",
       "configurePreset": "clang-tsan",
       "inherits": "base-test",
-      "description": "TSan test run. Two things it sets that a caller would otherwise have to know. (1) TSAN_OPTIONS points at cmake/tsan.supp through ${sourceDir}, i.e. absolutely, so it resolves from any working directory and cannot drift from the file it names. A relative 'suppressions=cmake/tsan.supp' does not survive test discovery: catch_discover_tests runs the binary with the working directory set to its own build directory, TSan then fails to open the file and exits with its default exitcode=66, and Catch2's CatchAddTests.cmake reports 'Result: 66 / Output:' with nothing in it -- the listing goes to a file via --out, so that error block is empty for every discovery failure -- attributed to whichever binary ctest enumerated first (examples/concepts), which is nowhere near the cause. (2) The OomInjector exclusion, for the reason spelled out on the clang-asan preset above.",
+      "description": "TSan test run. Three things it sets that a caller would otherwise have to know. (1) TSAN_OPTIONS points at cmake/tsan.supp through ${sourceDir}, i.e. absolutely, so it resolves from any working directory and cannot drift from the file it names. A relative 'suppressions=cmake/tsan.supp' does not survive test discovery: catch_discover_tests runs the binary with the working directory set to its own build directory, TSan then fails to open the file and exits with its default exitcode=66, and Catch2's CatchAddTests.cmake reports 'Result: 66 / Output:' with nothing in it -- the listing goes to a file via --out, so that error block is empty for every discovery failure -- attributed to whichever binary ctest enumerated first (examples/concepts), which is nowhere near the cause. (2) The OomInjector exclusion, for the reason spelled out on the clang-asan preset above. (3) second_deadlock_stack=1, colon-separated into the same value -- a second TSAN_OPTIONS key would replace the first and drop the suppressions file, which is this preset's own morph#688 failure one spelling over. It is what makes a lock-order inversion name where each already-held mutex was taken (36 report lines -> 55 on a two-mutex inversion, the extra 19 being two 'Mutex Mn previously acquired by the same thread here:' stacks); CI's two TSan legs pass it, and a local reproduction that did not would be less informative than the CI run it is reproducing -- the asymmetry morph#688 existed to remove (morph#736).",
       "environment": {
-        "TSAN_OPTIONS": "suppressions=${sourceDir}/cmake/tsan.supp"
+        "TSAN_OPTIONS": "suppressions=${sourceDir}/cmake/tsan.supp:second_deadlock_stack=1"
       },
       "filter": { "exclude": { "name": "OomInjector|morph#108" } }
     },