ci: nightly MSan over the Boost suites with an instrumented libc++

[danmcleran]

Closes the first of the two gaps flagged after #175 — MSan coverage for the Boost suites.

DRAFT STATE: this branch carries a temporary pull_request trigger so the job can actually run before it lands. It is removed in a follow-up commit on this branch before merge. See "Verification" below.

Why

The blocking MSan (uninitialized reads) job covers unit_test/embedded only. That scope was never a preference — MSan requires every linked object to be instrumented, and against a stock libstdc++ the Boost suites abort inside Boost.Test's own static initialization before reaching any TinyMind code. Those reports are artifacts of the uninstrumented runtime, not findings.

That gap is exactly where the real bug lived. The static-init-order UB fixed in #171 — a Q-learning suite training on all-zero weights and "passing" in 17ms — sat in a Boost suite, invisible to ASan, UBSan, and the embedded MSan job alike.

How

Builds libc++ and libc++abi from llvm-project with -DLLVM_USE_SANITIZER=MemoryWithOrigins, caches by LLVM version, and runs the ten Boost suites against it via make check-msan-libcxx.

Only the runtimes are built, not the compiler — the distro clang does the compiling and is merely given an instrumented libc++ to link against. That keeps a cold build in minutes rather than hours.

Nightly, not a PR gate. The cold build is 10–20 minutes, which is the wrong shape for something blocking every merge and the right shape for a scheduled run; the suites themselves take seconds once the runtime exists.

Boost.Test needs no separate handling: the suites include the header-only <boost/test/included/unit_test.hpp>, so it compiles into each TU and is instrumented with it. The C++ runtime was the only uninstrumented piece.

The job fails itself if MSan isn't working

A sanitizer job that reports nothing is indistinguishable from one that is broken, and this series has already produced two such false greens (stale artifacts in #168, a -maxdepth that silently skipped two directories in #175).

So before trusting anything the real run says, the job plants an uninitialized read and requires MSan to catch it:

TEETH CHECK FAILED: MSan did not report a planted uninitialized read.
This job would be green and blind. Refusing to continue.

That encodes the -O0 finding from #172 as a gate rather than a comment, so a later flag change cannot quietly hollow it out.

Verification

workflow_dispatch only fires for workflows already on the default branch, so a temporary pull_request trigger is the only way to exercise this before merging. Merging it unverified would contradict the whole point of the teeth check.

Plan: let this run on the PR, confirm the libc++ build, teeth check and suites all pass, then strip the trigger and merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_019tVgXeXCcfbHqfMjhufWzd

[danmcleran]

Verified and ready. The temporary pull_request trigger has been removed; the workflow is back to schedule + workflow_dispatch only.

Four runs on this PR, which is exactly what the trigger existed for:

1. libc++ configure failed — libunwind missing from LLVM_ENABLE_RUNTIMES (libc++abi defaults to LIBCXXABI_USE_LLVM_UNWINDER=ON). Fixed by adding it to the runtimes list, which also instruments it — an uninstrumented unwinder in an MSan binary is another phantom-report source.
2. libc++ built, teeth check passed, unit_test/nn failed two numeric tolerances.
3. Tolerance branch never fired — it matched Boost's exit 201, but the suite runs through a sub-make and make reports its own failure as 2. Now detected from Boost's failures are detected in the log.
4. Green.

Final run output:

Teeth check passed: MSan reports the planted read.
note: unit_test/nn has Boost assertion failures but no MSan report (see libc++ RNG note)
check-msan-libcxx: numeric divergence under libc++ in: unit_test/nn
check-msan-libcxx: no uninitialized reads across the Boost suites

The result is a clean negative: with an instrumented libc++, MSan finds no uninitialized reads anywhere in the ten Boost suites. That is the coverage the #171 static-init bug slipped through, now watched nightly.

The unit_test/nn divergence is not a defect: std::uniform_real_distribution is not required to produce the same sequence across implementations, so swapping libstdc++ for libc++ changes the initial weights and moves trained results outside tolerances tuned on libstdc++. Tolerances were deliberately not loosened — that would corrupt a real gate to make an unrelated one green.

Merging this unverified would have left a nightly failing silently at 06:00 UTC until someone noticed a red scheduled run.