How to run the port checks and the regression suites is undocumented, and the traps are the expensive part #685

Building the ports and running the regression suites has a set of traps that are not discoverable from the repository, and every one of them is a bug that built cleanly or a measurement that came out wrong while looking right. They are currently written down in a private working file. That is the wrong place: it helps one person, and it dies with the file.

CONTRIBUTING.md is 72 lines and mentions building once. AGENTS.md covers coding style, headers, MISRA and git rules well, and says nothing about running any of this.

The traps that need a home

Running the suites

• Never run scripts/test_tx.sh and scripts/test_smp.sh concurrently. Since Fixed the coverage report's paths and scoping #664 both root gcovr at the repository root, and gcov writes its intermediate .gcov files into the directory gcovr is rooted at. Two concurrent runs delete each other's output. Measured: two gcovr rooted at the repo root, concurrent, exit 64 with a SanityCheckError naming a .gcov file that "doesn't exist but no error from GCOV detected" — which reads like a broken toolchain and is not. The damage is a green suite with most of its coverage silently missing.
• Never type run.sh test all by hand. With no CTEST_PARALLEL_LEVEL it takes the parallel branch: five configurations at once, five ctest workers each, 25 workers on 16 cores. Several SMP tests create 1024 ThreadX threads by construction and the Linux SMP port backs each with a pthread, so that is on the order of 5000 threads. Four of twelve runs hung outright. CI is unaffected only because test_tx.sh and test_smp.sh set CTEST_PARALLEL_LEVEL=1 — a coincidence of one caller, not a property of the script.
• A hang leaks the test process, and it outlives the run. One was found alive 45 minutes after the run.sh that spawned it had been killed: 1030 threads, 1029 parked on futexes, 1% CPU — deadlocked, not slow. It silently loads every measurement taken afterwards. Reap between runs, and identify leftovers by resolving /proc/PID/exe under the build directory — never by a command-line pattern, which also matches the script doing the reaping.
• The SMP suite is flaky by itself. threadx_smp_time_slice_test failed ERROR #31 once in three uninstrumented runs, and survived --repeat until-pass:2. A red SMP run is not evidence about the change under test until it has been repeated.

Measuring coverage

• The numerator is not reproducible run to run. Same tree, same compiler, every .gcda deleted between runs, all tests passing each time: 3826, then 3827, then 3827. common/src/tx_thread_system_resume.c:529 is reached by timing rather than by construction. No conclusion of the form "this change moved coverage by one line" is sound from a single pair of runs.
• The line denominator is stable; the branch denominator is not. lines-valid agrees exactly between this machine and the runner. branches-valid differs — 3548 against 3560 on the same tree with the same gcovr. See Branch coverage is around 78% in both suites and is enforced by nothing, because fail_below_min compares the line rate only #684.
• An empty coverage report reads as 100%. gcovr warns and exits 0, writing line-rate="1.0" beside lines-valid="0", and every downstream consumer reads "no data at all" as "fully covered". Both coverage.sh scripts assert the report has content for this reason; do not remove those checks.

Building the ports

• The pinned toolchain is Arm GNU 14.3.rel1, and the arm-none-eabi-gcc a distro package installs is usually 13.2. scripts/check_gcc.sh prints both resolved versions for exactly this reason, and requires both drivers so that a partial setup errors instead of quietly assembling half the tree with the wrong compiler. Check the printed versions.
• The generated ports are generated. Anything under ports/cortex_a*/, ports_smp/cortex_a*/ and the Cortex-M copies comes from ports_arch/ via update.sh. Fix the source, re-run the generator, commit both. ports_arch_check will catch you, and that is the check working.
• Do not prefix-match core names. cortex_a5* also matches cortex_a53 and cortex_a55, which are AArch64; assembling those as ARM32 produces a flood of misleading errors.
• A .s file under a gnu tree is a bug. GAS runs the C preprocessor on .S and not on .s, so in a .s file a #define is never substituted and an #if/#else emits both arms. Twenty-nine files were in that state and only three failed to assemble (Assembled the module ports, which no check had ever compiled #672). check_ports.sh enforces this for gnu trees only — IAR, Arm Compiler 5 and Keil preprocess .s themselves and about three hundred files rely on that, so a tree-wide version of the rule would be wrong rather than merely noisy.

Working with the workflows

• A workflow that does not trigger on dev does not exist. dev is the integration branch. This has bitten the repository three times: ports_arch_check (eight months of drift), ci_cortex_m (three months failing in seven seconds), regression_test.
• A trigger change does not gate the pull requests already open, and close/reopen does not fix it — only a rebase. See Three open pull requests to dev are gated by no regression suite, because their branches predate the trigger #679.
• A fork pull request's workflow runs are held for maintainer approval, show as action_required, and the hold returns on every push. An empty checks list has these two quite different causes; Three open pull requests to dev are gated by no regression suite, because their branches predate the trigger #679 explains how to tell them apart.
• Merging a pull request that touches .github/workflows/ needs a token with the workflow scope. A token with repo is refused at the merge step — after the branch has pushed fine, because git push over SSH never sees the OAuth token. gh auth refresh -h github.com -s workflow, or merge from the web UI.
• Do not pipe a long CI-shaped run through tail while iterating: it buffers until the pipeline closes and a finished run looks like a hang. Redirect to a file and read the file.

Suggested shape

A docs/ci.md — or a substantial section in CONTRIBUTING.md — covering how to run the checks and the suites locally, what the pinned toolchains are, and the traps above. Written for a contributor, not for us.

Worth doing before #684, which is the issue most likely to be quietly sabotaged by the leaked-process trap.