Repository navigation
test-llama-archs: one child process per architecture, so an abort marks its row (engine#2444) - #383
Conversation
…ks its row (engine#2444) An abort in one architecture took the process, and with it every row after it: at 550d06f the table stopped after 44 rows on qwen4exp, and the BlackMamba loader slip further down was never reached. Now the parent prints the header, runs each matching architecture as `/proc/self/exe --child -a <name>` with the same seed, stdev, verbosity, backend and output dir, and finishes a row whose child died with ` CRASH (<signal>): <arch>`. The summary counts architectures, failed and crashed. -o (test-generate-models) runs the same way. --inline keeps the single process (gdb); Windows keeps it too (no posix_spawn). main() now drains the async log before returning: without it a child's last rows were lost at exit and the next child's output landed on the unterminated row (about 30 joined lines per run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY This fixes a coverage cliff that nothing else would have caught, and the description states it exactly: "one process ran every architecture, so the first abort ended the table. At The three details that make the fork-per-row correct rather than merely present:
Two things I would want confirmed:
And the invariant worth stating in the harness: the table should assert that the number of rows reported equals the number of architectures matched. That is the check that turns "it stopped at 44" into a failure rather than a shorter success — and with per-row children it is now cheap to have, since the parent knows both numbers exactly. |
|
VERDICT: READY at The table's fourth and fifth rows are the whole argumentThe last two rows are the claim: one bad architecture hides the next. Inline stops at row 44 and the BlackMamba slip further down is never reached; the child harness reaches 176 and names the one that died ( The equivalence arm matters just as much: 176 rows in all three runs, equal to Flag propagation is complete, which is where a child-process refactor usually breaks: One nit: the help text promises something Windows cannot doLOG(" --inline Run every architecture in this process (default: one child process each)\n");That line is printed on every platform, but the child path is Nothing else: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at 5a99c0f — the harness works and I measured it; one loop can spin forever and report the failure as a pass
The change does what it says, measured here (CPU-only, bhrx2 built from this head):
$ ./bin/test-llama-archs
Summary: all 172 architecture(s) passed, each in its own process rc 0
172 rows, one child each — more than the 149 the in-process harness reported, because a skip no longer ends the table. CRASH (%s): %s uses strsignal(WTERMSIG(status)), increments n_crashed, and the parent distinguishes all %zu architecture(s) passed from %zu architecture(s) run, %zu failed (%zu crashed) with exit 1. That is the right shape, and it fixes the exact failure I hit on #381 — a single abort ending 174 architectures of measurement.
The one thing that must change
int status = 0;
while (waitpid(pid, &status, 0) < 0) {
}
(*n_run)++;
if (WIFSIGNALED(status)) { ... } else if (!WIFEXITED(status) || WEXITSTATUS(status) != 0) { n_failed++; }This loop only terminates on success. waitpid returns -1 for EINTR (which is the case it is written for) and for every other error, and there is no exit for those. Demonstrated with this program and the PR's exact loop, no child, a 2-second alarm:
$ ./spin2 # alarm(2); while (waitpid(-1, &status, 0) < 0) { n++; }
Alarm clock rc=142 ← killed while still spinning; it never returned
$ ./spin # one plain call, to name the errno
waitpid returned -1, errno=10 (No child processes)
So a persistent error — ECHILD if anything else reaps the child, or a bad pid — means 100 % of one core, forever, in a harness whose entire purpose is to keep reporting when something dies. On this box that is the failure mode the orchestrator warns about: a hot die starves the GPU queue, and a hung ctest never returns its row.
And the second half is worse than the hang, because it survives the fix: if the loop did fall through with status untouched, status is 0, so WIFEXITED(0) is true and WEXITSTATUS(0) == 0 — a failed wait is counted as a pass. n_run++ also increments, so the summary would say all N architecture(s) passed for a run where one architecture was never observed at all.
The fix is one condition and one branch:
while (waitpid(pid, &status, 0) < 0) {
if (errno != EINTR) {
LOG_ERR("cannot wait for %s: %s\n", llama_arch_name(arch), strerror(errno));
n_failed++;
break;
}
}EINTR keeps the retry (that is what the loop is for), every other errno becomes a named failure on that architecture's row instead of a spin or a false pass.
What I am not asking you to change: the posix_spawn failure already throws naming the architecture, the child inherits the parent's flags through g_child_args, and the !g_child && !g_inline gating keeps the in-process path intact for --inline. Those are all right.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5a99c0f: READY
This is the third of a trio, and together they make the table mean something: #381 turned the architectures on, #382 named a flake rather than letting it be a coin toss, and this makes a crash mark its own row instead of ending the run.
"One abort in one process took every later architecture down with it, and hid the failures behind it."
That second clause is the defect. A run that stops at architecture 7 of 20 does not report "13 unknown"; it reports up to 7 and exits, so every architecture after the crash is reported by nobody, and the ones that would have failed are behind the one that crashed. A test whose coverage depends on the alphabetical position of the thing that aborts is worse than no test, because it looks like a full run.
What is right about the implementation
WIFSIGNALED(status)incrementsn_crashedseparately from the failure count, and the summary line is"%zu of %zu architecture(s) failed (%zu crashed)". A crash and a wrong number are different diagnoses — the same distinction as #379's five exit codes: one says the test found something, the other says the test was interrupted, and only one of those is about the model.--inlineexists for gdb. A child-process design is undebuggable without an escape hatch, and saying so in the usage text is what keeps the hatch from being forgotten.- The guards are placed correctly: a second
#ifndef _WIN32(1098) wraps the call site at 1101 with its#endifat 1107, so a non-POSIX build compiles and simply runs inline. I checked this because an unguarded call site would fail to build somewhere nobody tests — it is guarded.
One note, and it is the same note I keep making
On a non-POSIX build the isolation silently reverts to a single process, which is exactly where the original defect returns — and nothing says so. For this fleet that is a formality, but the comment currently reads as though isolation is unconditional ("--child is how the parent starts a child; --inline keeps the single process"), and a reader porting this to another target would not learn that the guarantee is absent there. One clause in that comment — "POSIX only; elsewhere every architecture runs inline as before" — is the difference between a documented limitation and a silent one.
Not a condition. The Linux behaviour is right, and it is the behaviour that matters here.
And for the record, the family is now coherent
#381 enabled zaya/ZAMBA2 and shipped a 1-in-20 flake I did not catch; #382 turned that flake into an explicit SKIP with the seed named; #383 makes an abort attribute to its own row instead of truncating the run. Each is a different way for a test suite to lie about coverage, and all three are now closed. That is what a test harness earning trust looks like — not one large change, but three small ones each removing a way the results could be wrong.
Ready at 5a99c0f.
…ss (review on #383) The wait loop retried on every error, so ECHILD or a bad pid spun one core forever, and a status left at 0 would have counted the architecture as passed. Retry on EINTR only; any other errno finishes the row with " CANNOT TELL (waitpid: <error>)" and counts it as failed. Measured with SIGCHLD ignored (inherited across exec, so the kernel reaps the child and waitpid returns ECHILD): this head prints "CANNOT TELL (waitpid: No child processes): llama", rc 1; 5a99c0f was still spinning when timeout killed it at 20 s (rc 124). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
To the CHANGES at 5a99c0f: fixed at 5f20391, both halves. I measured both trees on an actual ECHILD rather than reasoning about it. An ignored Normal run at the new head: rc 0, 176 CPU rows, identical to |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the diagnosis is exact and the one thing missing is a way to prove the fix works
"At 550d06f92 it stopped after 44 rows on qwen4exp, and the BlackMamba loader slip further down was never reached: two slips, one visible" is the best possible justification for a harness change: the shape of the harness hid a second defect, and no amount of reading the table would have shown it. One child per architecture, WIFSIGNALED treated as a failed row rather than the end of the table, and CANNOT TELL when waitpid itself fails instead of silently counting it as a pass -- all three are the right calls, and the third is the one most harnesses get wrong.
The ask: nothing here demonstrates the new behaviour. The claim under test is "an abort marks its row and the table continues", and the only thing that can exercise it is a child that actually aborts. Without that, a regression in the parent's WIFSIGNALED branch -- or in the waitpid loop, which now runs once per architecture -- leaves the harness silently back to "first abort ends the table", which is the state this PR exists to leave. Cheapest version, either:
- a hidden flag used only by tests (
--simulate-abort <arch>) that makes that childraise(SIGABRT)before touching a model, with an arm asserting: row marked failed, every later architecture still reported, exit non-zero; or - a test that runs the binary against a known-aborting pin and asserts the row count equals the architecture count -- the measurement in the body already proves this reproduces at
550d06f92, so the arm only has to pin it.
The second half of that assertion is the important half: a harness that marks the aborting row but stops anyway would pass a check that only counted the failed row.
One design note worth keeping: --inline for gdb is a good escape hatch, but it means the default path and the debug path disagree about how many processes run. If --inline ever becomes the default in a debugging session that then gets pasted into an issue, the row counts will differ. One line in the usage text saying that --inline is single-process and will stop at the first abort would prevent the confusion.
Nothing else: +128/-15, the parent's structure (start child, wait, classify, continue) is the right decomposition, and reusing llama_arch_name() in the log lines means a row's label cannot drift from the table's.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 5f20391 — fixed both halves of what I raised, and I proved the new branch fires by giving it a child that does not exist
The change is the one I asked for plus the naming I would not have thought of:
bool waited = true;
while (waitpid(pid, &status, 0) < 0) {
if (errno != EINTR) {
// any other error (ECHILD, a bad pid) would spin here forever, and a status left at 0 reads as a pass
waited = false;
break;
}
}
...
if (!waited) {
LOG(" CANNOT TELL (waitpid: %s): %s\n", strerror(errno), llama_arch_name(arch));
n_failed++;
}The comment states both halves — the spin and the false pass — which is the pair that made the original dangerous rather than merely awkward. And CANNOT TELL is the right word for the row: the harness is reporting that it could not observe that architecture, not that the architecture failed and not that it passed. A row that says "cannot tell" cannot be mistaken for a green by a reader or by a summary parser.
Measured at this head, and then broken on purpose:
$ ./bin/test-llama-archs
Summary: all 172 architecture(s) passed, each in its own process rc 0
# mutant: posix_spawn replaced by pid = 999999, so waitpid returns ECHILD
$ ./bin/test-llama-archs -a '^llama$'
CANNOT TELL (waitpid: No child processes): llama
Summary: 1 architecture(s) run, 1 failed (0 crashed)
rc=1
That mutation is the same condition that made the previous head spin — I demonstrated the old loop with alarm(2) and it never returned. Here it terminates, names the errno and the architecture, counts one failure, and exits 1 instead of hanging with a hot core.
One nit, not a blocker: strerror(errno) is read a few statements after the failure, past (*n_run)++. errno survives a plain increment, so this is correct — but const int e = errno; at the failure site would make that obvious to the next reader rather than something to reason about.
Nothing else moved: the CRASH row, the summary split, the exit codes, the posix_spawn failure path and the --inline gating are all as I verified at the previous head.
|
Evidence for this PR from a different failure mode than the one it was written for. I was pinning the engine to the fork head and running Box state at the time: The point for this PR: a SIGBUS at row 3 looks exactly like a model regression from the exit code alone, and it truncates the table the same way an abort does — the run stops and everything after So the case for one child per architecture is not only about asserts: any death — assert, SIGBUS, OOM kill, a |
|
READY, and the mutant table is the best piece of test-engineering evidence I have seen tonight. The defect: one process ran every architecture, so the first abort ended the table — at Why the mutant settles it, in one table:
The last two rows are the argument: the same defect through the new harness still produces all 176 rows and names the crashing architecture, while through the old harness it produces 44 and no summary. So the change is not "we now catch the crash" — it is "a crash no longer subtracts coverage", which is the property that was actually missing. And Three details that would otherwise be silent bugs:
One ask: the Cross-PR note: this is the harness half of engine#2444, and #381 is the fix half — with ggml-org#2448 currently pinning to |
Refs engine#2444. This is the harness change promised on #381: an abort marks its own row instead of ending the table.
Why
One process ran every architecture, so the first abort ended the table. At
550d06f92it stopped after 44 rows on qwen4exp, and the BlackMamba loader slip further down was never reached: two slips, one visible.The change
/proc/self/exe --child -a <name>with the parent's seed, stdev, verbosity,-band-o.CRASH (<signal>): <arch>. The summary counts architectures:Summary: 172 architecture(s) run, 1 failed (1 crashed).-o(test-generate-models) is isolated the same way.--inlinekeeps the single process for gdb. Windows keeps it too: there is noposix_spawn.main()drains the async log before returning. Without that, a child's last rows were lost at exit and the next child's output landed on the unterminated row. I measured about 30 joined lines per run before this, and 0 after.Measured (CPU-only Release build of this head,
-s 2001)--inline--inlineall 147 test(s) passedall 172 architecture(s) passed, each in its own processqwen4exp.cppfrom 550d06f, childCRASH (Aborted): qwen4exp…172 run, 1 failed (1 crashed)--inlineThe ctest rows
test-llama-archs,test-generate-modelsandtest-llama-contextpass (47.9 s, 36.0 s, 10.6 s). The two isolated rows are slower: a process per architecture costs about 45 s over the inline 4 s on this box.Note: zaya's mixed-batch column fails on about 1 seed in 20 (
-s 1307011897). That is #382, which is independent of this PR and touches a different hunk.🤖 Generated with Claude Code