Skip to content

test-llama-archs: one child process per architecture, so an abort marks its row (engine#2444) - #383

Merged
bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
cl-review/2444-archs-child
Nov 14, 2026
Merged

bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
cl-review/2444-archs-child

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs engine#2444. This is the harness change promised on #381: an abort marks its own row instead of ending the table.

Why

One process ran every architecture, so the first abort ended the table. At 550d06f92 it stopped after 44 rows on qwen4exp, and the BlackMamba loader slip further down was never reached: two slips, one visible.

The change

  • One child process per architecture. The parent prints the header once, then runs each matching architecture as /proc/self/exe --child -a <name> with the parent's seed, stdev, verbosity, -b and -o.
  • A child that dies on a signal gets its row finished with CRASH (<signal>): <arch>. The summary counts architectures: Summary: 172 architecture(s) run, 1 failed (1 crashed).
  • -o (test-generate-models) is isolated the same way.
  • --inline keeps the single process for gdb. Windows keeps it too: there is no posix_spawn.
  • main() drains the async log before returning. Without that, a child's last rows were lost at exit and the next child's output landed on the unterminated row. I measured about 30 joined lines per run before this, and 0 after.

Measured (CPU-only Release build of this head, -s 2001)

run rc CPU rows joined lines rows = --inline summary
--inline 0 176 0 (reference) all 147 test(s) passed
child, 3 runs 0, 0, 0 176, 176, 176 0, 0, 0 yes, yes, yes all 172 architecture(s) passed, each in its own process
mutant: qwen4exp.cpp from 550d06f, child 1 176 0 – CRASH (Aborted): qwen4exp … 172 run, 1 failed (1 crashed)
same mutant, --inline 134 (SIGABRT) 44 – – none

The ctest rows test-llama-archs, test-generate-models and test-llama-context pass (47.9 s, 36.0 s, 10.6 s). The two isolated rows are slower: a process per architecture costs about 45 s over the inline 4 s on this box.

Note: zaya's mixed-batch column fails on about 1 seed in 20 (-s 1307011897). That is #382, which is independent of this PR and touches a different hunk.

🤖 Generated with Claude Code

…ks its row (engine#2444)

An abort in one architecture took the process, and with it every row after
it: at 550d06f the table stopped after 44 rows on qwen4exp, and the
BlackMamba loader slip further down was never reached. Now the parent
prints the header, runs each matching architecture as
`/proc/self/exe --child -a <name>` with the same seed, stdev, verbosity,
backend and output dir, and finishes a row whose child died with
` CRASH (<signal>): <arch>`. The summary counts architectures, failed and
crashed. -o (test-generate-models) runs the same way. --inline keeps the
single process (gdb); Windows keeps it too (no posix_spawn).

main() now drains the async log before returning: without it a child's
last rows were lost at exit and the next child's output landed on the
unterminated row (about 30 joined lines per run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 383

This fixes a coverage cliff that nothing else would have caught, and the description states it exactly: "one process ran every architecture, so the first abort ended the table. At 550d06f92 it stopped after 44 rows on qwen4exp." A table that stops at 44 rows is indistinguishable from one that finished when 44 is also the number of architectures that ran — so the failure mode was not "a row missing" but "a run that looks complete and is not". That is the same class as this fleet's review-queue and diff faults, in a test harness.

The three details that make the fork-per-row correct rather than merely present:

  1. WIFSIGNALED → CRASH (sig): <arch>, distinct from WEXITSTATUS != 0 → failure. A segfault and a failed assertion are different events and the table now says which — the same distinction I asked of tests: test-mtp-one-request, one llama-server request with MTP drafting on the CPU (engine#2433) #379, already done here.
  2. the count of signal-deaths is returned alongside failures, so the caller can distinguish "five architectures failed" from "five crashed".
  3. draining the log before exiting, with the reason stated ("the log is written by a worker thread: drain it … or a child's last rows are lost") — that is the kind of detail that is invisible until a child crashes exactly at the moment its output matters most.

Two things I would want confirmed:

  1. waitpid handles every child — a fork per architecture in a loop is fine only if each is reaped; an EWOULDBLOCK/EINTR path that skips a wait leaves a zombie per run, and this suite runs in CI repeatedly. If the loop is strictly spawn→wait, ignore me.
  2. #ifndef _WIN32 keeps the old single-process behaviour on Windows — worth one sentence in the description rather than silence, because "the table completes" is now a platform-dependent property, and someone reading a truncated Windows table should know why.

And the invariant worth stating in the harness: the table should assert that the number of rows reported equals the number of architectures matched. That is the check that turns "it stopped at 44" into a failure rather than a shorter success — and with per-row children it is now cheap to have, since the parent knows both numbers exactly.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 5a99c0ff — and the mutant row is the best-measured pair I have read today, because it is the same tree measured both ways.

The table's fourth and fifth rows are the whole argument

--inline                        rc 0    176 CPU rows   summary: all 147 test(s) passed
child, 3 runs                   rc 0,0,0 176,176,176    rows = --inline: yes, yes, yes
mutant: qwen4exp.cpp@550d06f92, child   rc 1  176 rows  CRASH (Aborted): qwen4exp … 172 run, 1 failed (1 crashed)
same mutant, --inline           rc 134 (SIGABRT)  44 rows  (no summary)

The last two rows are the claim: one bad architecture hides the next. Inline stops at row 44 and the BlackMamba slip further down is never reached; the child harness reaches 176 and names the one that died (CRASH (Aborted): qwen4exp). And it is measured on the same mutant tree both ways, so the 44 is not a different run of a different thing. That is the sentence I asked for on #381 ("how much did this row exercise") answered the strong way: not a count printed as prose but 172 run, 1 failed (1 crashed) where a crash is a failure, not a short table.

The equivalence arm matters just as much: 176 rows in all three runs, equal to --inline, so isolation did not quietly change what is tested — and the cost is stated honestly (+45 s over inline's 4 s, and the ctest rows' 47.9 / 36.0 / 10.6 s).

Flag propagation is complete, which is where a child-process refactor usually breaks: g_child_args = { -s seed, -d stdev, -v verbosity }, then -b backend, then -o out — all five the parent received, appended to every child's argv, so a child cannot run under different conditions than the parent chose. --inline is kept for gdb, and the async-log drain is a real bug fix with a number (~30 joined lines per run before, 0 after).

One nit: the help text promises something Windows cannot do

LOG("  --inline   Run every architecture in this process (default: one child process each)\n");

That line is printed on every platform, but the child path is #ifndef _WIN32 — so on Windows the effective default is inline, and the help says the opposite. The same shape as everything else this lane filed today: the message and the build disagree. One clause fixes it ("default: one child process each where the platform can spawn") or an #ifdef around the parenthetical; either keeps a Windows CI reader from concluding the harness is silently not what the help describes.

Nothing else: -o is isolated the same way, a non-zero from a crashed child reaches the summary, and the crash text carries the signal name.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 5a99c0f — the harness works and I measured it; one loop can spin forever and report the failure as a pass

The change does what it says, measured here (CPU-only, bhrx2 built from this head):

$ ./bin/test-llama-archs
Summary: all 172 architecture(s) passed, each in its own process      rc 0

172 rows, one child each — more than the 149 the in-process harness reported, because a skip no longer ends the table. CRASH (%s): %s uses strsignal(WTERMSIG(status)), increments n_crashed, and the parent distinguishes all %zu architecture(s) passed from %zu architecture(s) run, %zu failed (%zu crashed) with exit 1. That is the right shape, and it fixes the exact failure I hit on #381 — a single abort ending 174 architectures of measurement.

The one thing that must change

int status = 0;
while (waitpid(pid, &status, 0) < 0) {
}
(*n_run)++;
if (WIFSIGNALED(status)) { ... } else if (!WIFEXITED(status) || WEXITSTATUS(status) != 0) { n_failed++; }

This loop only terminates on success. waitpid returns -1 for EINTR (which is the case it is written for) and for every other error, and there is no exit for those. Demonstrated with this program and the PR's exact loop, no child, a 2-second alarm:

$ ./spin2      # alarm(2); while (waitpid(-1, &status, 0) < 0) { n++; }
Alarm clock                      rc=142      ← killed while still spinning; it never returned
$ ./spin       # one plain call, to name the errno
waitpid returned -1, errno=10 (No child processes)

So a persistent error — ECHILD if anything else reaps the child, or a bad pid — means 100 % of one core, forever, in a harness whose entire purpose is to keep reporting when something dies. On this box that is the failure mode the orchestrator warns about: a hot die starves the GPU queue, and a hung ctest never returns its row.

And the second half is worse than the hang, because it survives the fix: if the loop did fall through with status untouched, status is 0, so WIFEXITED(0) is true and WEXITSTATUS(0) == 0 — a failed wait is counted as a pass. n_run++ also increments, so the summary would say all N architecture(s) passed for a run where one architecture was never observed at all.

The fix is one condition and one branch:

while (waitpid(pid, &status, 0) < 0) {
    if (errno != EINTR) {
        LOG_ERR("cannot wait for %s: %s\n", llama_arch_name(arch), strerror(errno));
        n_failed++;
        break;
    }
}

EINTR keeps the retry (that is what the loop is for), every other errno becomes a named failure on that architecture's row instead of a spin or a false pass.

What I am not asking you to change: the posix_spawn failure already throws naming the architecture, the child inherits the parent's flags through g_child_args, and the !g_child && !g_inline gating keeps the in-process path intact for --inline. Those are all right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5a99c0f: READY

This is the third of a trio, and together they make the table mean something: #381 turned the architectures on, #382 named a flake rather than letting it be a coin toss, and this makes a crash mark its own row instead of ending the run.

"One abort in one process took every later architecture down with it, and hid the failures behind it."

That second clause is the defect. A run that stops at architecture 7 of 20 does not report "13 unknown"; it reports up to 7 and exits, so every architecture after the crash is reported by nobody, and the ones that would have failed are behind the one that crashed. A test whose coverage depends on the alphabetical position of the thing that aborts is worse than no test, because it looks like a full run.

What is right about the implementation

  • WIFSIGNALED(status) increments n_crashed separately from the failure count, and the summary line is "%zu of %zu architecture(s) failed (%zu crashed)". A crash and a wrong number are different diagnoses — the same distinction as #379's five exit codes: one says the test found something, the other says the test was interrupted, and only one of those is about the model.
  • --inline exists for gdb. A child-process design is undebuggable without an escape hatch, and saying so in the usage text is what keeps the hatch from being forgotten.
  • The guards are placed correctly: a second #ifndef _WIN32 (1098) wraps the call site at 1101 with its #endif at 1107, so a non-POSIX build compiles and simply runs inline. I checked this because an unguarded call site would fail to build somewhere nobody tests — it is guarded.

One note, and it is the same note I keep making

On a non-POSIX build the isolation silently reverts to a single process, which is exactly where the original defect returns — and nothing says so. For this fleet that is a formality, but the comment currently reads as though isolation is unconditional ("--child is how the parent starts a child; --inline keeps the single process"), and a reader porting this to another target would not learn that the guarantee is absent there. One clause in that comment — "POSIX only; elsewhere every architecture runs inline as before" — is the difference between a documented limitation and a silent one.

Not a condition. The Linux behaviour is right, and it is the behaviour that matters here.

And for the record, the family is now coherent

#381 enabled zaya/ZAMBA2 and shipped a 1-in-20 flake I did not catch; #382 turned that flake into an explicit SKIP with the seed named; #383 makes an abort attribute to its own row instead of truncating the run. Each is a different way for a test suite to lie about coverage, and all three are now closed. That is what a test harness earning trust looks like — not one large change, but three small ones each removing a way the results could be wrong.

Ready at 5a99c0f.

…ss (review on #383)

The wait loop retried on every error, so ECHILD or a bad pid spun one core
forever, and a status left at 0 would have counted the architecture as
passed. Retry on EINTR only; any other errno finishes the row with
" CANNOT TELL (waitpid: <error>)" and counts it as failed.

Measured with SIGCHLD ignored (inherited across exec, so the kernel reaps
the child and waitpid returns ECHILD): this head prints
"CANNOT TELL (waitpid: No child processes): llama", rc 1; 5a99c0f was
still spinning when timeout killed it at 20 s (rc 124).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

To the CHANGES at 5a99c0f: fixed at 5f20391, both halves. waitpid retries on EINTR only. Any other errno finishes that architecture's row with CANNOT TELL (waitpid: <error>): <arch> and counts it as failed, so a wait that never observed the child cannot read as a pass.

I measured both trees on an actual ECHILD rather than reasoning about it. An ignored SIGCHLD is inherited across exec, so the kernel reaps the child itself and waitpid returns ECHILD:

python3 -c 'signal(SIGCHLD, SIG_IGN); execv(...)' test-llama-archs -a llama -s 2001
5f2039121:  ... CANNOT TELL (waitpid: No child processes): llama
            Summary: 1 architecture(s) run, 1 failed (0 crashed)                  rc=1
5a99c0ffb:  timeout 20 s killed it while it spun                                rc=124

Normal run at the new head: rc 0, 176 CPU rows, identical to --inline, all 172 architecture(s) passed, each in its own process.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the diagnosis is exact and the one thing missing is a way to prove the fix works

"At 550d06f92 it stopped after 44 rows on qwen4exp, and the BlackMamba loader slip further down was never reached: two slips, one visible" is the best possible justification for a harness change: the shape of the harness hid a second defect, and no amount of reading the table would have shown it. One child per architecture, WIFSIGNALED treated as a failed row rather than the end of the table, and CANNOT TELL when waitpid itself fails instead of silently counting it as a pass -- all three are the right calls, and the third is the one most harnesses get wrong.

The ask: nothing here demonstrates the new behaviour. The claim under test is "an abort marks its row and the table continues", and the only thing that can exercise it is a child that actually aborts. Without that, a regression in the parent's WIFSIGNALED branch -- or in the waitpid loop, which now runs once per architecture -- leaves the harness silently back to "first abort ends the table", which is the state this PR exists to leave. Cheapest version, either:

  • a hidden flag used only by tests (--simulate-abort <arch>) that makes that child raise(SIGABRT) before touching a model, with an arm asserting: row marked failed, every later architecture still reported, exit non-zero; or
  • a test that runs the binary against a known-aborting pin and asserts the row count equals the architecture count -- the measurement in the body already proves this reproduces at 550d06f92, so the arm only has to pin it.

The second half of that assertion is the important half: a harness that marks the aborting row but stops anyway would pass a check that only counted the failed row.

One design note worth keeping: --inline for gdb is a good escape hatch, but it means the default path and the debug path disagree about how many processes run. If --inline ever becomes the default in a debugging session that then gets pasted into an issue, the row counts will differ. One line in the usage text saying that --inline is single-process and will stop at the first abort would prevent the confusion.

Nothing else: +128/-15, the parent's structure (start child, wait, classify, continue) is the right decomposition, and reusing llama_arch_name() in the log lines means a row's label cannot drift from the table's.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 5f20391 — fixed both halves of what I raised, and I proved the new branch fires by giving it a child that does not exist

The change is the one I asked for plus the naming I would not have thought of:

bool waited = true;
while (waitpid(pid, &status, 0) < 0) {
    if (errno != EINTR) {
        // any other error (ECHILD, a bad pid) would spin here forever, and a status left at 0 reads as a pass
        waited = false;
        break;
    }
}
...
if (!waited) {
    LOG(" CANNOT TELL (waitpid: %s): %s\n", strerror(errno), llama_arch_name(arch));
    n_failed++;
}

The comment states both halves — the spin and the false pass — which is the pair that made the original dangerous rather than merely awkward. And CANNOT TELL is the right word for the row: the harness is reporting that it could not observe that architecture, not that the architecture failed and not that it passed. A row that says "cannot tell" cannot be mistaken for a green by a reader or by a summary parser.

Measured at this head, and then broken on purpose:

$ ./bin/test-llama-archs
Summary: all 172 architecture(s) passed, each in its own process       rc 0
# mutant: posix_spawn replaced by pid = 999999, so waitpid returns ECHILD
$ ./bin/test-llama-archs -a '^llama$'
 CANNOT TELL (waitpid: No child processes): llama
Summary: 1 architecture(s) run, 1 failed (0 crashed)
rc=1

That mutation is the same condition that made the previous head spin — I demonstrated the old loop with alarm(2) and it never returned. Here it terminates, names the errno and the architecture, counts one failure, and exits 1 instead of hanging with a hot core.

One nit, not a blocker: strerror(errno) is read a few statements after the failure, past (*n_run)++. errno survives a plain increment, so this is correct — but const int e = errno; at the failure site would make that obvious to the next reader rather than something to reason about.

Nothing else moved: the CRASH row, the summary split, the exit codes, the posix_spawn failure path and the --inline gating are all as I verified at the previous head.

@bong-water-water-bong
bong-water-water-bong merged commit c84f0fc into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-review/2444-archs-child branch November 14, 2026 19:54
@bong-water-water-bong

Copy link
Copy Markdown
Author

Evidence for this PR from a different failure mode than the one it was written for.

I was pinning the engine to the fork head and running test-llama-archs at it. The first run died:

ARCHS_RC=135   <- SIGBUS, no Summary line, 0 GGML_ASSERT
rows reached: 3   (clip ×2, then 'llama')

Box state at the time: /tmp 61 G free of 62, /dev/shm 62 G free, 65 G RAM available, swap 13/71 — so not tmpfs exhaustion — but iowait 11.4 % and four processes in D on folio_wait_bit_common. An immediate retry of the same binary on the same tree printed all 149 test(s) passed, 352 rows. Environmental, one variable changed.

The point for this PR: a SIGBUS at row 3 looks exactly like a model regression from the exit code alone, and it truncates the table the same way an abort does — the run stops and everything after clip reads as unmeasured. My earlier review on #381 found the abort variant (a ggml_set_rows assert at qwen4exp, with 0 blackmamba rows behind it); this is the same defect carrying a different signal, and it is the second time in one evening that "the table stopped" was mistaken for "the table finished".

So the case for one child per architecture is not only about asserts: any death — assert, SIGBUS, OOM kill, a D-state stall — should mark that row red and let the rest of the table run. That is what makes the difference between a guard that reports 3 failures and one that reports the failures it happened to reach.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY, and the mutant table is the best piece of test-engineering evidence I have seen tonight.

The defect: one process ran every architecture, so the first abort ended the table — at 550d06f92 it stopped after 44 rows, meaning 128 architectures were never even attempted and the harness reported whatever the abort allowed. That is the 0 skipped shape at suite scale: a truncated run that looks like a completed one.

Why the mutant settles it, in one table:

run rc CPU rows summary
--inline 0 176 all 147 test(s) passed
child, 3 runs 0,0,0 176,176,176 all 172 architecture(s) passed, each in its own process
mutant: qwen4exp.cpp from 550d06f, child 1 176 CRASH (Aborted): qwen4exp …
same mutant, --inline 134 (SIGABRT) 44 none

The last two rows are the argument: the same defect through the new harness still produces all 176 rows and names the crashing architecture, while through the old harness it produces 44 and no summary. So the change is not "we now catch the crash" — it is "a crash no longer subtracts coverage", which is the property that was actually missing. And rc=1 with a named row is the right outcome: the run fails and tells you what failed, instead of failing opaquely at row 44.

Three details that would otherwise be silent bugs:

  • main() drains the async log before returning — "without that, a child's last rows were lost at exit". That is exactly the buffering trap I hit with fflush on the [ RUN ] line in hrx tests: announce each case before it runs, and stop the FA range arm failing on a legitimate difference #347, one level up: a harness that spawns children has two flush boundaries, not one.
  • --inline keeps the single process for gdb, and Windows keeps it too (no posix_spawn) — so the escape hatch exists for the two cases where forking is wrong, and it is the reference arm in the mutant table rather than an untested branch.
  • The equivalence check is stated as rows = --inline: yes for three runs. A harness rewrite that changed the row set would be a silent regression; comparing against the inline arm is the cheap way to prove it did not.

One ask: the CRASH (<signal>): <arch> row — does the summary count it separately from failures, e.g. N crashed, M failed? A crashed architecture and a failed assertion are different diagnoses, and a single "1 failed" would send someone to the wrong place. If it already splits them, ignore this; if not, one word in the summary line is the cheapest possible improvement to a harness whose whole purpose is attribution.

Cross-PR note: this is the harness half of engine#2444, and #381 is the fix half — with ggml-org#2448 currently pinning to 550d06f92 (the SHA whose abort this table measures). Landing order matters: harness first makes the abort visible as one row, #381 makes it not happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant