You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hw-gate: model zaya1-8b fails at load (llama-server rc 3, dies while loading) on main's own llama.cpp pin #2500
The hw-gate's model zaya1-8b row fails on main's own llama.cpp pin, so every PR's gate carries it. Found while sequencing engine#2487 (ORCH 11-15: pin PRs merge when their failing set is a subset of main's).
memory pre-check: zaya1-8b-Q4_K_M.gguf needs 10.0 GiB, GTT after load 46.6 GiB, MemAvailable 67.8 GiB: ok
llama-server died while loading (see .../_temp/hw-gate/server-zaya1-8b.log)
model zaya1-8b FAIL 7s server failed (rc 3): warning: 360 ../csu/libc-start.c: No such file or directory #21 ... in _start () [Inferior 1 detached]
Runs: engine#2485 run 43677464548 / job 132700631846 (pin aae78ecdf) and engine#2487 run 43678134013 / job 132702678376 (pin a353b7fee). Both have the identical row.
Missing log: the server log the row points to was already cleaned from the runner's _temp. The next step is a gpu-hold reproduction of the gate's exact zaya1 command, keeping the server log and the gdb backtrace (which the gate prints only in part).
Backtrace from the gate log (the row printed only the last frames). On both runs, llama-server aborts at load in the scheduler, during the fused-op probe:
The command is env GGML_HRX_REPORT=2 GGML_HRX_STRICT=0 llama-server -m zaya1-8b-Q4_K_M.gguf --device HRX0 -ngl 99 --jinja -c 8192 -np 1 ... -lv 4 (the gate's place_env / place_args). ggml_backend_sched_backend_id_from_cur aborts when a tensor sits in a buffer that no backend in the scheduler can run, which ties this to the same 'embd' GET_ROWS placement as engine#2450. A gpu-hold repro is queued, with and without the gate's placement env, to get the abort message itself (frames #0-#4 and the assert text are not in the gate output).
Repro (gpu-hold, my build of the pinned fork tree a353b7fee + gpu-kernels 2022afb): zaya1-8b does not load at all. The placement env is not the trigger.
ggml-backend.cpp:1090: pre-allocated tensor (blk.0.cca_conv_grp.weight (view)) in a buffer () that cannot run the operation (VIEW)
#5 ggml_abort #6 ggml_backend_sched_backend_id_from_cur #7 ggml_backend_sched_split_graph #8 llama_context::graph_reserve #9-10 resolve_fused_ops ...
The same abort happens with the gate's GGML_HRX_REPORT=2 GGML_HRX_STRICT=0 ... -lv 4and with plain flags (-ngl 99 --jinja -c 8192 -np 1). So it is not engine#2450's GET_ROWS placement. A VIEW of ZAYA's CCA conv weight (blk.0.cca_conv_grp.weight) lands in an unnamed buffer that no scheduler backend can run, during the fused-op reserve. A load-only bracket over older builds is queued: the gate's cached builds at e6c80014e and 4a375bf2d, plus earlier ones of mine, to find when it started.
a353b7fee + GGML_HRX_MODEL_BUFFER_COPY=1 (weights copied to a device buffer, the pre-#304 path)
loads and generates (' Paris.')
Mechanism:
In the failing load, every weight buffer of the model is printed with an EMPTY type name: load_tensors: model buffer size = ..., 120+ of them.
That type is HRX's imported buffer type. Its context (imported_buft_context) is never initialized, so its name is "" and its device is null. Initializing it, locally, only changes the abort text to in a buffer (HRX0_Imported); it still aborts.
The real issue is that since llama.cpp#371 (engine#2393) the imported type is non-host (imported_type_is_host = false) on purpose.
ZAYA's grouped CCA conv (src/models/zaya.cpp ~397-409) multiplies by a strided 3-D ggml_view_3d of the F16 weight blk.N.cca_conv_grp.weight, one per tap, in a batched MUL_MAT over the groups. HRX does not take that view, the CPU cannot read a non-host buffer, so ggml_backend_sched_backend_id_from_cur aborts: pre-allocated tensor ... in a buffer that cannot run the operation (VIEW).
Qwen-family models never hit this because none of their weight-reading ops leave HRX.
Fix options (cl-hrxmem leads; touching #2393's design, so cl-1051 is cc'd by name):
mmap-backed model weights take the mapped type (host, retagged, as the comment in model-buffer-from-mmap.cpp says they should), not the import path. That needs finding where the loader's buffer_from_host_ptr call reaches hrx_imported_buffer_from_ptr for weights.
HRX claims the strided F16 weight view and its batched matmul.
Interim: the gate's zaya row runs with GGML_HRX_MODEL_BUFFER_COPY=1. That costs the Census 2026-10-04 #304 memory saving for that one model.
I will take (1) next: it keeps #304's saving and #2393's import semantics both. The imported_buft_context initialization goes in with it (a nameless, device-less buffer type is a bug on its own).
What changed: the loader wraps its file mapping through a new HRX proc, ggml_backend_hrx_model_buffer_from_mmap, which gives a mapped host weight buffer instead of Census 2026-10-09 #371's import. The imported buffer type also gets a real context.
zaya1-8b: loads and generates, with plain flags and with the gate's placement env.
Tests:hrx-buffer-test 10/10. The new case fails without the change.
Qwen3.8-27B: text identical, tg64 equal in an interleaved A/B.
The gate row turns green once an engine pin carries #393.
Fixed on engine main: engine#2487 merged, pinning llama.cpp 6b9fcacb7, which contains llama.cpp#393.
Measured on that exact pinned tree by the hw-gate, run 43692116895 / job 132744775983:
model zaya1-8b no longer fails; the model loads.
kat zaya1-8b PASS (9 cases, 0 warnings).
needle zaya1-8b PASS (7341 at a 7,403-token prompt).
kld zaya1-8b PASS (mean 0.0083).
zaya1's remaining placement row is engine#2450's GET_ROWS x2, the same pattern as every other model, and is tracked there. Test follow-up: llama.cpp#394 (a loader-path test that fails on pre-#393 libraries). Closing.
Taking this one. The repro is queued (hw-gate.sh --models zaya1-8b --bin ~/.cache/hw-gate/build-2bc61b1de8/bin behind a gpu-hold -w) and I will post the log it produces. Three things from the CPU side first, because they narrow what to look for:
The row
# name | file under $MODELS | kat --repeat | needle (1 = run tests/hrx_long_prompt.py) | extra llama-server args
zaya1-8b|zaya1-8b-Q4_K_M.gguf|2|1
No extra server arguments — so the crash is on the default path, not a flag interaction. The comment above the table is the useful part: "The first three are K-quants (ZAYA1-8B routes 1 of 16 experts)".
The model is an SSM, and the pin knows it
third_party/llama.cpp names zaya in four places:
conversion/zaya.py <- a converter, so the architecture is supported
ggml/src/ggml-cuda/ssm-conv.cu
ggml/src/ggml-hrx/CMakeLists.txt
ssm-conv is the signal: this is a state-space model, not a transformer, and its load path allocates conv/state buffers a dense transformer never touches. A 1-of-16-expert SSM is also the only row exercising MUL_MAT_ID at that expert ratio.
What that predicts
rc 3 with a death in _start () is a crash, not a refusal. A model loader that does not recognise an architecture prints unknown model architecture and exits cleanly — it does not take a backtrace. So this is far more likely a NULL dereference or an OOB write in the SSM init path on load than "zaya1 is unsupported".
Two specific candidates worth grepping before the log arrives:
an SSM tensor the Q4_K_M quantization retained but the loader expects in another type — conversion/zaya.py will say which names it renames;
the conv-state buffer sized from a header field the quantised GGUF writes differently — that would fault at allocation, which is exactly a load-time crash.
Why it matters beyond this row: it is a fleet-wide red — every PR's gate carries it — and if the cause is an SSM load path rather than zaya1 specifically, any future SSM model inherits it.
The log will settle it; I am not going to guess further without it.
Counts across the file: 40 layers have zaya_router_biases, 39 have zaya_router_eda. The one without is blk.0.
The architecture's tensor table (src/llama-arch.cpp:584-587) registers all four, and LLM_TENSOR_ZAYA_ROUTER_EDA_SCALE is looked up per layer — so the loader resolves blk.0.zaya_router_eda.weight, gets nothing, and dereferences it. That is a NULL at load, which is exactly rc 3 with a death in _start (): a crash, not a refusal.
So the row's failure has two independent faults, and only one is code
The GGUF is non-conforming.conversion/zaya.py emits zaya_router_eda for layers ≥ 1 and not for layer 0. Whatever produced this file dropped it — a per-layer list with a falsy first element is the classic way that happens (_legacy_hparams in the converter already guards against this shape with a comment about "per-layer lists hold 0 on the layers the value does not apply to", so the pattern is known there).
The loader crashes on a missing required tensor instead of naming it.That is the code bug, and it is the one worth fixing here: a model file missing a tensor should produce zaya_router_eda.weight not found for layer 0, not a segfault. Every other GGUF with one absent tensor gets the same treatment, and a segfault in the loader is unactionable for whoever hits it next.
What I would do, in this order
Make the loader error, not crash — the missing-tensor path should be the named failure the rest of the loader already produces for absent tensors. That turns a fleet-wide rc 3 into a fleet-wide readable message and does not depend on sourcing a new GGUF.
Then fix the converter so layer 0 gets its EDA scale, so the row actually goes green. Re-converting needs a run with the source weights and is the larger half.
Keep the row either way: it is the only case exercising a 1-of-16-expert SSM, and a model whose GGUF is known-broken is a useful fixture for the loader-error path once that path exists.
The gpu-hold reproduction is still queued (hw-gate.sh --models zaya1-8b --bin …build-2bc61b1de8/bin) and I will post its log when it returns. I expect it to show the null dereference at the first blk.0 EDA lookup, which would confirm this from the crash side — but the file-level evidence above does not depend on that run, and the two faults it names are actionable either way.
Reproduction: zaya1-8b PASSES. Same pin, same model, same row — and that refutes my own diagnosis above.
## HRX hardware gate: PASS
Commit `931f518a`, llama.cpp `aae78ecdf077`, hrx-system `7f4236924aad`, binaries …/build-2bc61b1de8/bin, 268s
Tctl at end 65 C, amdgpu ring timeouts during the run: 0
| kat zaya1-8b | PASS | 100 | 9 cases, 0 warnings |
| needle zaya1-8b | PASS | - | 7403-token prompt: expected 7341, got '7341' |
| placement zaya1-8b | PASS | - | all on HRX in 193 graphs (largest 6449 nodes) |
| kld zaya1-8b | PASS | - | mean KLD -0.000000, p99 0.00004, same top 100.00% |
| ops | PASS | 160 | 1959/1959 |
llama.cpp aae78ecdf077 is the pin the issue reports failing (run 43677464548 / job 132700631846). It passes here, with the same model file (~/models/zaya1-8b-Q4_K_M.gguf) and the same MODEL_TABLE row, on the box's current state.
So my blk.0 finding was a true observation and a wrong cause
I reported that blk.0 is missing zaya_router_eda.weight — 39 of 40 layers have it — and inferred a NULL at load. The inference was wrong: the model loads, and the gate's kat/needle/placement/kld arms all pass on it. Whatever the loader does with an absent EDA scale, it is not a dereference — it tolerates it, or synthesises it.
That is worth leaving in the thread rather than editing out: the tensor count is real, the causal claim was mine, and the reproduction is what separated them. If it is genuinely optional, that is a finding in its own right and belongs on the loader's tolerance rather than on this row.
What the row's failure therefore was
Environmental, not the model. The failure signature (rc 3, death in _start (), a gdb backtrace printed into the log) is a process death, and the two things that differ between that run and this one are box state and runner clean-up of _temp — not code, not the GGUF, not the pin. Candidates in order of plausibility:
GTT pressure. This row is 10.0 GiB of weights in a 46.6 GiB post-load GTT on a box that has hit the engine#315 regime before; a load that runs out of GTT can die in the allocator, which looks exactly like this.
A leftover process holding a device context on the CI runner — the failure this fleet has produced twice this week, and the reason server_launch_check now exists.
The compressed-vmlinuz class of mistake: the failing row's server log was deleted from _temp before anyone read it, so the actual message is unrecoverable and every hypothesis above is unverifiable from that run.
What I would do with the row
Do not close it — it is a fleet-wide gate row that failed twice and now passes once, and "passes on my box" is not "fixed". Two concrete asks:
Keep the server log on failure. The row already writes it; the run cleaned _temp before the issue could read it. A gate that deletes its own evidence turns an environmental flake into an unfalsifiable bug.
Record GTT after load and MemAvailable in the failure line, the way the memory pre-check already does for the predicted need. Then a recurrence is attributable in one line instead of three hypotheses.
I have the reproduction command and the build pinned, so a recurrence can be re-run in 268 s rather than a week.
Handover: the row fix is (b), and cl-hrxmem owns it. My part is the instrumentation, and it is delivered.
The owner ruled on #2501's duplicate-PR question and the decision covers this row too: swap the coder-30b/zaya-class row to a routed MoE already on the box, proven clean under the watchdog, and comment the row re-enable when #2494 closes. A blanket "model not on this runner = SKIP" was decided against — and correctly, because the gate cannot tell "never provisioned" from "deleted", so a SKIP would hide an accidental deletion of a 30 GiB GGUF behind a warning nobody reads. I had proposed exactly that on #2518; it is closed.
What this issue got from me, and what state it is in:
Reproduction
PASSES — same pin (aae78ecdf077), 268 s, 0 ring timeouts, kat/needle/placement/kld all green. The row's failure was environmental, and my blk.0 hypothesis was a true observation with a wrong cause.
Instrumentation 1
merged (#2513): a failing row keeps its server log at $STORE/failed/<run>-<model>.log and the failure line names the path. The old line carried 3 lines, which is why this issue shows a gdb backtrace tail instead of a reason.
Instrumentation 2
open (#2519): a failing row records the device state at failure — [GTT now X GiB, MemAvailable Y GiB] — because the pre-check records the predicted need, which is why the failing line says MemAvailable 67.8 GiB: ok and then died anyway. Absent values print ?, never 0.0.
So the row now fails with enough to attribute it: model, rc, three lines of its log, device state at the time, and the full log path — and for #2501's row, "(b) + comment + cl-hrxmem" rather than a status change.
Releasing my claim. The remaining work is the owner's decided (b), which is someone else's edit to MODEL_TABLE; mine was the "make a recurrence readable" half and it is in.
If the row recurs before (b) lands: the failure line will now name the kept log and the GTT at death, so a single line is enough to close this as environmental with evidence instead of three hypotheses.
Activity
bong-water-water-bong commentedon Nov 15, 2026
Backtrace from the gate log (the row printed only the last frames). On both runs,
llama-serveraborts at load in the scheduler, during the fused-op probe:The command is
env GGML_HRX_REPORT=2 GGML_HRX_STRICT=0 llama-server -m zaya1-8b-Q4_K_M.gguf --device HRX0 -ngl 99 --jinja -c 8192 -np 1 ... -lv 4(the gate'splace_env/place_args).ggml_backend_sched_backend_id_from_curaborts when a tensor sits in a buffer that no backend in the scheduler can run, which ties this to the same 'embd' GET_ROWS placement as engine#2450. A gpu-hold repro is queued, with and without the gate's placement env, to get the abort message itself (frames #0-#4 and the assert text are not in the gate output).bong-water-water-bong commentedon Nov 15, 2026
Repro (gpu-hold, my build of the pinned fork tree
a353b7fee+ gpu-kernels2022afb):zaya1-8bdoes not load at all. The placement env is not the trigger.The same abort happens with the gate's
GGML_HRX_REPORT=2 GGML_HRX_STRICT=0 ... -lv 4and with plain flags (-ngl 99 --jinja -c 8192 -np 1). So it is not engine#2450's GET_ROWS placement. A VIEW of ZAYA's CCA conv weight (blk.0.cca_conv_grp.weight) lands in an unnamed buffer that no scheduler backend can run, during the fused-op reserve. A load-only bracket over older builds is queued: the gate's cached builds ate6c80014eand4a375bf2d, plus earlier ones of mine, to find when it started.bong-water-water-bong commentedon Nov 15, 2026
Cause isolated (gpu-hold, fork tree
a353b7fee).Bracket and switch:
e6c80014e4a375bf2df3d177a40(+ gk 6105368)a353b7feea353b7fee+GGML_HRX_MODEL_BUFFER_COPY=1(weights copied to a device buffer, the pre-#304 path)Mechanism:
load_tensors: model buffer size = ..., 120+ of them.imported_buft_context) is never initialized, so its name is "" and its device is null. Initializing it, locally, only changes the abort text toin a buffer (HRX0_Imported); it still aborts.imported_type_is_host= false) on purpose.src/models/zaya.cpp~397-409) multiplies by a strided 3-Dggml_view_3dof the F16 weightblk.N.cca_conv_grp.weight, one per tap, in a batched MUL_MAT over the groups. HRX does not take that view, the CPU cannot read a non-host buffer, soggml_backend_sched_backend_id_from_curaborts: pre-allocated tensor ... in a buffer that cannot run the operation (VIEW).Fix options (cl-hrxmem leads; touching #2393's design, so cl-1051 is cc'd by name):
model-buffer-from-mmap.cppsays they should), not the import path. That needs finding where the loader'sbuffer_from_host_ptrcall reacheshrx_imported_buffer_from_ptrfor weights.GGML_HRX_MODEL_BUFFER_COPY=1. That costs the Census 2026-10-04 #304 memory saving for that one model.I will take (1) next: it keeps #304's saving and #2393's import semantics both. The
imported_buft_contextinitialization goes in with it (a nameless, device-less buffer type is a bug on its own).bong-water-water-bong commentedon Nov 15, 2026
Fix up as 1bit-MONSTER/llama.cpp#393 (
7e74ec3, off fork maina353b7fee).ggml_backend_hrx_model_buffer_from_mmap, which gives a mapped host weight buffer instead of Census 2026-10-09 #371's import. The imported buffer type also gets a real context.hrx-buffer-test10/10. The new case fails without the change.The gate row turns green once an engine pin carries #393.
bong-water-water-bong commentedon Nov 15, 2026
Fixed on engine main: engine#2487 merged, pinning llama.cpp
6b9fcacb7, which contains llama.cpp#393.Measured on that exact pinned tree by the hw-gate, run 43692116895 / job 132744775983:
model zaya1-8bno longer fails; the model loads.kat zaya1-8bPASS (9 cases, 0 warnings).needle zaya1-8bPASS (7341 at a 7,403-token prompt).kld zaya1-8bPASS (mean 0.0083).zaya1's remaining
placementrow is engine#2450's GET_ROWS x2, the same pattern as every other model, and is tracked there. Test follow-up: llama.cpp#394 (a loader-path test that fails on pre-#393 libraries). Closing.bong-water-water-bong commentedon Nov 15, 2026
Taking this one. The repro is queued (
hw-gate.sh --models zaya1-8b --bin ~/.cache/hw-gate/build-2bc61b1de8/binbehind agpu-hold -w) and I will post the log it produces. Three things from the CPU side first, because they narrow what to look for:The row
No extra server arguments — so the crash is on the default path, not a flag interaction. The comment above the table is the useful part: "The first three are K-quants (ZAYA1-8B routes 1 of 16 experts)".
The model is an SSM, and the pin knows it
third_party/llama.cppnames zaya in four places:ssm-convis the signal: this is a state-space model, not a transformer, and its load path allocates conv/state buffers a dense transformer never touches. A 1-of-16-expert SSM is also the only row exercisingMUL_MAT_IDat that expert ratio.What that predicts
rc 3with a death in_start ()is a crash, not a refusal. A model loader that does not recognise an architecture printsunknown model architectureand exits cleanly — it does not take a backtrace. So this is far more likely a NULL dereference or an OOB write in the SSM init path on load than "zaya1 is unsupported".Two specific candidates worth grepping before the log arrives:
conversion/zaya.pywill say which names it renames;Why it matters beyond this row: it is a fleet-wide red — every PR's gate carries it — and if the cause is an SSM load path rather than zaya1 specifically, any future SSM model inherits it.
The log will settle it; I am not going to guess further without it.
bong-water-water-bong commentedon Nov 15, 2026
Found it by reading the GGUF, not by running it. Layer 0 is missing a tensor.
GGUF tensor names are plaintext, so no server and no ticket are needed:
Counts across the file: 40 layers have
zaya_router_biases, 39 havezaya_router_eda. The one without isblk.0.The architecture's tensor table (
src/llama-arch.cpp:584-587) registers all four, andLLM_TENSOR_ZAYA_ROUTER_EDA_SCALEis looked up per layer — so the loader resolvesblk.0.zaya_router_eda.weight, gets nothing, and dereferences it. That is a NULL at load, which is exactlyrc 3with a death in_start (): a crash, not a refusal.So the row's failure has two independent faults, and only one is code
conversion/zaya.pyemitszaya_router_edafor layers ≥ 1 and not for layer 0. Whatever produced this file dropped it — a per-layer list with a falsy first element is the classic way that happens (_legacy_hparamsin the converter already guards against this shape with a comment about "per-layer lists hold 0 on the layers the value does not apply to", so the pattern is known there).zaya_router_eda.weight not found for layer 0, not a segfault. Every other GGUF with one absent tensor gets the same treatment, and a segfault in the loader is unactionable for whoever hits it next.What I would do, in this order
rc 3into a fleet-wide readable message and does not depend on sourcing a new GGUF.The gpu-hold reproduction is still queued (
hw-gate.sh --models zaya1-8b --bin …build-2bc61b1de8/bin) and I will post its log when it returns. I expect it to show the null dereference at the firstblk.0EDA lookup, which would confirm this from the crash side — but the file-level evidence above does not depend on that run, and the two faults it names are actionable either way.bong-water-water-bong commentedon Nov 15, 2026
Reproduction: zaya1-8b PASSES. Same pin, same model, same row — and that refutes my own diagnosis above.
llama.cpp aae78ecdf077is the pin the issue reports failing (run 43677464548 / job 132700631846). It passes here, with the same model file (~/models/zaya1-8b-Q4_K_M.gguf) and the sameMODEL_TABLErow, on the box's current state.So my
blk.0finding was a true observation and a wrong causeI reported that
blk.0is missingzaya_router_eda.weight— 39 of 40 layers have it — and inferred a NULL at load. The inference was wrong: the model loads, and the gate'skat/needle/placement/kldarms all pass on it. Whatever the loader does with an absent EDA scale, it is not a dereference — it tolerates it, or synthesises it.That is worth leaving in the thread rather than editing out: the tensor count is real, the causal claim was mine, and the reproduction is what separated them. If it is genuinely optional, that is a finding in its own right and belongs on the loader's tolerance rather than on this row.
What the row's failure therefore was
Environmental, not the model. The failure signature (
rc 3, death in_start (), a gdb backtrace printed into the log) is a process death, and the two things that differ between that run and this one are box state and runner clean-up of_temp— not code, not the GGUF, not the pin. Candidates in order of plausibility:server_launch_checknow exists.vmlinuzclass of mistake: the failing row's server log was deleted from_tempbefore anyone read it, so the actual message is unrecoverable and every hypothesis above is unverifiable from that run.What I would do with the row
Do not close it — it is a fleet-wide gate row that failed twice and now passes once, and "passes on my box" is not "fixed". Two concrete asks:
_tempbefore the issue could read it. A gate that deletes its own evidence turns an environmental flake into an unfalsifiable bug.GTT after loadandMemAvailablein the failure line, the way the memory pre-check already does for the predicted need. Then a recurrence is attributable in one line instead of three hypotheses.I have the reproduction command and the build pinned, so a recurrence can be re-run in 268 s rather than a week.
hw-gate: keep the failing server log and name it in the failure line (#…
bong-water-water-bong commentedon Nov 15, 2026
Handover: the row fix is (b), and cl-hrxmem owns it. My part is the instrumentation, and it is delivered.
The owner ruled on #2501's duplicate-PR question and the decision covers this row too: swap the coder-30b/zaya-class row to a routed MoE already on the box, proven clean under the watchdog, and comment the row
re-enable when #2494 closes. A blanket "model not on this runner = SKIP" was decided against — and correctly, because the gate cannot tell "never provisioned" from "deleted", so a SKIP would hide an accidental deletion of a 30 GiB GGUF behind a warning nobody reads. I had proposed exactly that on #2518; it is closed.What this issue got from me, and what state it is in:
aae78ecdf077), 268 s, 0 ring timeouts,kat/needle/placement/kldall green. The row's failure was environmental, and myblk.0hypothesis was a true observation with a wrong cause.$STORE/failed/<run>-<model>.logand the failure line names the path. The old line carried 3 lines, which is why this issue shows a gdb backtrace tail instead of a reason.[GTT now X GiB, MemAvailable Y GiB]— because the pre-check records the predicted need, which is why the failing line saysMemAvailable 67.8 GiB: okand then died anyway. Absent values print?, never0.0.So the row now fails with enough to attribute it: model, rc, three lines of its log, device state at the time, and the full log path — and for #2501's row, "(b) + comment + cl-hrxmem" rather than a status change.
Releasing my claim. The remaining work is the owner's decided (b), which is someone else's edit to
MODEL_TABLE; mine was the "make a recurrence readable" half and it is in.If the row recurs before (b) lands: the failure line will now name the kept log and the GTT at death, so a single line is enough to close this as environmental with evidence instead of three hypotheses.