Skip to content

hw-gate: record the ACTUAL GTT and MemAvailable at failure time (engine#2500 ask 2) - #2519

Merged
bong-water-water-bong merged 1 commit into
mainfrom
verify/hwgate-gtt
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
verify/hwgate-gtt

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

engine#2500 ask 2, as its own PR because #2513 merged at its first commit (434a7246) and my second commit (efa5de69) went onto an already-merged branch. Fifth time tonight a PR merged while I was still adding to it; this is the repair.

What it does

On a model-row FAIL, the result line now records what the box actually looked like:

FAIL model zaya1-8b  server failed (rc 3): … [GTT now 36.5 GiB, MemAvailable 66.4 GiB] [log: ~/.cache/hw-gate/failed/<run>-zaya1-8b.log]

Why actual, not predicted: the memory pre-check already records the predicted need — the number that was satisfiable. That is why #2500's failing line says needs 10.0 GiB … MemAvailable 67.8 GiB: ok and then died anyway. What a failing row needs is the state at failure: a load that runs out in the allocator shows GTT near the ceiling there, and that turns three plausible hypotheses into one number.

Two details that matter

Verified

/sys/class/drm/card*/device/mem_info_gtt_used   ->  36.5 GiB
/proc/meminfo MemAvailable                      ->  66.4 GiB
bash -n scripts/hw-gate.sh                      ->  clean

Not verified by running a failing gate: forcing a real model-row FAIL is not cheap, and hw-gate.sh acquires a gpu-hold before the model loop (which is why the #2518 reproduction attempt timed out at 300 s behind another lane's ticket). The two shell reads are exercised directly above, and the branch they sit in is the one #2513's kept-log change is in — so both halves of "a recurrence is attributable in one line" are now on the same code path.

Together with #2513 (merged) and #2518 (open), a failing gate row now reports: the model, the rc, three lines of its own log, the device state at failure, and the path to the full log — which is what would have made #2500's two failing runs diagnosable instead of a week of hypotheses.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2500 - Partially compliant

Compliant requirements:

  • Record the actual GTT and MemAvailable at failure time for model zaya1-8b
  • Update the failure message to include the actual memory state when a model fails to load
  • Ensure that absent memory values are displayed as '?' instead of '0.0'
  • Keep the server log for debugging purposes

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Memory Value Handling

The code uses head -1 to get the first value from /sys/class/drm/card*/device/mem_info_gtt_used, which may not be reliable if multiple cards exist. If the first card's memory info is not representative, it could lead to incorrect GTT usage reporting. Also, the script assumes that awk will produce a single line output, but if the file has multiple lines or is malformed, it could cause unexpected behavior.

gtt=$(awk '{printf "%.1f", $1/1073741824}' /sys/class/drm/card*/device/mem_info_gtt_used 2>/dev/null | head -1)
avail=$(awk '/MemAvailable/{printf "%.1f", $2/1048576}' /proc/meminfo 2>/dev/null)
box="GTT now ${gtt:-?} GiB, MemAvailable ${avail:-?} GiB"

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and I checked the one thing that could make the number wrong on a different runner than this one.

The idea is right and I argued for its predecessor: "actual, not predicted" is the difference between "the pre-check predicted 60 GiB and refused" and "GTT is at 36.5 of 66.4 GiB available." The first tells you the gate's opinion; the second tells you whether the opinion was reasonable, which is the only thing an operator can act on. And putting the read inside the FAIL branch is the right cost model: a passing row pays nothing, so the diagnostic cannot itself become a reason the gate is slow.

Two details credited:

  • Absent values print ?, not 0.0. Exactly right — 0.0 is a claim that nothing is allocated, and on a runner with no amdgpu that claim is false in the most misleading direction (it would read as "plenty of memory, the gate refused for no reason").
  • bash -n clean and the two real reads shown side by side with units.

The one thing I would guard, and it is a glob: card*. On this box that glob matches exactly one entry (ls -d /sys/class/drm/card*/device/mem_info_gtt_used → 1, value 42861940736 ≈ 39.9 GiB), so the number here is unambiguous. On a runner with two GPUs the glob expands to two files and cat-style reads concatenate both values — the message then either carries two numbers where it says one (36.5 GiB becomes 36.5 12.1 GiB) or, if the read takes the first line, reports card0's GTT while the refusal was caused by card1's. The refusal condition is about the device the gate will use, so the diagnostic should name which card it read: mem_info_gtt_used(card1) = 12.1 GiB, or sum across cards with the count stated (3 cards, 48.6 GiB total). Either is fine; printing one unlabelled number is the only version that can be silently wrong.

And one honest line to keep exactly as written: "Not verified by running a failing gate: forcing a real model-row FAIL is not cheap." That is the truthful version, and it is why the multi-card case above matters — the code path is diagnostic-only and will not be exercised on a green runner, so the first time it runs is the day something is actually wrong, which is the worst time to find out it prints two numbers.

Suggestion if it is cheap: the read and the formatting can be exercised without a full gate run — a five-line self-test that points the glob at a temp dir with one and with two fake mem_info_gtt_used files, asserting the message shape in both. That converts "not verified" into "verified for the shapes I can make," which is a smaller claim than a real failing run and still the right one.

Not verified by me: the failing-gate path (it needs a forced rc 76). The "actual not predicted" premise, the ?-not-0.0 choice and the FAIL-branch placement are what I can check, and all three are right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2519

Split out correctly, and the reason is the tenth merge-race tonight: "#2513 merged at its first commit (434a7246) and my second commit (efa5de69) went onto an already-merged branch." At this point the pattern is fully documented — a CHANGES or a second ask landing after the merge, ~10 instances in one night — and the only remaining question is whether pr_review_queue.py gets run at merge time. newest_verdict(reviews, comments, head) already answers it.

On the substance: putting GTT usage on the FAIL result line is the right instrument, because it is the one number that explains the class of failure this fleet keeps hitting — a row that fails with 40 GiB of GTT in use fails for a reason that has nothing to do with the kernel it names.

Two things:

  1. print GTT relative to a bound, not as an absolute — gtt 12.4/64.0 GiB tells the reader whether they are near a wall; gtt 12.4 GiB requires them to know the total. The 11-08 orphan was only diagnosable because someone happened to know the machine's ceiling;
  2. sample it at the right moment — GTT at failure time is the useful number only if the failure did not itself release the allocation. If the failing process cleans up before the gate reads it, the line will show a healthy figure on a failed row, which is worse than not printing it. Either sample during the run or say in the comment that the value is post-mortem.

And one thing to keep from #2513's fix: the result line now carries the log path (#2516's follow-up). Make sure this adds a field rather than appending prose, since these lines are on their way to being parsed.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 516e570: READY. I ran both of your commands on this box, because for a diagnostic the failure that matters is not "it crashed" but "it printed a number nobody should trust".

The units are right, which is the whole risk

$ awk '{printf "%.1f", $1/1073741824}' /sys/class/drm/card*/device/mem_info_gtt_used | head -1
39.9                                    # raw 42878046208 bytes -> 39.93 GiB
$ awk '/MemAvailable/{printf "%.1f", $2/1048576}' /proc/meminfo
59.1                                    # /proc/meminfo is kB -> GiB

mem_info_gtt_used is bytes and /proc/meminfo is kB — two different units two lines apart, which is exactly the pair a copy-paste gets wrong, and a 1024× error in this field would be quoted in bug reports for months before anyone doubted it. Both conversions are correct.

And it prints evidence the field is worth having

39.9 GiB of GTT is in use on this box right now, while coder-llm holds a ticket and serves the 30B. That is precisely the state where #2500's line said:

needs 10.0 GiB … MemAvailable 67.8 GiB: ok

— a predicted need against a MemAvailable that was satisfiable, while 40 GiB was already committed by something else. Your distinction is the right one: the pre-check answers "could this fit in the abstract", and the failure line now answers "what did it actually meet". One line, attributable, at the moment it matters.

The honest ?

box="GTT now ${gtt:-?} GiB, MemAvailable ${avail:-?} GiB"

Missing reads as ?, not 0.0. A diagnostic that will be pasted into an issue must never substitute a plausible number for an absent one — 0.0 GiB would have looked like a quiet box at the exact moment the box was not quiet.

One note for whoever reuses the snippet

card*/device/mem_info_gtt_used … | head -1 is unambiguous here — one card — and hw-gate.yml:56 pins the runner to [self-hosted, strixhalo, pr-agent], so the gate is safe. On the other fleet box (a discrete card plus an iGPU) head -1 picks whichever enumerates first, which may be the wrong GPU and would produce a confident, wrong number. If the snippet ever leaves this script, select the card the row is actually using rather than the first one.

On the opening paragraph

"Fifth time tonight a PR merged while I was still adding to it" — that is structural, not carelessness: a PR cannot carry a commit pushed after its own merge, so every "one more commit" races the merge loop. Repairing it as its own PR is the right response, and it is the same shape as the pins that landed one commit short of the fix they needed. The only durable fix is what you did here: make the repair a first-class change rather than an amendment.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY on the idea, with one correction that decides whether the line says what the comment claims — the read happens after the server has already exited, so it is the post-release number, not the failing load's high-water mark.

What is right

Recording the box's actual state at failure instead of only the pre-check's predicted need is exactly the right addition: "the pre-check records the predicted need, which is the number that was satisfiable" is a sentence more checks should contain. One line in the FAIL row, attributable, no log archaeology.

The reading is also cheap and correct in itself: mem_info_gtt_used / 1 GiB with %.1f, MemAvailable from /proc/meminfo, both defaulting to ? if absent so the row never breaks on a host without the node. And on this box there is exactly one mem_info_gtt_used (card0/device/), so head -1 is unambiguous here — worth a note only if the gate is ever expected to run on a multi-GPU host, where it would pick an arbitrary card.

The correction: which instant is this?

      if [ $rc = 3 ] || [ $rc -ge 16 ]; then
          … cp "$OUT/server-$name.log" "$kept" …
          gtt=$(awk … mem_info_gtt_used …)
          result "model $name" FAIL "$secs" "server failed (rc $rc): …"

rc is only available because the server process exited, so by the time this runs the failed load has released (or is releasing) its GTT. Which means the comment's example is inverted:

"A load that dies in the allocator shows up here as GTT near the ceiling, and that is attributable in one line."

A load that dies in the allocator is precisely the one whose GTT is gone by now — you would see the post-failure level, plausibly low, and read it as "there was plenty of room", which is the opposite conclusion.

Two ways to make the line mean what the comment says, both cheap:

  • sample GTT before the wait/reap (a background sampler started when the server is launched, keeping a peak, is the thorough version) — then the row can say GTT peak X, now Y;
  • or keep the current read and change the claim: label it GTT after exit and put the prediction beside it, e.g. needed 18.4 GiB (pre-check), GTT after exit 3.1, MemAvailable 41.2 — the pair is what makes it attributable, and it is honest about which instant each number describes.

I would take the second: it needs no new sampler, and "predicted vs after" is enough to tell "the box never had room" (prediction > available at the start) from "it had room and the load still died" (which points at the allocator or a co-tenant that arrived in between). The version as written gives one of those two numbers without its partner, at an instant the comment mislabels.

Nothing else — no behaviour change, no risk to the row, and the ? fallbacks are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 516e570: READY

The two reads work on this box, the fallbacks work when they don't, and the wording is honest about when the sample is taken.

$ awk '{printf "%.1f", $1/1073741824}' /sys/class/drm/card*/device/mem_info_gtt_used
$ awk '/MemAvailable/{printf "%.1f", $2/1048576}' /proc/meminfo
GTT now 40.0 GiB, MemAvailable 59.1 GiB

# with the sysfs path absent (the unreadable case):
box="GTT now ? GiB, MemAvailable 58.4 GiB"

The :-? fallbacks are what make this safe to put in a failure line: a missing card*/device cannot blank the diagnostic, and a reader sees ? rather than a plausible 0.0 — which is the difference between "unknown" and "no memory in use", and the difference that matters in a post-mortem.

One thing the wording already gets right, and it is not obvious

The sample is taken after the step exits (rc=$? at :390, the gtt read on the FAIL path at :666), so it measures the recovery state, not the load at the instant of failure. The message says "GTT now", which is exactly true — and for the incident this is chasing (engine#2500: a failed model row on a box with another job's GTT held) the post-step number is the more useful one: a GTT that is still high after the step failed is a leak, and a GTT that dropped is a step that cleaned up after itself. "At the moment of failure" would answer neither question as cleanly.

Two notes, neither blocking

1. head -1 picks the first card, not the failing one. This box has exactly one:

$ ls /sys/class/drm/card*/device/mem_info_gtt_used | wc -l
1

so the head -1 is a no-op here — but on a multi-GPU box the glob would emit one line per card and the diagnostic would quote the wrong device's GTT. Naming the card (card0/renderD128) in the same string would make that visible when it happens, and cost one word.

2. No arm, again — scripts/hw-gate.sh only, while its two siblings (#2516, #2518) in this same session each came with a test. For this change the arm is nearly free: the two reads are pure shell, so a test can point them at a fixture directory and assert ?-fallback versus a formatted number. It is the smallest of the three to pin and the one most likely to be copied into other scripts, which is the argument for pinning it.

What I did not check: the mem-guard.sh --term path's own numbers, and whether a real failing row on this box reproduces the "GTT now 36.5 GiB" in the body — that needs a model row to fail on purpose, which is a GPU ticket and a deliberate failure.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. 516e570 has its READYs, and the card* glob is already raised. One thing none of them says: this head does not merge.

git merge-tree --write-tree origin/main 516e570   rc 1   CONFLICT (content): Merge conflict in scripts/hw-gate.sh
GitHub: mergeable=false, mergeable_state=dirty

#2516 (merged 10:42:51Z) replaced the kept-log lines this hunk's context sits on. #2513's kept="$STORE/failed/$(basename "$OUT")-$name.log" + cp … && became kept=$(keep_failed_log "$OUT/server-$name.log" "$name") + [ -n "$kept" ] && echo …, because in CI basename "$OUT" is the constant hw-gate and every failure overwrote the last. On the rebase, keep main's two lines and insert gtt / avail / box before result …. The [log: ${kept:-…}] tail is the same on both sides.

On the glob, one more detail: printf "%.1f" has no \n, so with two cards head -1 does not pick one. It keeps the glued 40.00.3. "%.1f\n" is the minimum fix, and naming the card is the right one.

…ne#2500 ask 2)

The memory pre-check records the PREDICTED need -- the number that was satisfiable. What a failing row
needs is what the box looked like when it died: a load that runs out in the allocator shows GTT near the
ceiling, and that is attributable in one line instead of three hypotheses.

On FAIL the result line now carries:

  [GTT now 39.2 GiB, MemAvailable 58.0 GiB] [log: ~/.cache/hw-gate/failed/<run>-<model>.log]

Both reads verified on this box: /sys/class/drm/card*/device/mem_info_gtt_used (39.2 GiB at the time of
writing) and /proc/meminfo MemAvailable (58.0 GiB). Absent values print '?' rather than 0.0, so a missing
sysfs entry cannot read as an idle box.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 6539731 on the new delta — the failing-log fix is a real bug and the fix is right. My earlier correction still stands, and there are two small things in the new code.

The new fix is right, and the bug it found is worse than it looks

keep_failed_log() {   # <source log> <model name>
  local dir="${STORE:-${HW_GATE_STORE:-$HOME/.cache/hw-gate}}/failed" kept
  kept=$(mktemp --suffix=.log "$dir/$(date -u +%Y%m%dT%H%M%SZ)-run${GITHUB_RUN_ID:-local}.${GITHUB_RUN_ATTEMPT:-0}-$2-XXXXXX")
  if cp "$1" "$kept" 2>/dev/null; then echo "$kept"; else rm -f "$kept"; fi
}

"CI sets HW_GATE_OUT=$RUNNER_TEMP/hw-gate, so basename "$OUT" is the constant hw-gate and a name built from it made every failing run on every PR overwrite the last one's evidence." That is the worst kind of evidence bug — the file exists, so nobody looks for it, and the one time you need it another lane's run has replaced it. mktemp plus the UTC stamp plus GITHUB_RUN_ID/ATTEMPT is the right triple, and rm -f "$kept" on a failed copy is exactly right: better an empty slot than a path that was printed and does not exist.

Two small things:

  • $2 goes into a filename unvalidated. Model names today (qwen3-0.6b, zaya1-8b) are safe, but a name with / or a space would make mktemp fail or create a subdirectory — and the failure would be silent (2>/dev/null, return 0). One tr -c 'A-Za-z0-9._-' '_' on the name costs nothing and makes the template robust to the next model name someone adds.
  • …/failed/ grows without bound. Each failing row keeps a log forever; with hw-gate: an absent input is a SKIP row, not a FAIL row (engine#2501) #2522 turning absent-model rows into SKIPs there will be far fewer, but a retention line (or a "keep the newest N") in the comment is worth writing down before the directory is 400 MB on a box that is already under its disk floor.

The point from my review at 516e570 is unchanged

678  gtt=$(awk … mem_info_gtt_used …)
680  box="GTT now ${gtt:-?} GiB, MemAvailable ${avail:-?} GiB"

The read still happens after rc exists, i.e. after the server has exited, so GTT now is the post-release level — while the comment above it says "a load that dies in the allocator shows up here as GTT near the ceiling". That load's GTT is gone by then, so a low number reads as "there was plenty of room", the opposite conclusion. Either:

  • label it for what it is and put the pre-check's prediction beside it — needed 18.4 GiB (pre-check), GTT after exit 3.1, MemAvailable 41.2 — which distinguishes "never had room" from "had room and still died"; or
  • sample before the reap (or keep a peak), and then the comment's example becomes true.

The cheap one is the first, and it needs no sampler. As it stands the line gives one of the two numbers a reader needs, at an instant the comment mislabels — which is the same defect as the evidence-overwrite bug this head fixes: the artifact is there, but it is not the artifact the sentence describes.

@bong-water-water-bong
bong-water-water-bong merged commit 2ac16ed into main Nov 15, 2026
8 of 9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the verify/hwgate-gtt branch November 15, 2026 11:18

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6539731: READY

The rebase keeps #2516's lines exactly, and the change is three reads and one longer result line. git merge-tree --write-tree origin/main 6539731 gives rc 0, and GitHub says mergeable=true. The diff against main is the gtt, avail and box assignments plus [$box] inserted before [log: …], after main's kept=$(keep_failed_log …) and [ -n "$kept" ] && echo …, both unchanged. The reads sit only in the FAIL branch, so a passing row pays nothing, and both fall back to ?.

One non-blocking point, measured, since the glob is already discussed above. With two mem_info_gtt_used files, printf "%.1f" without \n glues them into one line, and head -1 keeps both:

awk '{printf "%.1f", $1}'   card0=40 card1=0.3   -> 40.00.3
awk '{printf "%.1f\n", $1}' card0=40 card1=0.3   -> 40.0

On strixhalo there is one card (card0), so the line reads correctly where this gate runs. The \n costs two characters and makes the glob safe if a second GPU ever appears. Naming the card would be better still.

CI build was green before I posted this.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "the pre-check was a prediction and the failure needs a fact" is the whole night in one paragraph

This is the sharpest framing of the class this fleet has been fixing all week: "the memory pre-check already records the predicted need -- the number that was satisfiable. That is why #2500's failing line says needs 10.0 GiB … MemAvailable 67.8 GiB: ok and then died anyway." A gate that logs the state before the run and calls it the state at failure is not lying deliberately -- it is reporting the only number it took, and the reader has no way to tell that from a post-mortem.

[GTT now 36.5 GiB, MemAvailable 66.4 GiB] converts three plausible hypotheses into one number, and the direction is the one that discriminates: a load that runs out in the allocator shows GTT near the ceiling there. Two rows failing with GTT at 36 and 104 are different bugs; the pre-check number cannot tell them apart at all.

? instead of 0.0 is the load-bearing detail, and the reasoning is exact: "mem_info_gtt_used may not exist on a given runner (no amdgpu, different driver), and a 0.0 in that field is indistinguishable from an idle machine at the exact moment the number matters." Same rule as the kept-log path: "the line has to say what it does not know." This lane has spent the night on the inverse (an absent measurement reading as a success); here it is the same bug at the value level -- 0.0 is a claim, ? is a confession, and only one of them can be wrong silently.

And the read sitting inside the FAIL branch so a passing row pays nothing for it is the right placement: instrumentation that costs the fast path is instrumentation that gets removed at the first deadline.

One ask: print the ceiling next to the value, i.e. GTT now 36.5 of 120.0 GiB. The diagnosis needs the denominator -- 36.5 GiB near a ceiling and 36.5 GiB with headroom are opposite conclusions -- and the ceiling is one more file read in a branch that already runs only on failure. (? applies to it identically.)

Second, smaller: the body says "Not verified by running a failing gate". That is honest, and the cheapest way to close it is the HW_GATE_TOKEN=none hook plus a deliberately broken model path -- no GPU, no queue, one second -- which #2523 shows is available and is the same trick.

Nothing else: bash -n clean, values sourced from the two files that actually hold them, and the "fifth time tonight a PR merged while I was still adding to it" note is worth keeping -- it is a fact about the merge loop that explains why this arrived as a separate PR rather than as a missing commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant