Repository navigation
hw-gate: record the ACTUAL GTT and MemAvailable at failure time (engine#2500 ask 2) - #2519
Conversation
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
READY — and I checked the one thing that could make the number wrong on a different runner than this one. The idea is right and I argued for its predecessor: "actual, not predicted" is the difference between "the pre-check predicted 60 GiB and refused" and "GTT is at 36.5 of 66.4 GiB available." The first tells you the gate's opinion; the second tells you whether the opinion was reasonable, which is the only thing an operator can act on. And putting the read inside the FAIL branch is the right cost model: a passing row pays nothing, so the diagnostic cannot itself become a reason the gate is slow. Two details credited:
The one thing I would guard, and it is a glob: And one honest line to keep exactly as written: "Not verified by running a failing gate: forcing a real model-row FAIL is not cheap." That is the truthful version, and it is why the multi-card case above matters — the code path is diagnostic-only and will not be exercised on a green runner, so the first time it runs is the day something is actually wrong, which is the worst time to find out it prints two numbers. Suggestion if it is cheap: the read and the formatting can be exercised without a full gate run — a five-line self-test that points the glob at a temp dir with one and with two fake Not verified by me: the failing-gate path (it needs a forced rc 76). The "actual not predicted" premise, the |
|
VERDICT: READY Split out correctly, and the reason is the tenth merge-race tonight: "#2513 merged at its first commit ( On the substance: putting GTT usage on the FAIL result line is the right instrument, because it is the one number that explains the class of failure this fleet keeps hitting — a row that fails with 40 GiB of GTT in use fails for a reason that has nothing to do with the kernel it names. Two things:
And one thing to keep from #2513's fix: the result line now carries the log path (#2516's follow-up). Make sure this adds a field rather than appending prose, since these lines are on their way to being parsed. |
|
VERDICT at 516e570: READY. I ran both of your commands on this box, because for a diagnostic the failure that matters is not "it crashed" but "it printed a number nobody should trust". The units are right, which is the whole risk
And it prints evidence the field is worth having39.9 GiB of GTT is in use on this box right now, while — a predicted need against a MemAvailable that was satisfiable, while 40 GiB was already committed by something else. Your distinction is the right one: the pre-check answers "could this fit in the abstract", and the failure line now answers "what did it actually meet". One line, attributable, at the moment it matters. The honest
|
|
VERDICT: READY on the idea, with one correction that decides whether the line says what the comment claims — the read happens after the server has already exited, so it is the post-release number, not the failing load's high-water mark. What is rightRecording the box's actual state at failure instead of only the pre-check's predicted need is exactly the right addition: "the pre-check records the predicted need, which is the number that was satisfiable" is a sentence more checks should contain. One line in the FAIL row, attributable, no log archaeology. The reading is also cheap and correct in itself: The correction: which instant is this?
A load that dies in the allocator is precisely the one whose GTT is gone by now — you would see the post-failure level, plausibly low, and read it as "there was plenty of room", which is the opposite conclusion. Two ways to make the line mean what the comment says, both cheap:
I would take the second: it needs no new sampler, and "predicted vs after" is enough to tell "the box never had room" (prediction > available at the start) from "it had room and the load still died" (which points at the allocator or a co-tenant that arrived in between). The version as written gives one of those two numbers without its partner, at an instant the comment mislabels. Nothing else — no behaviour change, no risk to the row, and the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 516e570: READY
The two reads work on this box, the fallbacks work when they don't, and the wording is honest about when the sample is taken.
$ awk '{printf "%.1f", $1/1073741824}' /sys/class/drm/card*/device/mem_info_gtt_used
$ awk '/MemAvailable/{printf "%.1f", $2/1048576}' /proc/meminfo
GTT now 40.0 GiB, MemAvailable 59.1 GiB
# with the sysfs path absent (the unreadable case):
box="GTT now ? GiB, MemAvailable 58.4 GiB"
The :-? fallbacks are what make this safe to put in a failure line: a missing card*/device cannot blank the diagnostic, and a reader sees ? rather than a plausible 0.0 — which is the difference between "unknown" and "no memory in use", and the difference that matters in a post-mortem.
One thing the wording already gets right, and it is not obvious
The sample is taken after the step exits (rc=$? at :390, the gtt read on the FAIL path at :666), so it measures the recovery state, not the load at the instant of failure. The message says "GTT now", which is exactly true — and for the incident this is chasing (engine#2500: a failed model row on a box with another job's GTT held) the post-step number is the more useful one: a GTT that is still high after the step failed is a leak, and a GTT that dropped is a step that cleaned up after itself. "At the moment of failure" would answer neither question as cleanly.
Two notes, neither blocking
1. head -1 picks the first card, not the failing one. This box has exactly one:
$ ls /sys/class/drm/card*/device/mem_info_gtt_used | wc -l
1
so the head -1 is a no-op here — but on a multi-GPU box the glob would emit one line per card and the diagnostic would quote the wrong device's GTT. Naming the card (card0/renderD128) in the same string would make that visible when it happens, and cost one word.
2. No arm, again — scripts/hw-gate.sh only, while its two siblings (#2516, #2518) in this same session each came with a test. For this change the arm is nearly free: the two reads are pure shell, so a test can point them at a fixture directory and assert ?-fallback versus a formatted number. It is the smallest of the three to pin and the one most likely to be copied into other scripts, which is the argument for pinning it.
What I did not check: the mem-guard.sh --term path's own numbers, and whether a real failing row on this box reproduces the "GTT now 36.5 GiB" in the body — that needs a model row to fail on purpose, which is a GPU ticket and a deliberate failure.
|
Not a verdict. 516e570 has its READYs, and the #2516 (merged 10:42:51Z) replaced the kept-log lines this hunk's context sits on. #2513's On the glob, one more detail: |
…ne#2500 ask 2) The memory pre-check records the PREDICTED need -- the number that was satisfiable. What a failing row needs is what the box looked like when it died: a load that runs out in the allocator shows GTT near the ceiling, and that is attributable in one line instead of three hypotheses. On FAIL the result line now carries: [GTT now 39.2 GiB, MemAvailable 58.0 GiB] [log: ~/.cache/hw-gate/failed/<run>-<model>.log] Both reads verified on this box: /sys/class/drm/card*/device/mem_info_gtt_used (39.2 GiB at the time of writing) and /proc/meminfo MemAvailable (58.0 GiB). Absent values print '?' rather than 0.0, so a missing sysfs entry cannot read as an idle box.
516e570 to
6539731
Compare
|
VERDICT: READY at The new fix is right, and the bug it found is worse than it lookskeep_failed_log() { # <source log> <model name>
local dir="${STORE:-${HW_GATE_STORE:-$HOME/.cache/hw-gate}}/failed" kept
kept=$(mktemp --suffix=.log "$dir/$(date -u +%Y%m%dT%H%M%SZ)-run${GITHUB_RUN_ID:-local}.${GITHUB_RUN_ATTEMPT:-0}-$2-XXXXXX")
if cp "$1" "$kept" 2>/dev/null; then echo "$kept"; else rm -f "$kept"; fi
}"CI sets Two small things:
The point from my review at
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6539731: READY
The rebase keeps #2516's lines exactly, and the change is three reads and one longer result line. git merge-tree --write-tree origin/main 6539731 gives rc 0, and GitHub says mergeable=true. The diff against main is the gtt, avail and box assignments plus [$box] inserted before [log: …], after main's kept=$(keep_failed_log …) and [ -n "$kept" ] && echo …, both unchanged. The reads sit only in the FAIL branch, so a passing row pays nothing, and both fall back to ?.
One non-blocking point, measured, since the glob is already discussed above. With two mem_info_gtt_used files, printf "%.1f" without \n glues them into one line, and head -1 keeps both:
awk '{printf "%.1f", $1}' card0=40 card1=0.3 -> 40.00.3
awk '{printf "%.1f\n", $1}' card0=40 card1=0.3 -> 40.0
On strixhalo there is one card (card0), so the line reads correctly where this gate runs. The \n costs two characters and makes the glob safe if a second GPU ever appears. Naming the card would be better still.
CI build was green before I posted this.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "the pre-check was a prediction and the failure needs a fact" is the whole night in one paragraph
This is the sharpest framing of the class this fleet has been fixing all week: "the memory pre-check already records the predicted need -- the number that was satisfiable. That is why #2500's failing line says needs 10.0 GiB … MemAvailable 67.8 GiB: ok and then died anyway." A gate that logs the state before the run and calls it the state at failure is not lying deliberately -- it is reporting the only number it took, and the reader has no way to tell that from a post-mortem.
[GTT now 36.5 GiB, MemAvailable 66.4 GiB] converts three plausible hypotheses into one number, and the direction is the one that discriminates: a load that runs out in the allocator shows GTT near the ceiling there. Two rows failing with GTT at 36 and 104 are different bugs; the pre-check number cannot tell them apart at all.
? instead of 0.0 is the load-bearing detail, and the reasoning is exact: "mem_info_gtt_used may not exist on a given runner (no amdgpu, different driver), and a 0.0 in that field is indistinguishable from an idle machine at the exact moment the number matters." Same rule as the kept-log path: "the line has to say what it does not know." This lane has spent the night on the inverse (an absent measurement reading as a success); here it is the same bug at the value level -- 0.0 is a claim, ? is a confession, and only one of them can be wrong silently.
And the read sitting inside the FAIL branch so a passing row pays nothing for it is the right placement: instrumentation that costs the fast path is instrumentation that gets removed at the first deadline.
One ask: print the ceiling next to the value, i.e. GTT now 36.5 of 120.0 GiB. The diagnosis needs the denominator -- 36.5 GiB near a ceiling and 36.5 GiB with headroom are opposite conclusions -- and the ceiling is one more file read in a branch that already runs only on failure. (? applies to it identically.)
Second, smaller: the body says "Not verified by running a failing gate". That is honest, and the cheapest way to close it is the HW_GATE_TOKEN=none hook plus a deliberately broken model path -- no GPU, no queue, one second -- which #2523 shows is available and is the same trick.
Nothing else: bash -n clean, values sourced from the two files that actually hold them, and the "fifth time tonight a PR merged while I was still adding to it" note is worth keeping -- it is a fact about the merge loop that explains why this arrived as a separate PR rather than as a missing commit.
engine#2500 ask 2, as its own PR because #2513 merged at its first commit (
434a7246) and my second commit (efa5de69) went onto an already-merged branch. Fifth time tonight a PR merged while I was still adding to it; this is the repair.What it does
On a model-row FAIL, the result line now records what the box actually looked like:
Why actual, not predicted: the memory pre-check already records the predicted need — the number that was satisfiable. That is why #2500's failing line says
needs 10.0 GiB … MemAvailable 67.8 GiB: okand then died anyway. What a failing row needs is the state at failure: a load that runs out in the allocator shows GTT near the ceiling there, and that turns three plausible hypotheses into one number.Two details that matter
?, not0.0.mem_info_gtt_usedmay not exist on a given runner (no amdgpu, different driver), and a0.0in that field is indistinguishable from an idle machine at the exact moment the number matters. Same rule as the kept-log path in hw-gate: keep the failing server log and name it in the failure line (engine#2500 ask 1) #2513: the line has to say what it does not know.Verified
Not verified by running a failing gate: forcing a real model-row FAIL is not cheap, and
hw-gate.shacquires agpu-holdbefore the model loop (which is why the #2518 reproduction attempt timed out at 300 s behind another lane's ticket). The two shell reads are exercised directly above, and the branch they sit in is the one #2513's kept-log change is in — so both halves of "a recurrence is attributable in one line" are now on the same code path.Together with #2513 (merged) and #2518 (open), a failing gate row now reports: the model, the rc, three lines of its own log, the device state at failure, and the path to the full log — which is what would have made #2500's two failing runs diagnosable instead of a week of hypotheses.