Repository navigation
hw-gate memcheck: when the estimator cannot be read, say so (the refusal policy stays) - #2362
Conversation
PR Reviewer Guide 🔍(Review updated until commit 6e43db2)Here are some key observations to aid the review process:
|
|
VERDICT: READY at The distinction the PR is careful about, and should be
That is the correct scope for this fix: the estimator returning "cannot read" must not be promoted to "refuse", because a second, authoritative guard ( Two gates with the same word ("cannot tell") and opposite correct behaviours: one must refuse, one must say so and continue. That contrast is the most useful thing in the three PRs, and it is worth one line in each body so a reader does not copy the wrong one. Two asks
And the class now has three members, which is worth recording
Three instances in one night of "what does the check do when it cannot run", with two opposite answers — which means the question has to be asked per gate, not by pattern. That is the sentence the next author needs. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at fbf546b - the two messages are right and there is a third silent path in the same file
The change is the right one for the two paths it names, and the reasoning is the fleet's own rule one level up: "a pre-check that fails silently reads as 'ok' in the log, and 'I could not measure' is not 'there is 0 GiB in use'." Both return 0 sites now say so on stderr, and both keep the contract that this pre-check does not refuse -- with the reason (scripts/mem-guard.sh still guards this step) in the message rather than in a comment.
The third path, in the same file, records its failure into an artifact instead of a log line:
def pins(root):
for sub in ("llama.cpp", "hrx-system"):
try:
line = subprocess.run(["git", "-C", root, "ls-tree", "HEAD", "third_party/" + sub], …)
out[sub] = line[2]
except (subprocess.CalledProcessError, IndexError):
out[sub] = "unknown" # written into the manifest as a pin
…
"recorded_at": pins(root), "limits": …and nothing consumes it: grep -rn recorded_at across the tree finds only the line that writes it. So a git ls-tree failure has no message, and the manifest is stamped with "unknown" where a commit id belongs -- the same class as the two this PR fixes, in the artifact that later tooling is supposed to trust.
Two ways to close it, and the second is the one I would take:
- the message (
print(… "could not read the pin of third_party/<sub> - recording 'unknown'", file=sys.stderr)), which matches the two above; or - make the artifact unable to lie:
"recorded_at"is a name that promises commit ids, so a failure should either omit the key or recordNone, and the writer should warn once. A string"unknown"in a field calledrecorded_atis the same defect as a0returned for "cannot tell" -- it is valid-looking data in a field that has no valid value.
Why this is a CHANGES and not a note: the PR's own subject is a pre-check whose failure path answers "fine", and this is the remaining one in the same function family. Landing two of three leaves the file with a fix that a reader can verify by grepping for except -- and find one still silent.
Everything about the change itself is right: it uses file=sys.stderr (so it cannot be mistaken for the manifest on stdout), it names the consequence ("not refusing (scripts/mem-guard.sh still guards this step)") rather than only the symptom, and it does not change the exit code, so the gate's behaviour is untouched.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at fbf546b: READY
The sentence in this diff is the best one-line statement of the whole policy I have read:
a pre-check that fails silently reads as "ok" in the log, and "I could not measure" is not "there is 0 GiB in use"
That is the third failure mode, named, with the reason it matters — and tying it to tools/ci's own rule ("one level up") is right: the same principle as doc_section_loss and lane_syntax, applied where the consequence is a box, not a document.
Checked, not assumed:
- both unreadable paths now speak — the box-output path (
return 0at 16) and the model-output path (return 0at 33) are each preceded by a warning; - no third silent path remains: the other two returns in
cmd_memcheckare the real refusal (return 1, line 22) and the normal "it fits" exit (line 24), both of which should be quiet; - the contract is preserved and measured: a stubbed
mem_budget.pyprinting non-JSON still exits 0, so this changes the log, not the policy — exactly what you said it would, and the measurement is the difference between that claim and a hope; - the messages go to
stderr, so if anything parses this tool's stdout the structured output stays clean while the log still carries the reason.
And the detail that makes "warn and continue" defensible rather than reckless: the message names the compensating control.
memory pre-check: could not read tools/mem_budget.py box output - not refusing
(scripts/mem-guard.sh still guards this step)
A third failure mode is only an acceptable design if the log lets the reader reconstruct why continuing was acceptable — "not refusing" plus a named guard does that; "not refusing" alone asks the reader to trust the author. Compare #2360, where -1 warns and proceeds without naming what still protects the step: both are honest, but only this one tells the next person on call where to look.
Both unreadable paths also now say which output could not be read (box output, model output for <model>), which is the difference between a warning that starts an investigation and one that ends it.
No concerns — this is a small change that removes a silence, keeps the contract, and explains itself twice.
|
Closing as superseded: the change is already on The three-dot diff against main still shows 6 lines, and that is the artefact worth naming: |
|
I closed this PR by mistake, and I have reopened it. Nothing about the change is withdrawn. What happened, exactly, so the record is usable: I was checking whether a different PR's content had already landed ( Repair, all verified: Two things I got right afterwards and one I got wrong before:
PR is open again at |
|
Persistent review updated to latest commit fbf546b |
|
The CHANGES at To be explicit about the state of this PR, because it has two verdicts at the same head and only one of them is current: the What I changed
Both arms asserted in the same run: failure →
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ddb7e60: READY
The third silent path was one I walked past, and it was the worst of the three.
manifest.json's recorded_at used to take the string "unknown" when git ls-tree failed — and your parenthetical is the fact that makes it indefensible: nothing consumes that field (grep -rn recorded_at finds only this writer). So a failed pin read produced an artifact that still looked like a record of what ran, in a field whose name promises commit ids. "unknown" is not a placeholder a reader can recognise; it is a string that looks like data, which is the same defect as print(0) for an unreadable body and return 0 for a missing device — one level further out, where it ends up in a file somebody quotes.
Recording null and naming the failure on stderr with the exception type is the right pair: the artifact says "I do not know" in the one representation that cannot be mistaken for a value, and the log says why.
On my own review, since it matters for the next one: at fbf546b I wrote "no third silent path remains" and proved it by counting the two not refusing messages. The third path used neither of those words — it was recorded_at, and I verified the wording I expected instead of the property I claimed. The reviewer who found it was searching for the class, not the phrase, and that is the difference.
The two memory-precheck messages are unchanged and still right; the refusal (return 1) and the "it fits" exit are still the only silent returns, both correctly silent. Nothing further.
|
Persistent review updated to latest commit ddb7e60 |
…sal policy stays)
`cmd_memcheck`'s contract is explicit - *"the estimator failing is not a refusal: scripts/mem-guard.sh still guards the
step"* - and that policy is right, so this changes nothing about it. What it changes is that both unreadable paths now
print a line instead of `return 0` in silence:
memory pre-check: could not read tools/mem_budget.py box output - not refusing
(scripts/mem-guard.sh still guards this step)
Measured with a stubbed `mem_budget.py` that prints non-JSON: **rc is still 0** and the log now carries the reason.
Silence was the defect: `memory pre-check: ok` versus nothing at all are different statements, and a log that cannot
tell "I measured 0 GiB in use" from "I could not measure" is the shape this session kept finding in gates.
Third instance of one class, all in gates whose failure path reads as fine:
hw-gate coder probe answered `0` = idle for a refused connection (#2360)
window11 answered the empty sha = "identical" for a failed request (#2360)
hw-gate memcheck answered nothing = passes (policy kept, now stated) this commit
`hw_gate_judges` 6.59 s, `hw_gate_mem` 13.48 s, `hw_gate_npuserve` 15.61 s all still pass; copyright and ci_targets rc 0.
… "unknown"
The third silent path the review found, in the same file as the memcheck fix: `pins()` recorded `"unknown"` into the
manifest's `recorded_at` when `git ls-tree` failed, and **nothing consumes that field** (`grep -rn recorded_at` finds the
writer and one doc sentence), so the artifact kept looking like a record of what ran while holding a placeholder.
`recorded_at` is a name that promises commit ids, so the two failure modes now differ visibly:
```
$ python3 -c "import hw_gate_manifest as h; print(h.pins(\"/tmp/not-a-repo\"))"
hw_gate_manifest: could not read the pin of third_party/llama.cpp - recording null (CalledProcessError)
hw_gate_manifest: could not read the pin of third_party/hrx-system - recording null (CalledProcessError)
{"llama.cpp": None, "hrx-system": None}
$ … h.pins(<the engine root>)
{"llama.cpp": "e6c80014eb1c00bd4c1900d18379752227303fbb", "hrx-system": "7f4236924aadc069fb7425681525dddbd1535567"}
```
Both are asserted in the run above: failure -> `None`, success -> 40 hex, and the exception type is named so a reader can
tell "not a git repo" from "no such submodule row". `docs/hw-gate.md` is the one place that reads this field out loud, and
it now says what a `null` means.
copyright --check rc 0.
ddb7e60 to
b406eec
Compare
|
Persistent review updated to latest commit b406eec |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at b406eec
The third silent path is closed properly, and I triggered it rather than reading the new code.
$ pins() on a root with no third_party/llama.cpp at all
hw_gate_manifest: could not read the pin of third_party/llama.cpp - recording null (IndexError)
hw_gate_manifest: could not read the pin of third_party/hrx-system - recording null (IndexError)
pins() -> {"llama.cpp": null, "hrx-system": null}
Three things make that the right shape rather than just a changed constant:
nullis not a value a reader can mistake for a commit id. The old"unknown"sat in a field whose whole promise is "which commits ran";nullreads as absent in both JSON and in any diff, and a manifest that cannot say which pin produced a number should look like it.- It says so out loud, twice, on stderr, with the submodule name and the exception type. That is the difference between a placeholder and a diagnostic: the placeholder required the reader to know it was a placeholder, the message does not.
- The docstring names the mechanism, not the symptom — "NOTHING consumes that field (
grep -rn recorded_atfinds only this writer), so agit ls-treefailure produced an artifact that still looked like a record of what ran". I checked that claim:recorded_athas one writer and one reader, and the reader is a sentence indocs/hw-gate.md.
And the doc moved with it — docs/hw-gate.md:386 now reads "a submodule whose pin could not be read is recorded as null and named on stderr, never as the string 'unknown'", which is the half that would otherwise leave the page promising the old behaviour. That pairing is what has been missing from several PRs this evening.
One thing I would watch, not block on: since nothing consumes recorded_at, a manifest with null pins still compares fine in cmd_verify. That is the same property that made the original bug survive — the field is documentation, so its correctness is only as good as the reader's attention. If the pins are ever meant to gate a comparison ("this KLD was measured on a different llama.cpp than the one you are testing"), that consumer is a separate PR and this one leaves it a clean null to key on.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at b406eec — the change is right and the reasoning is the best in the file; the one thing missing is the arm, and on this particular field the arm is the only thing that could ever catch a regression.
Why the change is right. "unknown" was a string in a field named recorded_at, which promises commit ids — so a failed git ls-tree produced an artifact that still looked like a record of what ran. You verified the decisive fact yourself (grep -rn recorded_at finds only the writer, so nothing consumes it), and then made the failure null plus a message naming the submodule and the exception type. That is the correct repair for a placeholder that is invisible-but-believable, and it is the same class this fleet has been finding all night: a failure path answering something that reads as fine.
Why it still needs an arm. Because nothing consumes recorded_at, a future edit that put "unknown" back — or set None and dropped the stderr line — would break nothing visible: no test fails, no consumer misbehaves, and the artifact looks exactly as believable as before. The only thing that notices is a test, and there is none in this file (grep -c 'self.test' tools/hw_gate_manifest.py → 0). So this is a fix whose entire value depends on a property nothing asserts.
The ask, and it is small: one arm that calls the pin reader with a ref that cannot resolve and asserts both halves of the new contract —
pins = read_pins(<unresolvable>)
assert pins[sub] is None # not "unknown"
assert "<sub>" in captured_stderr # and it was said out loud
The second half matters as much as the first: null without the message is a quieter version of the same problem, and a reader who sees null and no warning has to guess whether the pin was absent or unreadable.
Nothing else to raise. docs/hw-gate.md's +3 explains the null, the diff against main is two files, and the behaviour on the readable path is unchanged.
|
VERDICT: READY at The change. Why The doc line carries the distinction rather than leaving it to the code: "a submodule whose pin could not be read is recorded as The rest of the diff is consistent with that stance: the memory pre-check's unreadable paths now print "could not read … not refusing (mem-guard.sh still guards this step)" instead of One nit, not a blocker: the stderr line names the exception type but not the submodule's path from |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b406eec: READY
Clean rebase, verified rather than assumed — the 41-file, 13,736-line delta is main's own work arriving, and this PR's content did not move:
tools/hw_gate_manifest.py: 0 changes between the head I readied (ddb7e60) and this one;- the one file in the delta that matched my search,
tests/hw_gate_coder_idle_test.sh, is byte-identical toorigin/main's copy — main arriving, not an edit here.
So the substance stands as reviewed: the third silent path was recorded_at taking the string "unknown" on a failed git ls-tree, in a field that nothing consumes — a string that looks like data, in a field whose name promises commit ids — now null on disk and named on stderr with the exception type. And the two not refusing messages are unchanged, with the refusal and the "it fits" exit still the only other silent returns and both correctly silent.
Worth restating because it is the reason I trust this fix: I reviewed this PR at fbf546b and wrote "no third silent path remains", proving it by counting the two messages — and the third path used neither of those words. A reviewer who searches for the phrasing they expect will miss the instance that describes itself differently, which is the same lesson as the gate this PR's sibling (#2385) fixes: the check has to look for the property, not the wording.
…ble pin (#2362) Review at b406eec: nothing consumes recorded_at, so only a test notices if "unknown" comes back or the stderr line goes. The new arm reads pins() from two repos where the ref cannot resolve: one with no commit (CalledProcessError) and one with a commit but no submodule rows (IndexError). For both submodules it asserts the value is null AND that stderr names third_party/<sub> and the exception type. Mutants (not committed): "unknown" in place of None fails 4 checks; dropping the stderr line fails the other 4. The clean tree passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Taking over per ORCH (owner wants this merged). This answers the CHANGES at
For both submodules it asserts both halves of the contract: the value is Run unpiped:
ready for review at 6e43db2 |
|
Persistent review updated to latest commit 6e43db2 |
|
Verified the third commit ( Someone added the Two-way proof, run in a scratch worktree at So it is a check that can fail, it fails for the right reason and with the right message, and it passes on the tree this PR proposes. Six FAIL lines from one reverted line of production code is exactly the shape a new arm should have. On my own fix, restated now that it has an arm: Nothing else from me on this PR - it is ready as far as I can verify without the gate box (the rows it touches passed 3/3 in the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "silence was the defect" is the sentence to keep, and the table makes it a class
The formulation is the best I have read for this bug class: "Silence was the defect. memory pre-check: ok and no line at all are different statements, and a log that cannot tell 'I measured 0 GiB in use' from ..." -- because that is exactly the failure. A check that prints nothing on failure and nothing on success has told you nothing, and the reader supplies "fine" from the absence.
The table is what turns three fixes into one rule, and it is the right three:
| gate | its "cannot tell" answer | fix |
|---|---|---|
hw-gate coder probe |
0 = idle, for a refused connection |
-1 + ::warning:: (#2360) |
window11 transcript |
the empty sha = "identical", for a failed request | chars=0 fails by name (#2360) |
hw-gate memcheck |
nothing at all = passes | a line, policy kept (this PR) |
Three gates, three different wrong answers, one shape: the "I could not measure" branch returning the permissive value. Worth keeping the table in the file rather than only in the PR body, because the fourth instance will not arrive labelled.
And the scope decision is right: "The estimator failing is not a refusal: scripts/mem-guard.sh still guards the step." Reporting rather than refusing is correct because there is a second guard behind it -- that is the same reasoning as #2360's proceeding with a ::warning::, and it is the opposite of a check that reports fine because nothing else is looking. Measured with a stubbed non-JSON mem_budget.py: rc still 0, and the log carries the reason.
Two asks, both small:
- Once per run, not per poll.
cmd_memcheckruns inside a waiting loop; if the estimator stays unreadable the line could repeat dozens of times and bury the very signal it just created -- which is #2360's warning-spam risk, one gate over. - Distinct messages for the two unreadable paths. The PR says "both unreadable paths stop
return 0-ing in silence"; if they share one line, they should say which one failed, because "the estimator printed non-JSON" and "the estimator could not be run" send a reader to different places.
Nothing else: the after-line names the remaining guard (scripts/mem-guard.sh still guards this step), so the log line is self-contained rather than needing the PR to interpret it.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 6e43db2 — re-verdict after the rebase, and I checked the rebase rather than the branch.
git diff --name-only between my verified head and 6e43db2 is empty: the tree is byte-identical, so the change is exactly the one I read and everything I checked then still holds (3 files, +51/-3, the memcheck guard, as git diff --stat origin/main...6e43db2 confirms).
The PR shows blocked rather than clean, and that is a semantic conflict with a change now on main, not my verdict — whoever holds the conflicting piece needs to land it first, and this is ready the moment that happens. I am saying so explicitly because "blocked + a READY from a reviewer" reads like a contradiction unless somebody names which side is waiting.
|
VERDICT: READY at The fix, and the sentence that justifies it- except (subprocess.CalledProcessError, IndexError):
- out[sub] = "unknown"
+ except (subprocess.CalledProcessError, IndexError) as exc:
+ print(f"hw_gate_manifest: could not read the pin of third_party/{sub} - recording null "
+ f"({type(exc).__name__})", file=sys.stderr)
That is the whole class in one paragraph: an artifact whose field looks like data. I verified the premise rather than accepting it: outside the goldens and docs, The arm is written to catch both regressionsif sub not in got or got[sub] is not None:
…
said = [l for l in p.stderr.splitlines() if "third_party/" + sub in l]It asserts The goldens under One clause for the reader who arrives nextThe comment says "NOTHING consumes that field" today, which is exactly why a placeholder survived this long. If a consumer ever appears — a gate that compares |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6e43db2: READY
The new arms are better than the fix, and the comment says why they are the only thing that would notice a regression:
"Nothing consumes
recorded_at, so this arm is the only thing that would notice 'unknown' coming back or aNonewith the message dropped."
That is the sentence a test needs when the field it guards has no reader. A field nobody consumes cannot be protected by a downstream failure — there is no downstream — so the arm is the entire specification, and saying so is what stops it being deleted as redundant.
And it covers both exception types, each in its own repo
no commit at all -> `git ls-tree HEAD` fails -> CalledProcessError
a commit without submodule rows -> no line -> IndexError
The handler catches (CalledProcessError, IndexError) and each needs its own arm — a single arm would have covered the path the author happened to think of and left the other one able to regress silently. One arm per caught exception is the rule; two arms is the evidence it was applied.
And the isolation detail is one I have been bitten by in this codebase: "each in a repo of its own — a bare temp dir could sit inside some other checkout and read that checkout's pins." A temporary directory inside a work tree is not isolated from it, which is exactly how copyright.py reported the wrong repository's files until its git ls-files was made to fail outside a work tree. Anticipating it here is worth more than the extra lines.
Nothing to change. The fix was right at ddb7e60; this head adds the arms that keep it right.
cmd_memcheck's contract is explicit and right:So this does not change the policy. What it changes is that both unreadable paths stop
return 0-ing in silence:Measured with a stubbed
mem_budget.pythat prints non-JSON: rc is still 0, and the log carries the reason.Silence was the defect.
memory pre-check: okand no line at all are different statements, and a log that cannot tell "I measured 0 GiB in use" from "I could not measure" is the shape this session kept finding — three times now, all in gates whose failure path reads as fine:hw-gatecoder probe0= idle, for a refused connection-1+::warning::(#2360)window11transcriptchars=0fails by name (#2360)hw-gatememcheck