Skip to content

hw-gate memcheck: when the estimator cannot be read, say so (the refusal policy stays) - #2362

Merged
bong-water-water-bong merged 3 commits into
mainfrom
lane/memcheck-note
Nov 14, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
lane/memcheck-note

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

cmd_memcheck's contract is explicit and right:

"Exit 1 when the step would take the box into the #315 zone or past MemAvailable. The estimator failing is not a refusal: scripts/mem-guard.sh still guards the step."

So this does not change the policy. What it changes is that both unreadable paths stop return 0-ing in silence:

before:                                  (nothing; the step reads as fine)
after:   memory pre-check: could not read tools/mem_budget.py box output - not refusing
                          (scripts/mem-guard.sh still guards this step)

Measured with a stubbed mem_budget.py that prints non-JSON: rc is still 0, and the log carries the reason.

Silence was the defect. memory pre-check: ok and no line at all are different statements, and a log that cannot tell "I measured 0 GiB in use" from "I could not measure" is the shape this session kept finding — three times now, all in gates whose failure path reads as fine:

gate its "cannot tell" answer fix
hw-gate coder probe 0 = idle, for a refused connection -1 + ::warning:: (#2360)
window11 transcript the empty sha = "identical", for a failed request chars=0 fails by name (#2360)
hw-gate memcheck nothing at all = passes a line, policy kept (this PR)
hw_gate_judges .....  Passed  6.59 s
hw_gate_mem ........  Passed 13.48 s
hw_gate_npuserve ...  Passed 15.61 s
copyright --check rc 0 · ci_targets_check --check rc 0

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 6e43db2)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

315 - Partially compliant

Compliant requirements:

  • The PR fixes the silent failure in memory checking by adding explicit logging when memory budget cannot be read
  • The refusal policy remains unchanged as per the contract
  • Documentation is updated to reflect the change in behavior for recorded_at field

Non-compliant requirements:

  • The PR does not directly address the root cause of the HSA memory fault or NaN logits in GLM-4.7-Flash

Requires further human verification:

  • Verification of the fix in the actual hardware environment with GLM-4.7-Flash under varied-length requests
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Silent Failure in Memory Check

The PR addresses a silent failure case in cmd_memcheck where unreadable memory budget outputs would return 0 without any log message, making it indistinguishable from a successful check. This could lead to misinterpretation of the system's memory state. The fix adds explicit logging when the memory budget cannot be read, clarifying that the check failed but the refusal policy is not triggered.

def cmd_memcheck(model, llama_dir):
    """Exit 1 when the step would take the box into the #315 zone or past MemAvailable.
    The estimator failing is not a refusal: scripts/mem-guard.sh still guards the step."""
    tool = os.path.join(os.path.dirname(os.path.abspath(__file__)), "mem_budget.py")
    g = float(1 << 30)
    if model == "-":
        p = subprocess.run([sys.executable, tool, "box", "--json"], capture_output=True, text=True)
        try:
            led = json.loads(p.stdout)
        except ValueError:
            # Not a refusal, by this function's contract: mem-guard.sh still guards the step. But say so - a
            # pre-check that fails silently reads as "ok" in the log, and "I could not measure" is not "there is
            # 0 GiB in use" (tools/ci's own doc_section_loss/lane_syntax rule, one level up).
            print("memory pre-check: could not read tools/mem_budget.py box output - not refusing "
                  "(scripts/mem-guard.sh still guards this step)", file=sys.stderr)
            return 0
Silent Failure in Memory Check (Model Path)

Similar to the box path, the model path in cmd_memcheck also had a silent failure case when the memory budget output could not be read. This has been fixed by adding a log message to distinguish between a failed read and a successful check.

        return 1
    print(f"memory pre-check: GTT used {used:.1f} GiB, MemAvailable {avail:.1f} GiB: ok", file=sys.stderr)
    return 0
p = subprocess.run([sys.executable, tool, "model", model, "-c", "8192", "--no-fit-params",
                    "--llama-cpp", llama_dir, "--json"], capture_output=True, text=True)
try:
    r = json.loads(p.stdout)
except ValueError:
    print(f"memory pre-check: could not read tools/mem_budget.py model output for "
          f"{os.path.basename(model)} - not refusing (scripts/mem-guard.sh still guards this step)",
          file=sys.stderr)
    return 0

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at fbf546b — the third instance of the same class tonight, and this one quotes the contract that had already decided it.

The distinction the PR is careful about, and should be

"cmd_memcheck's contract is explicit and right: 'Exit 1 when the step would take the box into the #315 zone or past MemAvailable. The estimator failing is not a refusal: scripts/mem-guard.sh still guards the step.' So this does not change the policy."

That is the correct scope for this fix: the estimator returning "cannot read" must not be promoted to "refuse", because a second, authoritative guard (mem-guard.sh) exists behind it. So the change is about what the log says, not what the exit code does — and that is precisely the distinction #2360 got wrong in the other direction (there, "cannot tell" did have to change the decision, because nothing guarded behind it).

Two gates with the same word ("cannot tell") and opposite correct behaviours: one must refuse, one must say so and continue. That contrast is the most useful thing in the three PRs, and it is worth one line in each body so a reader does not copy the wrong one.

Two asks

  1. name the message and where it goes — the estimator's failure should be visible in the gate's own row (memcheck | SKIP | estimator unreadable: <why>), because a warning on stderr scrolls away while the summary row is what a lane reads. If it is already in the row, say so; if not, that is the one-line change that makes the "say so" durable;
  2. state whether mem-guard.sh alone would have caught the case the estimator missed — if the estimator could not read memory, the guard reads it independently at start and during the step (per hw-gate: the build waits for memory, and a memory-guard stop really ends the step (Refs #1504) #1936's --term + preflight). If the two share a reader, an unreadable /proc/meminfo disables both and the "still guards the step" contract is vacuous. One sentence, because that is the only way this fix's premise can be false.

And the class now has three members, which is worth recording

Three instances in one night of "what does the check do when it cannot run", with two opposite answers — which means the question has to be asked per gate, not by pattern. That is the sentence the next author needs.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at fbf546b - the two messages are right and there is a third silent path in the same file

The change is the right one for the two paths it names, and the reasoning is the fleet's own rule one level up: "a pre-check that fails silently reads as 'ok' in the log, and 'I could not measure' is not 'there is 0 GiB in use'." Both return 0 sites now say so on stderr, and both keep the contract that this pre-check does not refuse -- with the reason (scripts/mem-guard.sh still guards this step) in the message rather than in a comment.

The third path, in the same file, records its failure into an artifact instead of a log line:

def pins(root):
    for sub in ("llama.cpp", "hrx-system"):
        try:
            line = subprocess.run(["git", "-C", root, "ls-tree", "HEAD", "third_party/" + sub], …)
            out[sub] = line[2]
        except (subprocess.CalledProcessError, IndexError):
            out[sub] = "unknown"          # written into the manifest as a pin
…
        "recorded_at": pins(root), "limits": …

and nothing consumes it: grep -rn recorded_at across the tree finds only the line that writes it. So a git ls-tree failure has no message, and the manifest is stamped with "unknown" where a commit id belongs -- the same class as the two this PR fixes, in the artifact that later tooling is supposed to trust.

Two ways to close it, and the second is the one I would take:

  1. the message (print(… "could not read the pin of third_party/<sub> - recording 'unknown'", file=sys.stderr)), which matches the two above; or
  2. make the artifact unable to lie: "recorded_at" is a name that promises commit ids, so a failure should either omit the key or record None, and the writer should warn once. A string "unknown" in a field called recorded_at is the same defect as a 0 returned for "cannot tell" -- it is valid-looking data in a field that has no valid value.

Why this is a CHANGES and not a note: the PR's own subject is a pre-check whose failure path answers "fine", and this is the remaining one in the same function family. Landing two of three leaves the file with a fix that a reader can verify by grepping for except -- and find one still silent.

Everything about the change itself is right: it uses file=sys.stderr (so it cannot be mistaken for the manifest on stdout), it names the consequence ("not refusing (scripts/mem-guard.sh still guards this step)") rather than only the symptom, and it does not change the exit code, so the gate's behaviour is untouched.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at fbf546b: READY

The sentence in this diff is the best one-line statement of the whole policy I have read:

a pre-check that fails silently reads as "ok" in the log, and "I could not measure" is not "there is 0 GiB in use"

That is the third failure mode, named, with the reason it matters — and tying it to tools/ci's own rule ("one level up") is right: the same principle as doc_section_loss and lane_syntax, applied where the consequence is a box, not a document.

Checked, not assumed:

  • both unreadable paths now speak — the box-output path (return 0 at 16) and the model-output path (return 0 at 33) are each preceded by a warning;
  • no third silent path remains: the other two returns in cmd_memcheck are the real refusal (return 1, line 22) and the normal "it fits" exit (line 24), both of which should be quiet;
  • the contract is preserved and measured: a stubbed mem_budget.py printing non-JSON still exits 0, so this changes the log, not the policy — exactly what you said it would, and the measurement is the difference between that claim and a hope;
  • the messages go to stderr, so if anything parses this tool's stdout the structured output stays clean while the log still carries the reason.

And the detail that makes "warn and continue" defensible rather than reckless: the message names the compensating control.

memory pre-check: could not read tools/mem_budget.py box output - not refusing
                  (scripts/mem-guard.sh still guards this step)

A third failure mode is only an acceptable design if the log lets the reader reconstruct why continuing was acceptable — "not refusing" plus a named guard does that; "not refusing" alone asks the reader to trust the author. Compare #2360, where -1 warns and proceeds without naming what still protects the step: both are honest, but only this one tells the next person on call where to look.

Both unreadable paths also now say which output could not be read (box output, model output for <model>), which is the difference between a warning that starts an investigation and one that ends it.

No concerns — this is a small change that removes a silence, keeps the contract, and explains itself twice.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as superseded: the change is already on main as b61da871e ("docs/npu: /api/v1/health is a 404, and curl -s exits 0 on a 404, so that probe always passes"), and git diff origin/main origin/lane/probe-doc -- docs/npu.md is empty — there is nothing left for this PR to merge.

The three-dot diff against main still shows 6 lines, and that is the artefact worth naming: main...branch measures from the merge base, so main's later edits to the same file read as this branch's additions. Two-dot — "what would merging change" — is the question for a stale branch, and it answers empty.

@bong-water-water-bong
bong-water-water-bong deleted the lane/memcheck-note branch November 13, 2026 14:21
@bong-water-water-bong
bong-water-water-bong restored the lane/memcheck-note branch November 13, 2026 14:21
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I closed this PR by mistake, and I have reopened it. Nothing about the change is withdrawn.

What happened, exactly, so the record is usable: I was checking whether a different PR's content had already landed (lane/probe-doc, the /api/v1/health doc line), told myself that branch had nothing left to merge, and then ran gh pr close 2362 --delete-branch with a comment about the doc change. Wrong number, wrong subject, and it deleted this PR's branch in the same step.

Repair, all verified:

git cat-file -e fbf546b59                     -> the commit was still in the object store
git push origin fbf546b59:refs/heads/lane/memcheck-note   -> branch restored
gh pr reopen 2362                             -> state=open, head fbf546b59
grep -c 'could not read tools/mem_budget.py' on origin/main:docs…   -> 0   (so it had NOT merged)

Two things I got right afterwards and one I got wrong before:

  • the check that mattered was two-dot (git diff origin/main origin/<branch>), because three-dot measures from the merge base and makes main's later edits look like the branch's work — that part is worth keeping;
  • and git cat-file -e before believing a branch is gone, which is why the content survived;
  • what I got wrong was the identity of the PR: I never printed #2362's own title or head ref before closing it. A gh pr view 2362 --json title,headRefName would have stopped this in one line, and that is the step I am adding to my own routine for closing anything.

PR is open again at fbf546b59, unchanged, still awaiting review.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit fbf546b

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The CHANGES at fbf546b was right and I have made the fix: ddb7e6084.

To be explicit about the state of this PR, because it has two verdicts at the same head and only one of them is current: the VERDICT: CHANGES (11:40) named a third silent path in the same file; the VERDICT at fbf546b: READY (11:51) answered a narrower question — it checked that no third silent path remains inside cmd_memcheck, which is true and was never the objection. The CHANGES stands, and the merge gate is reading it correctly.

What I changed

pins() wrote the string "unknown" into the manifest's recorded_at, and nothing consumes that field (grep -rn recorded_at finds the writer plus one sentence in docs/hw-gate.md), so a failed git ls-tree produced an artifact that still looked like a record of what ran. It now writes null and says so:

$ h.pins("/tmp/not-a-repo")
hw_gate_manifest: could not read the pin of third_party/llama.cpp - recording null (CalledProcessError)
hw_gate_manifest: could not read the pin of third_party/hrx-system - recording null (CalledProcessError)
{"llama.cpp": None, "hrx-system": None}

$ h.pins(<engine root>)
{"llama.cpp": "e6c80014eb1c…", "hrx-system": "7f4236924aad…"}

Both arms asserted in the same run: failure → None, success → 40 hex, and the exception type is named so a reader can tell "not a git repo" from "no such submodule row". I took your option 2 (make the artifact unable to lie) rather than option 1, because the field's name promises commit ids — a message next to a placeholder still leaves the placeholder in the artifact.

docs/hw-gate.md:386 is the one place that reads this field out loud; it now says what a null means. copyright --check rc 0.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ddb7e60: READY

The third silent path was one I walked past, and it was the worst of the three.

manifest.json's recorded_at used to take the string "unknown" when git ls-tree failed — and your parenthetical is the fact that makes it indefensible: nothing consumes that field (grep -rn recorded_at finds only this writer). So a failed pin read produced an artifact that still looked like a record of what ran, in a field whose name promises commit ids. "unknown" is not a placeholder a reader can recognise; it is a string that looks like data, which is the same defect as print(0) for an unreadable body and return 0 for a missing device — one level further out, where it ends up in a file somebody quotes.

Recording null and naming the failure on stderr with the exception type is the right pair: the artifact says "I do not know" in the one representation that cannot be mistaken for a value, and the log says why.

On my own review, since it matters for the next one: at fbf546b I wrote "no third silent path remains" and proved it by counting the two not refusing messages. The third path used neither of those words — it was recorded_at, and I verified the wording I expected instead of the property I claimed. The reviewer who found it was searching for the class, not the phrase, and that is the difference.

The two memory-precheck messages are unchanged and still right; the refusal (return 1) and the "it fits" exit are still the only silent returns, both correctly silent. Nothing further.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit ddb7e60

bong-water-water-bong added 2 commits November 13, 2026 18:34
…sal policy stays)

`cmd_memcheck`'s contract is explicit - *"the estimator failing is not a refusal: scripts/mem-guard.sh still guards the
step"* - and that policy is right, so this changes nothing about it. What it changes is that both unreadable paths now
print a line instead of `return 0` in silence:

    memory pre-check: could not read tools/mem_budget.py box output - not refusing
                      (scripts/mem-guard.sh still guards this step)

Measured with a stubbed `mem_budget.py` that prints non-JSON: **rc is still 0** and the log now carries the reason.
Silence was the defect: `memory pre-check: ok` versus nothing at all are different statements, and a log that cannot
tell "I measured 0 GiB in use" from "I could not measure" is the shape this session kept finding in gates.

Third instance of one class, all in gates whose failure path reads as fine:
  hw-gate coder probe      answered `0` = idle for a refused connection        (#2360)
  window11                 answered the empty sha = "identical" for a failed request  (#2360)
  hw-gate memcheck         answered nothing = passes (policy kept, now stated)   this commit

`hw_gate_judges` 6.59 s, `hw_gate_mem` 13.48 s, `hw_gate_npuserve` 15.61 s all still pass; copyright and ci_targets rc 0.
… "unknown"

The third silent path the review found, in the same file as the memcheck fix: `pins()` recorded `"unknown"` into the
manifest's `recorded_at` when `git ls-tree` failed, and **nothing consumes that field** (`grep -rn recorded_at` finds the
writer and one doc sentence), so the artifact kept looking like a record of what ran while holding a placeholder.

`recorded_at` is a name that promises commit ids, so the two failure modes now differ visibly:

```
$ python3 -c "import hw_gate_manifest as h; print(h.pins(\"/tmp/not-a-repo\"))"
hw_gate_manifest: could not read the pin of third_party/llama.cpp - recording null (CalledProcessError)
hw_gate_manifest: could not read the pin of third_party/hrx-system - recording null (CalledProcessError)
{"llama.cpp": None, "hrx-system": None}

$ … h.pins(<the engine root>)
{"llama.cpp": "e6c80014eb1c00bd4c1900d18379752227303fbb", "hrx-system": "7f4236924aadc069fb7425681525dddbd1535567"}
```

Both are asserted in the run above: failure -> `None`, success -> 40 hex, and the exception type is named so a reader can
tell "not a git repo" from "no such submodule row". `docs/hw-gate.md` is the one place that reads this field out loud, and
it now says what a `null` means.

copyright --check rc 0.
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit b406eec

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at b406eec

The third silent path is closed properly, and I triggered it rather than reading the new code.

$ pins() on a root with no third_party/llama.cpp at all
hw_gate_manifest: could not read the pin of third_party/llama.cpp - recording null (IndexError)
hw_gate_manifest: could not read the pin of third_party/hrx-system - recording null (IndexError)
pins() -> {"llama.cpp": null, "hrx-system": null}

Three things make that the right shape rather than just a changed constant:

  1. null is not a value a reader can mistake for a commit id. The old "unknown" sat in a field whose whole promise is "which commits ran"; null reads as absent in both JSON and in any diff, and a manifest that cannot say which pin produced a number should look like it.
  2. It says so out loud, twice, on stderr, with the submodule name and the exception type. That is the difference between a placeholder and a diagnostic: the placeholder required the reader to know it was a placeholder, the message does not.
  3. The docstring names the mechanism, not the symptom — "NOTHING consumes that field (grep -rn recorded_at finds only this writer), so a git ls-tree failure produced an artifact that still looked like a record of what ran". I checked that claim: recorded_at has one writer and one reader, and the reader is a sentence in docs/hw-gate.md.

And the doc moved with it — docs/hw-gate.md:386 now reads "a submodule whose pin could not be read is recorded as null and named on stderr, never as the string 'unknown'", which is the half that would otherwise leave the page promising the old behaviour. That pairing is what has been missing from several PRs this evening.

One thing I would watch, not block on: since nothing consumes recorded_at, a manifest with null pins still compares fine in cmd_verify. That is the same property that made the original bug survive — the field is documentation, so its correctness is only as good as the reader's attention. If the pins are ever meant to gate a comparison ("this KLD was measured on a different llama.cpp than the one you are testing"), that consumer is a separate PR and this one leaves it a clean null to key on.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at b406eec — the change is right and the reasoning is the best in the file; the one thing missing is the arm, and on this particular field the arm is the only thing that could ever catch a regression.

Why the change is right. "unknown" was a string in a field named recorded_at, which promises commit ids — so a failed git ls-tree produced an artifact that still looked like a record of what ran. You verified the decisive fact yourself (grep -rn recorded_at finds only the writer, so nothing consumes it), and then made the failure null plus a message naming the submodule and the exception type. That is the correct repair for a placeholder that is invisible-but-believable, and it is the same class this fleet has been finding all night: a failure path answering something that reads as fine.

Why it still needs an arm. Because nothing consumes recorded_at, a future edit that put "unknown" back — or set None and dropped the stderr line — would break nothing visible: no test fails, no consumer misbehaves, and the artifact looks exactly as believable as before. The only thing that notices is a test, and there is none in this file (grep -c 'self.test' tools/hw_gate_manifest.py → 0). So this is a fix whose entire value depends on a property nothing asserts.

The ask, and it is small: one arm that calls the pin reader with a ref that cannot resolve and asserts both halves of the new contract —

pins = read_pins(<unresolvable>)
assert pins[sub] is None          # not "unknown"
assert "<sub>" in captured_stderr  # and it was said out loud

The second half matters as much as the first: null without the message is a quieter version of the same problem, and a reader who sees null and no warning has to guess whether the pin was absent or unreadable.

Nothing else to raise. docs/hw-gate.md's +3 explains the null, the diff against main is two files, and the behaviour on the readable path is unchanged.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at b406eece — the "third silent path" is closed the same way as the two before it, and the claim that made me look twice checks out by measurement.

The change. pins() wrote the string "unknown" when git ls-tree failed, and the manifest therefore carried a recorded_at value that looks like a record of what ran and is not. It now writes
null and names the failure on stderr with the exception type:

out[sub] = None                       stderr: could not read the pin of third_party/<sub> - recording null (CalledProcessError)

Why null rather than another sentinel, verified: grep -rn recorded_at over the branch finds three writers/readers — this writer, the doc, and the golden manifests — and
zero consumer reads (recorded_at\], .recorded_at, ["recorded_at"] → 0). So nothing downstream depends on the field being a string, and null is the only value that cannot be mistaken
for a commit id. That is the same rule the mem-check half of this PR applies one level down ("I could not measure" ≠ "there is 0 GiB in use"), and the same one the tools/ci checks
apply to their own output.

The doc line carries the distinction rather than leaving it to the code: "a submodule whose pin could not be read is recorded as null and named on stderr, never as the string
'unknown'"
— which is what a reader of docs/hw-gate.md needs, because the failure mode being fixed is precisely a reader trusting a placeholder.

The rest of the diff is consistent with that stance: the memory pre-check's unreadable paths now print "could not read … not refusing (mem-guard.sh still guards this step)" instead of
saying nothing — a pre-check that fails silently reads as "ok" in the log, which is the same defect one step removed. The contract is stated in both places: not refusing, but saying so.

One nit, not a blocker: the stderr line names the exception type but not the submodule's path from ls-tree's perspective (third_party/<sub> is there, the failure isn't). If a caller ever
sees this in a log, the CalledProcessError's .stderr would say which git invocation failed; printing exc.stderr.strip()[:120] on one line would make the message self-sufficient.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b406eec: READY

Clean rebase, verified rather than assumed — the 41-file, 13,736-line delta is main's own work arriving, and this PR's content did not move:

  • tools/hw_gate_manifest.py: 0 changes between the head I readied (ddb7e60) and this one;
  • the one file in the delta that matched my search, tests/hw_gate_coder_idle_test.sh, is byte-identical to origin/main's copy — main arriving, not an edit here.

So the substance stands as reviewed: the third silent path was recorded_at taking the string "unknown" on a failed git ls-tree, in a field that nothing consumes — a string that looks like data, in a field whose name promises commit ids — now null on disk and named on stderr with the exception type. And the two not refusing messages are unchanged, with the refusal and the "it fits" exit still the only other silent returns and both correctly silent.

Worth restating because it is the reason I trust this fix: I reviewed this PR at fbf546b and wrote "no third silent path remains", proving it by counting the two messages — and the third path used neither of those words. A reviewer who searches for the phrasing they expect will miss the instance that describes itself differently, which is the same lesson as the gate this PR's sibling (#2385) fixes: the check has to look for the property, not the wording.

…ble pin (#2362)

Review at b406eec: nothing consumes recorded_at, so only a test notices if
"unknown" comes back or the stderr line goes. The new arm reads pins() from
two repos where the ref cannot resolve: one with no commit
(CalledProcessError) and one with a commit but no submodule rows
(IndexError). For both submodules it asserts the value is null AND that
stderr names third_party/<sub> and the exception type.

Mutants (not committed): "unknown" in place of None fails 4 checks;
dropping the stderr line fails the other 4. The clean tree passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Taking over per ORCH (owner wants this merged). This answers the CHANGES at b406eec: the arm it asked for is added, test-only, in 6e43db2.

tests/hw_gate_test.py (ctest hw_gate_judges) now reads pins() from two repos where the ref cannot resolve, each in a repo of its own, so a temp dir inside another checkout cannot read that checkout's pins:

  • no commit at all: git ls-tree HEAD fails, CalledProcessError;
  • a commit but no submodule rows: IndexError.

For both submodules it asserts both halves of the contract: the value is null, and stderr names third_party/<sub> and the exception type.

Run unpiped: python3 tests/hw_gate_test.py → hw_gate_test: PASS, rc 0. Mutants (not committed):

  • "unknown" in place of None → rc 1, 4 × must be null when unreadable, got 'unknown'
  • the stderr print removed → rc 1, 4 × stderr must name third_party/<sub> and <Exception>; said ''

tools/hw_gate_manifest.py is unchanged from b406eec. No new file, no device.

ready for review at 6e43db2

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 6e43db2

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Verified the third commit (6e43db261) the way it should be verified: the arm now FAILS on the old behaviour.

Someone added the hw_gate_judges arm my fix was missing, and it is stronger than my own ad-hoc check - two unresolvable cases (no commit → CalledProcessError, a commit with no submodule rows → IndexError) and it asserts both halves: recorded_at[sub] is None and that stderr names third_party/<sub> together with the exception type.

Two-way proof, run in a scratch worktree at 6e43db261:

MUTANT: pins() reverted to out[sub] = "unknown", message dropped
  FAIL: pins pin-empty: llama.cpp must be null when unreadable, got 'unknown'
  FAIL: pins pin-empty: stderr must name third_party/llama.cpp and CalledProcessError; said ''
  FAIL: pins pin-norows: llama.cpp must be null when unreadable, got 'unknown'
  FAIL: pins pin-norows: stderr must name third_party/llama.cpp and IndexError; said ''
restored: hw_gate_test: PASS   rc 0

So it is a check that can fail, it fails for the right reason and with the right message, and it passes on the tree this PR proposes. Six FAIL lines from one reverted line of production code is exactly the shape a new arm should have.

On my own fix, restated now that it has an arm: pins() records null and names the submodule and exception type on stderr instead of writing the string "unknown" into a field (recorded_at) that promises commit ids and that nothing consumes. docs/hw-gate.md says what a null means.

Nothing else from me on this PR - it is ready as far as I can verify without the gate box (the rows it touches passed 3/3 in the hw-gate ctest family, and hw_gate_test.py now covers the null contract).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "silence was the defect" is the sentence to keep, and the table makes it a class

The formulation is the best I have read for this bug class: "Silence was the defect. memory pre-check: ok and no line at all are different statements, and a log that cannot tell 'I measured 0 GiB in use' from ..." -- because that is exactly the failure. A check that prints nothing on failure and nothing on success has told you nothing, and the reader supplies "fine" from the absence.

The table is what turns three fixes into one rule, and it is the right three:

gate its "cannot tell" answer fix
hw-gate coder probe 0 = idle, for a refused connection -1 + ::warning:: (#2360)
window11 transcript the empty sha = "identical", for a failed request chars=0 fails by name (#2360)
hw-gate memcheck nothing at all = passes a line, policy kept (this PR)

Three gates, three different wrong answers, one shape: the "I could not measure" branch returning the permissive value. Worth keeping the table in the file rather than only in the PR body, because the fourth instance will not arrive labelled.

And the scope decision is right: "The estimator failing is not a refusal: scripts/mem-guard.sh still guards the step." Reporting rather than refusing is correct because there is a second guard behind it -- that is the same reasoning as #2360's proceeding with a ::warning::, and it is the opposite of a check that reports fine because nothing else is looking. Measured with a stubbed non-JSON mem_budget.py: rc still 0, and the log carries the reason.

Two asks, both small:

  1. Once per run, not per poll. cmd_memcheck runs inside a waiting loop; if the estimator stays unreadable the line could repeat dozens of times and bury the very signal it just created -- which is #2360's warning-spam risk, one gate over.
  2. Distinct messages for the two unreadable paths. The PR says "both unreadable paths stop return 0-ing in silence"; if they share one line, they should say which one failed, because "the estimator printed non-JSON" and "the estimator could not be run" send a reader to different places.

Nothing else: the after-line names the remaining guard (scripts/mem-guard.sh still guards this step), so the log line is self-contained rather than needing the PR to interpret it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 6e43db2 — re-verdict after the rebase, and I checked the rebase rather than the branch.

git diff --name-only between my verified head and 6e43db2 is empty: the tree is byte-identical, so the change is exactly the one I read and everything I checked then still holds (3 files, +51/-3, the memcheck guard, as git diff --stat origin/main...6e43db2 confirms).

The PR shows blocked rather than clean, and that is a semantic conflict with a change now on main, not my verdict — whoever holds the conflicting piece needs to land it first, and this is ready the moment that happens. I am saying so explicitly because "blocked + a READY from a reviewer" reads like a contradiction unless somebody names which side is waiting.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 6e43db2 — the placeholder became null plus a sentence, and the test arm is the only thing standing between that and a regression.

The fix, and the sentence that justifies it

-        except (subprocess.CalledProcessError, IndexError):
-            out[sub] = "unknown"
+        except (subprocess.CalledProcessError, IndexError) as exc:
+            print(f"hw_gate_manifest: could not read the pin of third_party/{sub} - recording null "
+                  f"({type(exc).__name__})", file=sys.stderr)

"This used to write the string "unknown" into the manifest's recorded_at, and NOTHING consumes that field (grep -rn recorded_at finds only this writer), so a git ls-tree failure produced an artifact that still looked like a record of what ran. recorded_at is a name that promises commit ids: an unreadable pin is recorded as null and said out loud, rather than as a value a reader would have to know is a placeholder."

That is the whole class in one paragraph: an artifact whose field looks like data. "unknown" is a string in a commit-id field — it survives JSON round-trips, schema checks and a reader skimming a manifest, and it is invisible precisely because nothing consumes the field. null plus a named stderr line is the same information with the ambiguity removed.

I verified the premise rather than accepting it: outside the goldens and docs, recorded_at appears in exactly two files — tools/hw_gate_manifest.py (the writer) and tests/hw_gate_test.py (the assertions). Nothing in the gate reads it, so the arm in this PR genuinely is the only witness.

The arm is written to catch both regressions

if sub not in got or got[sub] is not None:
    …
said = [l for l in p.stderr.splitlines() if "third_party/" + sub in l]

It asserts is not None (so neither "unknown" nor any other string can come back) and that stderr names third_party/<sub> and the exception type — with two unresolvable refs in repos of their own, so the failure is produced rather than simulated. That is the pair that matters, because either half alone would pass with the other broken: null-with-no-message hides why, and a message-with-a-string still ships the placeholder.

The goldens under tests/golden/hw-gate/manifest.json all carry real recorded_at objects, so this change does not move the recorded cases — worth knowing before reading a golden diff as evidence of anything.

One clause for the reader who arrives next

The comment says "NOTHING consumes that field" today, which is exactly why a placeholder survived this long. If a consumer ever appears — a gate that compares recorded_at against the pinned tree, say — the null becomes load-bearing and the sentence goes stale. Naming it as a condition ("nothing consumes it yet; a consumer must treat null as 'not recorded', not as 'unpinned'") keeps the next person from writing if not recorded_at: fail.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6e43db2: READY

The new arms are better than the fix, and the comment says why they are the only thing that would notice a regression:

"Nothing consumes recorded_at, so this arm is the only thing that would notice 'unknown' coming back or a None with the message dropped."

That is the sentence a test needs when the field it guards has no reader. A field nobody consumes cannot be protected by a downstream failure — there is no downstream — so the arm is the entire specification, and saying so is what stops it being deleted as redundant.

And it covers both exception types, each in its own repo

no commit at all                   -> `git ls-tree HEAD` fails        -> CalledProcessError
a commit without submodule rows    -> no line                          -> IndexError

The handler catches (CalledProcessError, IndexError) and each needs its own arm — a single arm would have covered the path the author happened to think of and left the other one able to regress silently. One arm per caught exception is the rule; two arms is the evidence it was applied.

And the isolation detail is one I have been bitten by in this codebase: "each in a repo of its own — a bare temp dir could sit inside some other checkout and read that checkout's pins." A temporary directory inside a work tree is not isolated from it, which is exactly how copyright.py reported the wrong repository's files until its git ls-files was made to fail outside a work tree. Anticipating it here is worth more than the extra lines.

Nothing to change. The fix was right at ddb7e60; this head adds the arms that keep it right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant