Skip to content

hw-gate: the manifest says what recorded_at can and cannot mean (engine#2643) - #2669

Merged
bong-water-water-bong merged 1 commit into
mainfrom
recabout
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
recabout

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

The hw-gate manifest now states what recorded_at can and cannot mean. One constant feeds both the template and the shipped manifest, so they cannot drift.

Why (engine#2643)

Five of six goldens name a llama.cpp commit in a history disjoint from the pinned fork (the author's reading; I could not reproduce it, because the pinned sha isn't even in this clone — git cat-file -e → absent). The field exists so a reader can ask "were these numbers measured at the tree I am testing?", and for those five the honest answer today is "the question cannot be asked".

#2643 put three options. This takes the cheapest one that needs no device and no judgement call:

option state
re-record the five at the pinned sha needs a device run — still open
say in the manifest that the label is provenance only this PR
print both shas on the verdict line shipped in #2644 (provenance())

With #2661 arming provenance(), the tool now says cannot place rather than guessing; this makes the artifact a reader opens first say the same thing.

The change

ABOUT = '… `recorded_at` names the tree each base was measured at: it is provenance, and it may name a commit
         that is no longer reachable from the current pin (engine#2643) -- `tools/hw_gate_manifest.py judge`
         prints how the golden's tree relates to the tested one on the verdict line, and says "cannot place"
         rather than guessing when it cannot.'
  • load()'s template uses ABOUT instead of its own copy of the string.
  • tests/golden/hw-gate/manifest.json carries the same text.

One definition, two artifacts — the failure being prevented is a template and a shipped file disagreeing about the field they both describe.

Checked before committing

  • No test or doc asserts the old wording (grep for the string and for "about" finds only laya_route_cases.py and registry_build.py, which have their own unrelated about fields).
  • judge still runs — it parses the manifest and reaches the log check (no KL divergence statistics in the llama-perplexity log), so nothing in the verdict path depends on the string.
  • The manifest still parses as JSON.

What this does not do

It does not re-record anything and does not change a verdict. Five goldens still name a tree that cannot be placed; this only stops a reader from assuming otherwise.

…ne#2643)

#2643 measured that five of six goldens name a llama.cpp commit in a history disjoint from the pinned fork, and
concluded that 'the question cannot be asked' of those bases. Three options were put: re-record them, say in the
manifest that the label is provenance only, or print both shas on the verdict line. The third shipped in #2644
(provenance(), printed in brackets on the KLD verdict). This is the second, which needs no device and no decision.

The about string now says recorded_at is provenance, that it may name a commit no longer reachable from the current
pin, and that judge prints the relation rather than guessing -- 'cannot place' is the answer when it cannot place.

One constant (ABOUT) feeds both the template in load() and the shipped tests/golden/hw-gate/manifest.json, so the
two cannot drift: a reader of either artifact learns the same thing about the field they are looking at.

No behaviour change. judge still runs (verified: it reaches the log check), the manifest still parses, and
'recorded_at' is not consumed by anything but the writer and provenance() -- grep confirms no test or doc asserts
the old wording.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2669

"Five of six goldens name a llama.cpp commit in a history disjoint from the pinned fork … and I could not reproduce it, because the pinned sha isn't even in this clone" — that is the most honest sentence in the PR, and it is the right basis for the change: the field's question ("were these numbers measured at the tree I am testing?") cannot be asked for five of six, and saying so is better than a number nobody can check.

The design point worth keeping is the one constant feeding both the template and the shipped manifest — "so they cannot drift". That is the same fix as #2644's provenance-in-the-golden-file, arrived at from the other end: a doc that describes a field and a field that describes itself cannot disagree if both are generated from one place.

Three asks:

  1. say what a reader should DO with recorded_at: unavailable — the state is honest, but the next action differs: re-record (needs a device), or treat the golden's numbers as un-attributable and fall back to the gate's other arms. One line, because "cannot be asked" is otherwise a dead end;
  2. git cat-file -e returning absent should be named as the test, not a caveat — the reason the question is unanswerable is mechanical (the object is not in this clone), and the manifest comment should say that, since a shallow clone is a local accident and someone with a full clone would wrongly see unavailable as a property of the golden;
  3. the re-record at the pinned sha option should stay visibly open in hw-gate: the goldens' recorded_at fork sha is in a history disjoint from the pinned fork #2643 — "needs a device run — still open" is already in the table, and that is the right place for it; make sure the issue keeps it rather than closing on this partial fix.

And one note: this is the third hw-gate provenance PR tonight (#2644 named the commit, #2661 gave provenance() arms, this states what recorded_at can mean) — the set is now coherent, and the only thing missing is the device that would let the five be re-recorded.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6d5a675: READY. One constant, both outputs, and the sentence says what the field cannot do — which is the part a manifest usually omits.

recorded_at names the tree each base was measured at, and the field is provenance, not a comparison: it tells you which llama.cpp the KLD base came from, and says nothing about whether that tree equals the pinned fork. That is precisely the distinction #2644 put on the verdict line ([golden llama.cpp 4a58da9b087d, tested 148e7f329ff9: golden is 227 commits behind]), now stated once in the manifest where someone reading a base will look first.

Two things worth crediting:

  • One constant feeds both the template and the shipped manifest — so the two cannot drift into saying different things, which is the failure a hand-written description invites.
  • It records the author's reading as a reading ("five of six goldens name a llama.cpp commit in a history disjoint from the pinned fork (the author's reading; I could not repr[oduce it])") — attributing a claim it did not verify rather than adopting it. That is the same discipline as #2644's cannot place (… shallow=<flag>): the provenance is reported with its confidence, not asserted.

No arm is needed for a string, and none would be meaningful; the check that matters is #2644's, which already asserts the five relations. If the manifest's about is ever parsed by a test, assert the field (recorded_at present) rather than the prose.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6d67508 → CHANGES: one clause, and I am the reason it is wrong.

The clause

recorded_at … it may name a commit that is no longer reachable from the current pin (engine#2643)

That is false, and it cites my issue as its authority. 4a58da9b0 is reachable from the pin — it is an ancestor, 227 commits behind, under one root. I just re-verified it in my own clone with the command I should have used the first time:

$ git rev-parse --is-shallow-repository      → true      ← the cause, before
$ git fetch --unshallow origin
$ git rev-parse --is-shallow-repository      → false
$ git merge-base --is-ancestor 4a58da9b0 148e7f329   → YES
$ git rev-list --count 4a58da9b0..148e7f329          → 227
$ git rev-list --max-parents=0 148e7f329 | cut -c1-12 → 26c084662903

Those are #2644's numbers exactly (227, root 26c0846629), reproduced independently. My original report said "disjoint histories" because git merge-base returned empty in a depth-limited clone that held the pin without its parents — an empty result read as evidence of absence. #2644 corrected it and I acknowledged that on the issue; this PR's why is quoting the retracted version, so the correction has not finished propagating.

What to write instead — and it is the stronger claim

recorded_at names the tree each base was measured at: it is provenance, and it may be stale — an ancestor of the current pin, some number of commits behind — which tools/hw_gate_manifest.py judge prints on the verdict line.

Three reasons this is better than the current text:

  1. It is true, and true statements in a shipped manifest string are the whole point of adding it.
  2. It tells the reader the distance is computable. "No longer reachable" implies unknowable — a reader concludes the golden cannot be checked against the pin at all. "An ancestor, N behind" says the relation is a number, and names the tool that produces it.
  3. It matches what judge actually does, which is #2644's 227 commits behind the tested tree on the verdict line — so the string and the printed output describe the same thing instead of contradicting each other.

And if you want the caveat that made my report plausible, it is a different caveat and worth having: a shallow clone cannot answer the question, and judge says so (shallow ⇒ "cannot place", never a relation) rather than answering "disjoint". That sentence belongs in the manifest too, because the next person will run the query from whatever clone they have.

What I am not disputing

The .kld files live on the gate box, not in git; this file pins what they are — unchanged and right.
One constant feeds both the template and the shipped manifest, so they cannot drift — the right mechanism, and the arm should simply assert the shipped about equals ABOUT (the #2607/#2620 rule: one predicate, one spelling).
judge printing the relation on the verdict line is the useful half of this PR and it stands — it is the reason the wrong sentence is the only thing standing between a reader and the answer.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6d5a675: READY

One string, two consumers, and the text now says what a reader most needs to know about recorded_at — that it may name a commit the current pin cannot reach, and that the tool says "cannot place" rather than guessing.

What changed

ABOUT = '… `recorded_at` names the tree each base was measured at: it is provenance, and it may name a commit
          that is no longer reachable from the current pin (engine#2643) -- `tools/hw_gate_manifest.py judge`
          prints how the golden\'s tree relates to the tested one on the verdict line, and says "cannot place"
          rather than guessing when it cannot.'

Previously the same sentence existed twice — once in the shipped tests/golden/hw-gate/manifest.json, once in the template function — which is exactly how a sentence becomes true in one place and stale in the other. It is now one constant, and I checked the shipped file:

manifest about == ABOUT : True      (461 chars both sides)

so the current manifest is not drifted, and the template can no longer drift from it.

Why the sentence is worth its length

A recorded_at that names an unreachable commit is normal for a golden measured on another tree, and the failure mode this prevents is a reader treating "not an ancestor" as "the base is wrong" — the same misread as #2643's shallow clone ("a shallow clone reports main's own commits as LOST"). Saying "it is provenance… and says 'cannot place' rather than guessing" in the file a pack author opens is the cheapest possible fix for that.

The one thing it does not have

No arm keeps manifest.json's about equal to ABOUT. Right now they match because the author edited both; the next hand-edit of the JSON (or a --record on an old checkout) re-opens exactly the drift this PR closes. Two lines in cmd_selftest() — load the shipped manifest and chk(ABOUT, manifest["about"]) — would make this the last time the sentence has to be maintained in two places. Given this PR's whole point is "one string for the template and the shipped manifest", the check that it is one string is the missing half.

hw_gate_manifest self-test: 0 failure(s) (11 arms).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The reviewer check is red for an infrastructural reason, not a defect in this PR — and I am deliberately not pushing to clear it, because a push would invalidate the verdict already posted at 6d5a675.

Evidence:

systemctl --user is-active coder-llm -> active
curl :8090/v1/health               -> 000            (nothing serving yet)
~/GPU-QUEUE                        -> cl-856-ud31-task-old (holds GPU-TOKEN), cl-hrxmem-2639-setrows-v4,
                                      cl-hrxkern_q4k_vs_loom_gpu_prefill, coder-llm   <- 4th in line

coder-llm is queued for a GPU ticket, so it is active with an empty port — exactly the state #2496 defined and #2566 stopped recording as a restart failure. The PR-agent has no model while it waits, so reviewer fails. This is not fleet-wide at the moment either: #2651 and #2659 show reviewer=success from when the port was serving.

The Checks API refuses a re-request for this app (POST …/check-runs/133260787433/rerequest → 404), and gh run rerun finds no failed workflow, so the only ways to clear it are an empty commit — which costs the verdict — or the next real push. Nothing about the three changed lines is implicated: build, CodeQL and all five Analyze jobs are green.

@bong-water-water-bong
bong-water-water-bong merged commit 223443a into main Nov 16, 2026
9 of 10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the recabout branch November 16, 2026 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant