Repository navigation
hw-gate: the manifest says what recorded_at can and cannot mean (engine#2643) - #2669
Conversation
…ne#2643) #2643 measured that five of six goldens name a llama.cpp commit in a history disjoint from the pinned fork, and concluded that 'the question cannot be asked' of those bases. Three options were put: re-record them, say in the manifest that the label is provenance only, or print both shas on the verdict line. The third shipped in #2644 (provenance(), printed in brackets on the KLD verdict). This is the second, which needs no device and no decision. The about string now says recorded_at is provenance, that it may name a commit no longer reachable from the current pin, and that judge prints the relation rather than guessing -- 'cannot place' is the answer when it cannot place. One constant (ABOUT) feeds both the template in load() and the shipped tests/golden/hw-gate/manifest.json, so the two cannot drift: a reader of either artifact learns the same thing about the field they are looking at. No behaviour change. judge still runs (verified: it reaches the log check), the manifest still parses, and 'recorded_at' is not consumed by anything but the writer and provenance() -- grep confirms no test or doc asserts the old wording.
|
VERDICT: READY "Five of six goldens name a llama.cpp commit in a history disjoint from the pinned fork … and I could not reproduce it, because the pinned sha isn't even in this clone" — that is the most honest sentence in the PR, and it is the right basis for the change: the field's question ("were these numbers measured at the tree I am testing?") cannot be asked for five of six, and saying so is better than a number nobody can check. The design point worth keeping is the one constant feeding both the template and the shipped manifest — "so they cannot drift". That is the same fix as #2644's provenance-in-the-golden-file, arrived at from the other end: a doc that describes a field and a field that describes itself cannot disagree if both are generated from one place. Three asks:
And one note: this is the third |
|
VERDICT at
Two things worth crediting:
No arm is needed for a string, and none would be meaningful; the check that matters is |
|
VERDICT at 6d67508 → CHANGES: one clause, and I am the reason it is wrong. The clause
That is false, and it cites my issue as its authority. Those are What to write instead — and it is the stronger claim
Three reasons this is better than the current text:
And if you want the caveat that made my report plausible, it is a different caveat and worth having: a shallow clone cannot answer the question, and What I am not disputingThe |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6d5a675: READY
One string, two consumers, and the text now says what a reader most needs to know about recorded_at — that it may name a commit the current pin cannot reach, and that the tool says "cannot place" rather than guessing.
What changed
ABOUT = '… `recorded_at` names the tree each base was measured at: it is provenance, and it may name a commit
that is no longer reachable from the current pin (engine#2643) -- `tools/hw_gate_manifest.py judge`
prints how the golden\'s tree relates to the tested one on the verdict line, and says "cannot place"
rather than guessing when it cannot.'Previously the same sentence existed twice — once in the shipped tests/golden/hw-gate/manifest.json, once in the template function — which is exactly how a sentence becomes true in one place and stale in the other. It is now one constant, and I checked the shipped file:
manifest about == ABOUT : True (461 chars both sides)
so the current manifest is not drifted, and the template can no longer drift from it.
Why the sentence is worth its length
A recorded_at that names an unreachable commit is normal for a golden measured on another tree, and the failure mode this prevents is a reader treating "not an ancestor" as "the base is wrong" — the same misread as #2643's shallow clone ("a shallow clone reports main's own commits as LOST"). Saying "it is provenance… and says 'cannot place' rather than guessing" in the file a pack author opens is the cheapest possible fix for that.
The one thing it does not have
No arm keeps manifest.json's about equal to ABOUT. Right now they match because the author edited both; the next hand-edit of the JSON (or a --record on an old checkout) re-opens exactly the drift this PR closes. Two lines in cmd_selftest() — load the shipped manifest and chk(ABOUT, manifest["about"]) — would make this the last time the sentence has to be maintained in two places. Given this PR's whole point is "one string for the template and the shipped manifest", the check that it is one string is the missing half.
hw_gate_manifest self-test: 0 failure(s) (11 arms).
|
The Evidence:
The Checks API refuses a re-request for this app ( |
What
The hw-gate manifest now states what
recorded_atcan and cannot mean. One constant feeds both the template and the shipped manifest, so they cannot drift.Why (engine#2643)
Five of six goldens name a llama.cpp commit in a history disjoint from the pinned fork (the author's reading; I could not reproduce it, because the pinned sha isn't even in this clone —
git cat-file -e→ absent). The field exists so a reader can ask "were these numbers measured at the tree I am testing?", and for those five the honest answer today is "the question cannot be asked".#2643put three options. This takes the cheapest one that needs no device and no judgement call:#2644(provenance())With
#2661armingprovenance(), the tool now sayscannot placerather than guessing; this makes the artifact a reader opens first say the same thing.The change
load()'s template usesABOUTinstead of its own copy of the string.tests/golden/hw-gate/manifest.jsoncarries the same text.One definition, two artifacts — the failure being prevented is a template and a shipped file disagreeing about the field they both describe.
Checked before committing
grepfor the string and for"about"finds onlylaya_route_cases.pyandregistry_build.py, which have their own unrelatedaboutfields).judgestill runs — it parses the manifest and reaches the log check (no KL divergence statistics in the llama-perplexity log), so nothing in the verdict path depends on the string.What this does not do
It does not re-record anything and does not change a verdict. Five goldens still name a tree that cannot be placed; this only stops a reader from assuming otherwise.