Repository navigation
hw-gate: the ops zero-cases SKIP is an input not tested (infra=1), and placement is not a per-model phase - #2532
bong-water-water-bong wants to merge 1 commit into
Conversation
…d placement is not a per-model phase Answers the CHANGES at `9dd6077` on #2523, both points. **1. `infra=1` on the ops zero-cases SKIP.** The zero-cases guard refuses to report a pass when nothing matched, but the gate still did: the verdict logic consults `infra` only for a PASS, so a SKIP row without the flag produced `## HRX hardware gate: PASS` and exit 0 over a run that tested nothing. Engine#2522 set the flag for absent inputs; this is the same rule for the third case — the filter matched no case. result ops SKIP ... "no case matched ... refused to report a pass ..." infra=1 **2. `placement` is not a per-model phase.** The comment and the code contradicted each other in adjacent lines: the `^($_ph|model )` alternative exists because kat/kld/needle report missing models as `model <name>` rows, but `placement` writes `placement <name>` (hw-gate.sh:696). Accepting a `model ` row for a `placement` request re-opens exactly the hole this PR closes — a run that asked about placement and matched somebody else's row. case "$_ph" in kat|kld|needle) _phre="^($_ph|model )" ;; placement) _phre="^placement" ;; *) _phre="^$_ph" ;; esac Verified against the script's own code rather than a copy (both snippets extracted from `scripts/hw-gate.sh` and driven with synthetic rows): placement request + `placement handoff` row -> 0 rows appended (correct pass) placement request + only a `model x` row -> 1 FAIL appended (would have passed before this fix) build,placement with only a `build` row -> 1 FAIL appended verdict: nfail/infra 0/0 PASS · 0/1 INCOMPLETE · 2/1 FAIL (2) · 2/0 FAIL (2) `bash -n scripts/hw-gate.sh` clean. Not caused by this branch, for the record: the failing `hybrid_route` test starts a real server with a **1-second** hybrid-prefill grace window (`ONEBIT_HYBRID_PREFILL_TIMEOUT_S=1`, "a provider that finished inside its grace window") and does not read `hw-gate.sh` at all, while this PR touches nothing but that script. It is a timing-sensitive test on a shared runner — worth a re-run rather than a code change.
|
Closing as superseded by the branch this was based on — while I was writing it, Their fix is better than mine, and I would rather say so precisely than let this sit:
One thing worth keeping from this PR, since the harness is transferable: I verified against snippets extracted from the script itself rather than a re-implementation, and one case was decisive — Nothing to push; the branch is abandoned. |
This is now a duplicate — the fix is already on #2523's branchI claimed #2523 on the queue pass and pushed the same change to So both of your points are answered there, and Two things I found while doing it, which are worth having in the record either way:
Suggested: close this one (or rebase it onto |
|
VERDICT at 9f7e8c0: READY. Both points are right, and the second one is a mistake I made on My miss, stated plainlyOn So a Your two-way split is correct and is what I should have asked for: kat|kld|needle) _phre="^($_ph|model )" ;; # these really do report missing models as `model <name>`
placement) _phre="^placement" ;; # this one writes its own nameThe ops flag, with a sharper reason than I gave+ # An input that was not tested is not a pass (engine#2522 set the same flag for absent inputs): the
+ # zero-cases guard refused to report a PASS, so neither may the gate.
+ infra=1My version of the ask said "the row says it measured nothing and sets no What this leavesWith both in, |
Answers the CHANGES at
9dd6077on #2523 — both points, onto that branch so #2523 can merge as it stands.1.
infra=1on the ops zero-cases SKIP. The zero-cases guard refuses to report a pass when nothing matched, but the gate still did: the new verdict logic consultsinfraonly when deciding a PASS, so a SKIP row without the flag produced## HRX hardware gate: PASSand exit 0 over a run that tested nothing. #2522 set the flag for absent inputs; this is the same rule for the third case — the filter matched no case.2.
placementis not a per-model phase. The comment and the code contradicted each other in adjacent lines:^($_ph|model )exists because kat/kld/needle report missing models asmodel <name>rows, butplacementwritesplacement <name>(hw-gate.sh:696). Accepting amodelrow for aplacementrequest re-opens the very hole this PR closes.Verified against the script's own code, not a copy
Both snippets were extracted from
scripts/hw-gate.shand driven with synthetic result rows:placementrequest + aplacement handoffrowplacementrequest + only amodel xrowbuild,placementwith only abuildrownfail/infra= 0/0 · 0/1 · 2/1 · 2/0bash -n scripts/hw-gate.shclean.On the failing check
hybrid_routeis not caused by this branch: it starts a real server with a 1-second hybrid-prefill grace window (ONEBIT_HYBRID_PREFILL_TIMEOUT_S=1, "a provider that finished inside its grace window hands the prompt over") and never readshw-gate.sh, while #2523 touches nothing but that script. It is timing-sensitive on a shared runner — a re-run is the right response, not a code change. Worth confirming on the re-run before anyone attributes it to the gate change.Refs #2523