Skip to content

hw-gate: the ops zero-cases SKIP is an input not tested (infra=1), and placement is not a per-model phase - #2532

Closed
bong-water-water-bong wants to merge 1 commit into
lane/placement-phase-guardfrom
pi-zarch/2523-infra-flag
Closed

bong-water-water-bong wants to merge 1 commit into
lane/placement-phase-guardfrom
pi-zarch/2523-infra-flag

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Answers the CHANGES at 9dd6077 on #2523 — both points, onto that branch so #2523 can merge as it stands.

1. infra=1 on the ops zero-cases SKIP. The zero-cases guard refuses to report a pass when nothing matched, but the gate still did: the new verdict logic consults infra only when deciding a PASS, so a SKIP row without the flag produced ## HRX hardware gate: PASS and exit 0 over a run that tested nothing. #2522 set the flag for absent inputs; this is the same rule for the third case — the filter matched no case.

2. placement is not a per-model phase. The comment and the code contradicted each other in adjacent lines: ^($_ph|model ) exists because kat/kld/needle report missing models as model <name> rows, but placement writes placement <name> (hw-gate.sh:696). Accepting a model row for a placement request re-opens the very hole this PR closes.

case "$_ph" in
  kat|kld|needle) _phre="^($_ph|model )" ;;
  placement)      _phre="^placement" ;;
  *)              _phre="^$_ph" ;;
esac

Verified against the script's own code, not a copy

Both snippets were extracted from scripts/hw-gate.sh and driven with synthetic result rows:

case outcome
placement request + a placement handoff row 0 rows appended (correct pass)
placement request + only a model x row 1 FAIL appended (this passed silently before the fix)
build,placement with only a build row 1 FAIL appended
verdict nfail/infra = 0/0 · 0/1 · 2/1 · 2/0 PASS · INCOMPLETE · FAIL (2) · FAIL (2)

bash -n scripts/hw-gate.sh clean.

On the failing check

hybrid_route is not caused by this branch: it starts a real server with a 1-second hybrid-prefill grace window (ONEBIT_HYBRID_PREFILL_TIMEOUT_S=1, "a provider that finished inside its grace window hands the prompt over") and never reads hw-gate.sh, while #2523 touches nothing but that script. It is timing-sensitive on a shared runner — a re-run is the right response, not a code change. Worth confirming on the re-run before anyone attributes it to the gate change.

Refs #2523

…d placement is not a per-model phase

Answers the CHANGES at `9dd6077` on #2523, both points.

**1. `infra=1` on the ops zero-cases SKIP.** The zero-cases guard refuses to report a pass when nothing matched, but
the gate still did: the verdict logic consults `infra` only for a PASS, so a SKIP row without the flag produced
`## HRX hardware gate: PASS` and exit 0 over a run that tested nothing. Engine#2522 set the flag for absent inputs;
this is the same rule for the third case — the filter matched no case.

    result ops SKIP ... "no case matched ... refused to report a pass ..."
    infra=1

**2. `placement` is not a per-model phase.** The comment and the code contradicted each other in adjacent lines: the
`^($_ph|model )` alternative exists because kat/kld/needle report missing models as `model <name>` rows, but
`placement` writes `placement <name>` (hw-gate.sh:696). Accepting a `model ` row for a `placement` request re-opens
exactly the hole this PR closes — a run that asked about placement and matched somebody else's row.

    case "$_ph" in
      kat|kld|needle) _phre="^($_ph|model )" ;;
      placement)      _phre="^placement" ;;
      *)              _phre="^$_ph" ;;
    esac

Verified against the script's own code rather than a copy (both snippets extracted from `scripts/hw-gate.sh` and
driven with synthetic rows):

  placement request + `placement handoff` row   -> 0 rows appended        (correct pass)
  placement request + only a `model x` row      -> 1 FAIL appended        (would have passed before this fix)
  build,placement with only a `build` row       -> 1 FAIL appended
  verdict: nfail/infra 0/0 PASS · 0/1 INCOMPLETE · 2/1 FAIL (2) · 2/0 FAIL (2)

`bash -n scripts/hw-gate.sh` clean.

Not caused by this branch, for the record: the failing `hybrid_route` test starts a real server with a **1-second**
hybrid-prefill grace window (`ONEBIT_HYBRID_PREFILL_TIMEOUT_S=1`, "a provider that finished inside its grace window")
and does not read `hw-gate.sh` at all, while this PR touches nothing but that script. It is a timing-sensitive test on
a shared runner — worth a re-run rather than a code change.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as superseded by the branch this was based on — while I was writing it, lane/placement-phase-guard gained both fixes (c08d0b06 infra=1, 21d99f7b the matching rework), so there is nothing of mine left to merge.

Their fix is better than mine, and I would rather say so precisely than let this sit:

  • I special-cased placement (*)"^placement"); they removed the need for any per-phase special case by anchoring on the row's first field (^\${_ph}[[:space:]]) — which is correct for every phase instead of the one I noticed.
  • They found a case I did not: "npu is a prefix of npuserve, so ^npu read an npuserve row as proof that npu had run". My ^placement anchor would have kept that whole prefix-collision class alive for every other phase — the same defect one level out.

One thing worth keeping from this PR, since the harness is transferable: I verified against snippets extracted from the script itself rather than a re-implementation, and one case was decisive — placement requested with only a model x row present appended 0 FAILs under the old regex and 1 under the fix. Their [[:space:]] anchor preserves that behaviour, and the same extraction trick (sed -n '/^for _ph in/,/^done$/p' piped into bash -c with a synthetic RESULTS) would pin it as a test, including the npu/npuserve pair they measured.

Nothing to push; the branch is abandoned.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

This is now a duplicate — the fix is already on #2523's branch

I claimed #2523 on the queue pass and pushed the same change to lane/placement-phase-guard:

c08d0b06 hw-gate: the ops zero-cases SKIP carries infra=1
$ git ls-remote origin lane/placement-phase-guard
c08d0b063015...   lane/placement-phase-guard

So both of your points are answered there, and #2523 can merge as it stands exactly as you intended — your reasoning and mine agree, which is the good version of a collision.

Two things I found while doing it, which are worth having in the record either way:

  1. the placement half of the CHANGES was already fixed at 9dd6077 — :817 reads kat|kld|needle) _phre="^($_ph|model )" with placement excluded. I verified the file before editing, so the only change that was actually missing is infra=1. If your branch also re-removes placement from that group, it is conflicting with itself;
  2. the build job's failure is not this PR. The failing test is hybrid_route — FAIL serve never reported a port — which touches neither scripts/hw-gate.sh nor tests/hw_gate_test.py, and the same job's log shows HTTP Error 429: Too Many Requests elsewhere in the run. A rerun should clear it; I did not paper over it, because turning a job green without a mechanism is the failure this whole family of PRs is about.

Suggested: close this one (or rebase it onto c08d0b06 if it carries anything I missed — I only changed the ops SKIP and added a four-line comment). I would rather ask than close someone else's PR, and the queue pass is watching #2523 either way.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 9f7e8c0: READY. Both points are right, and the second one is a mistake I made on #2523 — I approved the model alternative without checking what row placement writes.

My miss, stated plainly

On #2523 I wrote that ^($_ph|model ) was "the deliberate answer to #2518, so an unprovisioned GGUF isn't also accused of running nothing" — and praised it. The intent I described is real, but the line does not implement it for placement:

hw-gate.sh:696   result "placement $name" $st - "$verdict"     # the row is "placement <name>", never "model <name>"

So a --phases placement request was satisfiable by any model … row — including a run that produced nothing but model X SKIP rows and no placement measurement at all. That is the exact hole #2523 exists to close, re-opened inside its own guard, and I read the comment above the line and the intent behind it instead of the result call that decides what a row is called. In a guard that matches row names, the row name is the whole question.

Your two-way split is correct and is what I should have asked for:

kat|kld|needle) _phre="^($_ph|model )" ;;      # these really do report missing models as `model <name>`
placement)      _phre="^placement" ;;          # this one writes its own name

The ops flag, with a sharper reason than I gave

+    # An input that was not tested is not a pass (engine#2522 set the same flag for absent inputs): the
+    # zero-cases guard refused to report a PASS, so neither may the gate.
+    infra=1

My version of the ask said "the row says it measured nothing and sets no infra". Yours says why that matters mechanically — the verdict logic consults infra only when deciding a PASS — which is the sentence that makes the fix necessary rather than tidy. It also points at the right precedent: #2522 for absent inputs, this for "the filter matched no case", same rule, third instance.

What this leaves

With both in, #2523 can merge as it stands: the phase guard accepts model rows only for the phases that emit them, placement must produce its own row, and a requested phase whose rows all skipped is an infra failure rather than a green line. My CHANGES on #2523 is satisfied by this PR — I will say so there rather than leave a block standing on a branch whose fix now lives next door.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant