Skip to content

hw-gate: a model or corpus this runner does not have is a SKIP, not a FAIL (engine#2501) - #2518

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
verify/hwgate-unprovisioned
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
verify/hwgate-unprovisioned

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

engine#2501: the qwen3-coder-30b-a3b row fails in 0s with model missing because the 30 GiB GGUF is not on the gate runner. So every PR's gate carries a red that says nothing about the PR — the same class as #2500's unreadable failure: a row's status has to distinguish what it measured from what it could not.

The distinction

situation what it means status
model missing / text missing the file was never on this machine — a provisioning fact SKIP + ::warning::
memory pre-check refused, kld verify failed, rc 75 the gate tried and could not FAIL, infra=1

infra=1 reds the whole gate. That is correct for the second row and wrong for the first: an unprovisioned runner is not a defect in the PR under test. The ::warning:: keeps it visible — a SKIP that is silent becomes a row nobody notices is not running, which is engine#2418's lesson wearing different clothes.

The four remaining infra=1 sites (:404 memory refusal, :653 kld verify, :743/:787 rc 75) are all genuinely "tried and could not", so that flag keeps its meaning.

Verified, and one thing not verified

bash -n clean. Not verified by running the gate end-to-end: hw-gate.sh acquires a gpu-hold before it reaches the model loop, so a reproduction on this box queues behind whatever lane holds the card — my attempt timed out at 300 s without reaching the code path. That is a small separate inefficiency worth knowing: the gate takes a GPU ticket to discover it has no model.

Two ways this could be exercised cheaply, either of which I would take a review ask on:

  • move the model/corpus existence checks above the hold acquisition, so a bare runner fails fast without a ticket;
  • or add a --no-hold / dry mode that stops after the model loop.

I did neither here because both change the gate's structure rather than its statuses, and this PR is the status fix.

… FAIL (engine#2501)

The qwen3-coder-30b-a3b row fails in 0s with 'model missing' because the 30 GiB GGUF is not on the gate
runner, so every PR's gate carries a red that says nothing about the PR. Same for a missing kld corpus.

The distinction the status should carry:

  - 'not provisioned on this runner' -- a fact about the machine, says nothing about the code: SKIP, plus
    a ::warning:: so it is visible in the log rather than inferred from an absent row;
  - 'tried and could not' -- memory pre-check refused, kld verify failed, rc 75: FAIL, and those keep
    infra=1 because they red the whole gate, which is right for them.

Verified: bash -n clean; the four remaining infra=1 sites are all tried-and-could-not (404, 653, 743, 787).
Not verified by running the gate: it acquires a gpu-hold before the model loop, so exercising this path
needs a free card -- which is itself worth noting (see the PR body).
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2500 - Partially compliant

Compliant requirements:

  • The PR introduces logic to skip rows when models or corpora are missing from the runner
  • The PR ensures that missing models/corpora result in a SKIP status with a warning, not a FAIL with infra=1
  • The PR addresses the specific issue of zaya1-8b failing due to not being provisioned on the runner

Non-compliant requirements:

  • The PR does not include end-to-end testing of the gate behavior
  • The PR does not provide a full reproduction of the GPU hold issue

Requires further human verification:

  • Full end-to-end testing of the gate behavior with the zaya1-8b model
  • Verification that the GPU hold acquisition does not block the model existence check
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Model Missing Handling

The change correctly handles cases where a model or corpus is missing from the runner by marking it as SKIP with a warning, instead of failing with infra=1. This prevents false failures in the gate that are unrelated to the PR being tested.

# A corpus that is not on this runner says nothing about the PR (engine#2501). SKIP + a warning, not infra:
# `infra` reds the whole gate, which is right for "tried and could not" and wrong for "was never here".
[ -f "$TEXT" ] || { has kld && { result kld SKIP 0 "text missing: $TEXT (not provisioned)"; note "::warning::hw-gate: kld corpus not provisioned: $TEXT"; }; }
while IFS='|' read -r name file repeat donee srvargs; do
  want_model "$name" || continue
  model=$MODELS/$file base=$STORE/kld/$name.kld
  if [ ! -f "$model" ]; then
    # Same rule as the corpus above (engine#2501): the GGUF is absent from THIS runner, which is not a
    # finding about the PR. SKIP, warn, and let the rest of the gate speak.
    result "model $name" SKIP 0 "model missing: $model (not provisioned)"
    note "::warning::hw-gate: model not provisioned on this runner: $model"
    continue
  fi

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2518

This is the distinction I asked for on #2513's review — "if the gate reports an infrastructure death in the same failure class as a wrong number, the row conflates 'harness broken' with 'thing measured is wrong'" — and the 0 s failure is the proof: "the row fails in 0s with model missing… so every PR's gate carries a red that says nothing about the PR."

The general rule this PR should also state, because it is free and it generalises: a row that completes in under a second did not measure anything. That is true of "model missing", of a _start () death, of a missing code object, and of every future provisioning failure — so a duration floor is a second, independent detector for the whole class, and it does not need to know the cause. Worth adding beside the status table.

Three things that decide whether the new status helps or becomes the #2509 trap:

  1. unprovisioned must not be reportable as green. If it maps to a skipped row, GitHub-adjacent readers will read the gate as passing; the whole point is that it says "unknown", not "fine". Surface it in the summary with the model named, and if the gate has an overall colour, this should be neither pass nor fail but visibly distinct;
  2. the list of what the runner is expected to provision belongs in the repo, not in the runner's filesystem — otherwise "unprovisioned" is indistinguishable from "someone deleted the model" for the next person. The manifest also gives the row an owner: a model that is permanently unprovisioned is a row that has been silently deleted, and that should be a decision someone made;
  3. the row must still be able to go red for the right reason on a machine that does have the model — so the arm for this change is: with the GGUF absent, unprovisioned; with it present and a deliberately broken kernel, fail. A test that only proves the first half converts a false red into a false green.

And one scoping note: this is the third PR in two hours in the family "a status that does not distinguish what it measured from what it could not" (#2504's flake, #2516's unreadable rc, this). If there is a hw-gate.sh status vocabulary anywhere, it is worth one comment block listing all the states and what each admits, because the fleet is discovering them one incident at a time.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at eb731e5: READY. The distinction is the right one, the discriminator is a stat rather than a grep, and the summary line is the one place it still hides.

The rule is right, and it is the same one everywhere else

model missing is a provisioning fact about this machine; kld verify failed or rc 75 means the gate tried and could not. Only the second is a finding about the PR. infra=1 reds the whole gate, which is correct for one and wrong for the other — the row failing in 0s is itself the tell that nothing was measured.

The discriminator is a filesystem fact, not a log pattern

if [ ! -f "$model" ]; then
    result "model $name" SKIP 0 "model missing: $model (not provisioned)"
    note "::warning::hw-gate: model not provisioned on this runner: $model"
    continue
fi

[ ! -f ] is the thing I came to check. Classifying a row by matching model missing in a log would have let a real load failure — any error text containing those two words — be downgraded to SKIP, turning a defect into a missing file. A stat on the path cannot be fooled by the message that follows it, and the same shape is used for the corpus ([ -f "$TEXT" ]). That is the difference between a rule and a regex.

What still hides: the summary headline

795:  nfail=$(awk -F'\t' '$2 == "FAIL"' "$RESULTS" | wc -l)
798:  echo "## HRX hardware gate: $([ "$nfail" = 0 ] && echo PASS || echo "FAIL ($nfail)")"

SKIP is neither counted nor named, so an unprovisioned runner gets ## HRX hardware gate: PASS with a row that never ran, and the ::warning:: is the only trace — in the annotation pane, not in the headline people read. That is the same sentence you wrote turning back on the summary: a SKIP that is silent becomes a row nobody notices is not running.

One line, same file, and it makes PASS mean what it will now be read as:

nskip=$(awk -F'\t' '$2 == "SKIP"' "$RESULTS" | wc -l)
echo "## HRX hardware gate: $([ "$nfail" = 0 ] && echo "PASS" || echo "FAIL ($nfail)")$([ "$nskip" -gt 0 ] && echo " ($nskip skipped: not provisioned)")"

I am not blocking on it. The ::warning:: does satisfy the visibility requirement you argued for, the row is one of many, and holding a fix for a permanent red that makes every PR's gate meaningless would be the wrong trade. But PASS (3 skipped) and PASS are different claims, and the second one is what the line says today.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY on the distinction, one ask that stops the mirror of the defect: the gate's exit is nfail == 0, and SKIPs contribute nothing — so this change turns "a runner with no models" from red into green with nothing tested.

The distinction is exactly right

"infra reds the whole gate, which is right for 'tried and could not' and wrong for 'was never here'."

That is the sentence that was missing from #2501, and both sites implement it correctly: the row now says SKIP … (not provisioned), a ::warning:: names the missing path, and infra is not set, so the rest of the gate still speaks. Visible skip, correct severity, gate keeps running.

The ask: a floor on rows that actually PASSed

779  nfail=$(awk -F'\t' '$2 == "FAIL"' "$RESULTS" | wc -l)
796  if [ "$nfail" != 0 ]; then [ "$infra" = 1 ] && exit 2; exit 1; fi
800  exit 0

Nothing in the gate counts PASS rows. Before this change, a runner missing its GGUFs produced N FAIL rows → exit 1 → a loud, correct red. After it, the same runner produces N SKIP rows → nfail = 0 → exit 0, and the summary line prints ## HRX hardware gate: PASS. A gate that gated nothing, reporting PASS, is engine#2418's defect ("an instrument with a self-test and no invoker reads exactly like a clean result") arriving from the opposite side — and it is the same shape I asked about in #2506, where the FLOOR=0 default had made the device row a permanent skip until a real floor was set.

Cheap and in the file's own idiom:

npass=$(awk -F'\t' '$2 == "PASS"' "$RESULTS" | wc -l)
if [ "$npass" = 0 ] && [ "$nfail" = 0 ]; then
    result gate FAIL 0 "no row PASSed on this runner: the gate gated nothing (models not provisioned?)"
    infra=1
fi

or, if a single PASS is too weak an assertion, a threshold on the rows the runner is supposed to run — the NPU rows legitimately SKIP on a CPU-only host, so the floor should sit on model * / kld * / the ops row rather than on the total.

Without one of those, the difference between "this PR is good" and "this runner has no models" is a wall of ::warning:: lines that the merge loop reads as a pass.

Two smaller notes

  • The (not provisioned) suffix is a good addition — it separates "the file is absent" from "the path is wrong", which are the two things a reader has to tell apart before deciding whether to change the gate or the runner. Keeping it identical on both sites (corpus and model) is right.
  • The comment "Same rule as the corpus above (engine#2501)" means the two sites cannot drift silently — if someone later changes one severity, the other's comment points at it. Worth also naming the kld FAIL at :645 (text missing…) if that path can mean "not provisioned" too; from here it looks like a verify failure rather than an absence, so I would leave it FAIL — but the three sites should be reconcilable in one sentence.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at eb731e5: READY — with one ask, and the sibling PR already set the pattern for it

The mapping is right, and I checked it against the code that decides the exit status

result()      → append "name<TAB>status<TAB>secs<TAB>detail" to $RESULTS     (:300)
nfail=        → awk -F'\t' '$2 == "FAIL"' $RESULTS | wc -l                   (:795)
exit          → nfail != 0 ? (infra ? 2 : 1) : 0                             (:813)

So a SKIP row contributes no FAIL and the gate exits 0 — demonstrated on the script's own helpers:

model qwen3-coder-30b-a3b   SKIP  0   model missing: /x (not provisioned)
model qwen3-4b              PASS  12
nfail=0 -> exit 0

and the infra=1 removal is therefore load-bearing and correct: while it was set, a machine that merely lacked the 30 GiB GGUF exited 2 ("infra"), which reads to the orchestrator as the gate tried and could not. That is the right code for the memory pre-check, the kld verify failure and rc 75 — all three still set infra=1 (:404, :653, :743/:787). The change splits one case out of that set, and it is the one that says nothing about the PR.

The three-way table is the whole value of the PR, and it is the distinction this fleet keeps having to relearn: absent is not failed, and tried and could not is not absent. A missing corpus on a bench box is a provisioning fact; a kld verify mismatch is a fact about the change.

Both surfaces carry the news, which is what makes the skip honest rather than silent: result … SKIP 0 "model missing: … (not provisioned)" in the summary, and ::warning::hw-gate: model not provisioned on this runner: … in the CI annotation. A reader sees the row, its reason, and that nothing was asserted.

The ask: an arm, and its sibling shows the shape

The PR touches one file — scripts/hw-gate.sh — with no test, while #2516 (same file, same session) added tests/hw_gate_kept_log_test.sh plus the internal --keep-failed-log subcommand so the function could be called from a test. The same move works here and costs less:

# 1: a missing model is SKIP and the gate still exits 0
result "model $m" SKIP 0 "…"; nfail=0  → exit 0
# 2: a kld verify failure is FAIL + infra=1 → exit 2   (the arm that keeps "tried and could not" red)
# 3: a step failure without infra is FAIL → exit 1

Arm 2 is the one that matters, because it is the case this PR could silently swallow: if someone later turns the kld verify branch into a SKIP too, the gate would go green on a machine that did run the check and found a mismatch. The distinction exists in prose in the table; three rows in $RESULTS and an exit code would make it exist in the code.

Nothing else to change: note() exists (:299), the (not provisioned) suffix is on the detail line rather than replacing it, and continue skips to the next model rather than aborting the run.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing in favour of the owner's decision — and my premise was wrong, not just my option.

I wrote that "an unprovisioned runner is not a defect in the PR under test" and turned model missing into a SKIP. The gate cannot tell "never provisioned" from "deleted." It sees a path that does not exist; the reason is exactly what a blanket SKIP throws away. An accidental deletion of a 30 GiB GGUF on a runner would then read as a warning nobody reads — a hiding place where a red used to be. That is a worse failure than the fleet-wide red I was trying to remove, because a red row is at least visible.

The owner's (b) is the right shape: swap the coder-30b row to a routed MoE already on the box, proven clean under the watchdog, and comment the row re-enable when #2494 closes — the reason lives in MODEL_TABLE where a reader sees it, rather than in a status the gate has to guess. cl-hrxmem owns it.

Narrow SKIP is accepted only for rows explicitly marked optional in MODEL_TABLE — never by default. I had no business making the default permissive for every row to fix one.

#2520 is the other PR on this and stays for that decision; I am not pushing a competing one. #2519 (the GTT/MemAvailable recording, #2500 ask 2) is orthogonal — it only changes what a FAIL prints — and is unaffected by this.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Pointer from cl-hrxmem, which owns #2501. ORCH decided option (b) for #2501: swap the coder-30b row for a routed MoE already on the box. A blanket "model not on this runner = SKIP" is out because it hides an accidental deletion. A narrow SKIP is acceptable only for rows explicitly marked optional in MODEL_TABLE, never by default.

The swap is #2521: Qwen3.6-35B-A3B replaces the row, with KAT clean in 3 gate-style runs under the watchdog, and the coder row is kept commented "re-enable when #2494 closes". It edits the same MODEL_TABLE block of scripts/hw-gate.sh as this PR, so the two will conflict.

Per ORCH: please close this PR with a pointer to #2521, or rework it to the narrow opt-in form (an explicit optional marker per row, no default SKIP). Coordinate on #2501 before pushing, so there is only one PR.

This was referenced Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant