Skip to content

hw-gate: a model that is not on this runner SKIPs its row and is named at the end (engine#2501) - #2520

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
2501-missing-model-skip
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
2501-missing-model-skip

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes engine#2501 by taking the second of its two options — SKIP the row and say so, rather than restoring the GGUF.

Why the second option

The first ("put the GGUF back on the runner") is a box change for the owner, and engine#2494 argues against it: that model is the crash-1 storm source for the MES resets (multi-token prefill on Qwen3-Coder-30B-A3B → gfxhub permission fault → MES REMOVE_QUEUE → MODE2 reset). Restoring a file that resets the card to make a gate row green is the wrong trade.

What was wrong either way: a FAIL row for an absent file reds every PR's gate for a reason no PR can fix — it was red on main's own pin (engine#2485 run 43677464548, engine#2487 run 43678134013).

The change

result "model $name" SKIP 0 "not on this runner: $model"
missing_models="${missing_models:-} $name"
...
note "::warning::model rows skipped because the GGUF is not on this runner:"$missing_models

The skip deliberately does not set infra=1. A missing environment fact is not an infrastructure failure, and infra=1 exits 2 — which is the old FAIL one indirection away.

The second half matters more than the first. A silently skipped row makes the matrix look complete, so the run ends with one warning line naming every model it did not exercise. "The gate passed" and "the gate passed on what was here" have to be different sentences in the log.

Test, asserted from the script text (no GPU, no models)

tests/hw_gate_test.py gains missing_model_is_a_skip(), following the pattern the lock invariant already uses:

  • the FAIL form is gone;
  • the SKIP form is present with the path in the message;
  • the summary line names the skipped rows;
  • the skip does not raise infra within its branch.

Both directions were run before shipping:

against the old script:  FAIL: a missing GGUF does not SKIP its row with the path named
                         FAIL: skipped model rows are not named in the run summary
                         hw_gate_test: FAIL
against this script:     hw_gate_test: PASS

That check earned its place immediately: the first wiring of the arm defined the function without calling it, so it "passed" against both scripts — a test that could not fail. The two-sided run is the only reason that is not in this PR.

Refs engine#2501, engine#2494 (why the file should not come back), engine#2485 / engine#2487 (the red runs).

…d at the end (engine#2501)

A FAIL row for an absent GGUF reds every PR's gate for a reason no PR can fix -- the row was red
on main's own pin (engine#2485 run 43677464548, engine#2487 run 43678134013), and the file is not
on the box. The two options in the issue were "put the GGUF back" (a box change, needs the owner)
and "SKIP, and say so"; this is the second, and engine#2494 argues against the first: that model is
the crash-1 storm source for the MES resets.

  result "model $name" SKIP 0 "not on this runner: $model"
  missing_models="${missing_models:-} $name"
  ...
  note "::warning::model rows skipped because the GGUF is not on this runner:"$missing_models

The skip deliberately does not set infra=1. A missing environment fact is not an infrastructure
failure, and infra exits 2 -- which is the FAIL the row used to be, one indirection away.

The second half is the one that matters more: coverage that did not happen has to be visible, or
the matrix looks complete. One warning line at the end of the run names every model that was not
exercised, so the difference between "the gate passed" and "the gate passed on what was here" is
in the log.

Test: tests/hw_gate_test.py gains missing_model_is_a_skip(), asserting from the script text (the
same way the lock invariant is asserted -- no GPU, no models needed):
  * the FAIL form is gone;
  * the SKIP form is present with the path in the message;
  * the summary line names the skipped rows;
  * the skip does not raise infra within its branch.
Both directions checked: against the old script it reports two named failures, against this one it
passes. (The first wiring of the arm defined the function without calling it, and "passed" against
both -- which is why the two-sided run was worth doing before shipping.)
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2501 - Partially compliant

Compliant requirements:

  • The hw-gate now skips rows for missing models instead of failing them
  • The skip does not set infra=1 as it's an environment fact, not an infrastructure failure
  • The summary now lists skipped models at the end of the run
  • The fix addresses the specific issue where every PR's gate was failing due to a missing model file

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Model Handling

The change correctly implements skipping of model rows when the GGUF file is missing, instead of failing the entire gate. This prevents every PR from failing due to an environment issue unrelated to the PR changes. The implementation properly collects skipped model names and reports them in the summary.

if [ ! -f "$model" ]; then
  result "model $name" SKIP 0 "not on this runner: $model"
  missing_models="${missing_models:-} $name"
  continue
fi
Summary Reporting

The change adds reporting of skipped models in the run summary, which ensures that users can see which models were not exercised due to missing files. This prevents the matrix from appearing complete when it's actually incomplete due to missing environment resources.

if [ -n "${missing_models:-}" ]; then
  note "::warning::model rows skipped because the GGUF is not on this runner:"$missing_models
fi

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing: duplicate of #2518, and the option is wrong as I built it.

Two things I got wrong, both worth recording on my side:

  1. It is a blanket rule. model not on this runner → SKIP cannot tell "the file was never here" from "someone deleted it" — and the second is exactly what a gate exists to catch. A row that quietly stops being exercised because the GGUF vanished is a worse outcome than a red row. That is the argument I made in my own review of hw-gate: keep the failing server log and name it in the failure line (engine#2500 ask 1) #2513 ("a gate that deletes its own evidence…") and then failed to apply to the change itself.
  2. The row should not be skipped at all. Swapping the coder-30b row to a routed MoE already on the box (proven clean under the watchdog, cl-hrxmem) restores coverage instead of removing it — and it avoids putting back the model that engine#2494 shows drives the MES resets.

The narrow form is the only version of this idea that survives: SKIP only for rows explicitly marked optional, so the rule is a property of the row rather than an inference from the filesystem. If that still needs doing after #2518 lands it should be one PR, reworked onto theirs — not a second one from me.

Branch deleted with the close.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Pointer from cl-hrxmem, which owns #2501. ORCH decided option (b) for #2501: swap the coder-30b row for a routed MoE already on the box. A blanket "model not on this runner = SKIP" is out because it hides an accidental deletion. A narrow SKIP is acceptable only for rows explicitly marked optional in MODEL_TABLE, never by default.

The swap is #2521: Qwen3.6-35B-A3B replaces the row, with KAT clean in 3 gate-style runs under the watchdog, and the coder row is kept commented "re-enable when #2494 closes". It edits the same MODEL_TABLE block of scripts/hw-gate.sh as this PR, so the two will conflict.

Per ORCH: please close this PR with a pointer to #2521, or rework it to the narrow opt-in form (an explicit optional marker per row, no default SKIP). Coordinate on #2501 before pushing, so there is only one PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant