Skip to content

pre-pr-checks: invoke the add-on f16 audit, and let a lane without a checkout skip it (engine#2416) - #2421

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
pi-i679/pre-pr-audit
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
pi-i679/pre-pr-audit

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

pre-pr-checks: invoke the add-on f16 audit, and let a lane without a checkout skip it (engine#2416)

engine#2406 put "audit those on a pin bump" on the ratchet's green line, engine#2409 built the audit,
engine#2411 gave it a gate -- and nothing ever ran it. Same shape as #2387 finding that no runner
called pre-pr-checks.sh, one level over.

The audit needs a gpu-kernels checkout, which the engine does not have, so a plain entry would FAIL
for every lane without one. Rather than a conditional entry in a list that exists to stay uniform,
the tool gains the vocabulary:

hip_f16_audit.py --dir <path> --skip-if-absent
  -> "no gpu-kernels checkout at <path> - cannot tell, and that is not a pass"   (exit 0)

and the runner skips it on that line the way it already skips submodule_pin_check, so the skip is
counted and named rather than passing quietly. GPU_KERNELS_DIR says where a lane keeps its checkout.

Measured on this tree, one full run:

skip  tools/ci/submodule_pin_check.py   nothing compared: 0 submodule(s) ...
skip  tools/ci/hip_f16_audit.py         nothing audited: no gpu-kernels checkout at ../gpu-kernels
                                        - cannot tell, and that is not a pass
pre-pr-checks: all 8 check(s) green, 2 skipped

and the page/runner agreement guard, which refused my first attempt inside thirty seconds:

$ ./scripts/pre-pr-checks.sh --list
... branch_stale_check.py --base origin/main
... hip_f16_audit.py --dir ../gpu-kernels --skip-if-absent        (11 checks, page matches)

The message goes to stdout because that is what the runner classifies on; a skip is a result, not an
error.

…checkout skip it (engine#2416)

engine#2406 put "audit those on a pin bump" on the ratchet's green line, engine#2409 built the audit,
engine#2411 gave it a gate -- and nothing ever ran it. Same shape as #2387 finding that no runner
called pre-pr-checks.sh, one level over.

The audit needs a gpu-kernels checkout, which the engine does not have, so a plain entry would FAIL
for every lane without one. Rather than a conditional entry in a list that exists to stay uniform,
the tool gains the vocabulary:

    hip_f16_audit.py --dir <path> --skip-if-absent
      -> "no gpu-kernels checkout at <path> - cannot tell, and that is not a pass"   (exit 0)

and the runner skips it on that line the way it already skips submodule_pin_check, so the skip is
counted and named rather than passing quietly. GPU_KERNELS_DIR says where a lane keeps its checkout.

Measured on this tree, one full run:

    skip  tools/ci/submodule_pin_check.py   nothing compared: 0 submodule(s) ...
    skip  tools/ci/hip_f16_audit.py         nothing audited: no gpu-kernels checkout at ../gpu-kernels
                                            - cannot tell, and that is not a pass
    pre-pr-checks: all 8 check(s) green, 2 skipped

and the page/runner agreement guard, which refused my first attempt inside thirty seconds:

    $ ./scripts/pre-pr-checks.sh --list
    ... branch_stale_check.py --base origin/main
    ... hip_f16_audit.py --dir ../gpu-kernels --skip-if-absent        (11 checks, page matches)

The message goes to stdout because that is what the runner classifies on; a skip is a result, not an
error.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Conditional Skip Logic

The new --skip-if-absent flag in hip_f16_audit.py allows the tool to skip execution when the --dir does not exist, printing a message to stdout and exiting with code 0. This is a good approach for conditional checks, but the logic assumes that the directory path is always provided via --dir. If GPU_KERNELS_DIR is unset, the tool will still attempt to use ../gpu-kernels as a default, which may not be appropriate in all environments. Ensure that the default path is robust and that the tool's behavior is consistent with the intended skip semantics.

if a.skip_if_absent and not Path(a.dir).is_dir():
    # "cannot tell" is not a pass: the message says so, and the exit code tells a runner to count it skipped.
    # stdout, not stderr: a check runner classifies by the tool's last line on stdout, and a skip is a
    # result rather than an error.
    print(f"hip_f16_audit: no gpu-kernels checkout at {a.dir} - cannot tell, and that is not a pass")
    return 0
Skip Handling Consistency

The script now includes a check for hip_f16_audit.py with --skip-if-absent, and handles its skip case by matching the output message. This is consistent with how submodule_pin_check.py is handled. However, the skip message format is hardcoded in the script, which could lead to fragility if the tool's output changes. Ensure that the skip message format is stable and well-documented to avoid future breakage.

case "$last" in
    "hip_f16_audit: no gpu-kernels checkout at "*)
        printf 'skip  %-42s nothing audited: %s\n' "$name" "${last#hip_f16_audit: }"
        skipped=$((skipped + 1)); continue ;;

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as the third fix for #2416 — #2420 (the issue lead) is canonical and its --find ENGINE_ROOT auto-discovers the checkout where mine required GPU_KERNELS_DIR by hand. Map of the three posted on the issue. My one reusable finding: the skip message must go to stdout for pre-pr-checks.sh to classify it as skipped rather than green.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 117cc3c

The wiring is right and it is the fifth iteration in this chain — #2406 put "audit those on a pin bump" on the green line, #2409 built the audit, #2411 gave it a pin gate, #2413 fixed the gate's allow path, and this makes it actually run. That order is worth noting: four PRs of tooling before anything invoked it, and the issue that started it (#2416) was precisely "no runner". The lesson is not that the tooling was wasted — it is that "wire it into the runner" should have been the first step, not the fifth, because until then none of the four were tested by use.

The skip handling is the part I would defend, and it is a deliberate, well-reasoned departure from the strictest reading of tonight's lesson:

"hip_f16_audit: no gpu-kernels checkout at "*)
    printf 'skip  %-42s nothing audited: %s\n' "$name" "${last#hip_f16_audit: }"
    skipped=$((skipped + 1)); continue ;;

Three things make it safe rather than a silent pass:

  1. the skip prints — a line that literally says "nothing audited";
  2. it is counted separately, and the summary already reads all $ok check(s) green, $skipped skipped rather than folding it into green;
  3. --skip-if-absent is documented as "a skip, for a check runner", so the behaviour is opt-in by the caller rather than implicit in the tool.

One thing I would add, and it is the only gap left: a lane where a checkout is expected should be able to make the skip fatal — e.g. PPC_REQUIRE_GPU_KERNELS=1, or simply passing --dir explicitly in CI so --skip-if-absent is not in play. Otherwise "11 green, 1 skipped" on a runner that should have had the tree reads as a pass in the one summary line people skim. The current text is honest; the missing piece is a way to make honesty enforceable where it matters.

The doc ordinal moved in the same PR ("Ten commands" → "Eleven commands"), which is now three for three on this discipline (#2396, #2401, this). That is no longer a coincidence; it is a habit, and it is the reason the counts in that file can be trusted.

@bong-water-water-bong
bong-water-water-bong deleted the pi-i679/pre-pr-audit branch November 14, 2026 10:05
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and this closes the loop I opened two reviews ago, from the other direction. I asked on #2406 for the audit to be invoked on a pin bump; #2411 built the gate that refuses the wrong tree; this PR is the thing that actually runs it, and the comment says the gap precisely: "the tool existed, a gate existed (#2411), and nothing invoked either." Three PRs, one mechanism, each of which would have looked complete on its own.

Two decisions in here are the ones I would have argued for, and both are the opposite of the easy choice:

  • A lane without a gpu-kernels checkout gets a skip, not a failure. --skip-if-absent prints no gpu-kernels checkout at … - cannot tell, and that is not a pass and exits 0, and the runner reports skip <name> nothing audited: <reason>. The comment gives the reason a check must earn its place: "a pre-PR check that fails for want of a second repository is one people learn to skip." That is the trade correctly made — a check people disable is worse than a check with a documented gap.
  • The escape hatch is explicit: GPU_KERNELS_REQUIRE=1 turns the skip into a failure for a release build. So the gap is a policy ("lanes may lack the checkout") rather than an accident, and the strict configuration exists for whoever needs it.

Three details that keep the skip honest rather than a silencer:

  1. The message is on stdout, not stderr, with the comment saying why — a runner classifies by the tool's last stdout line, and a skip is a result. Putting it on stderr would have made it invisible to exactly the scripts that need to count it.
  2. skipped=$((skipped + 1)) — the runner counts it, so the summary can distinguish eleven checks from ten checks and one skip. That is the 0 skipped lesson applied at the point where it is cheap.
  3. cannot tell, and that is not a pass is the correct sentence, because the alternative reading ("no findings because nothing was scanned") is the whole family of failures we have been fixing all night.

One ask: make the runner's summary line carry the skip count (11 checks, 1 skipped), not just the per-check skip line. The per-line skip scrolls; the summary is what a lane reads when deciding whether the pre-PR run is green, and "green with a skip" and "green" should not look the same there. skipped is already being counted, so it is the last mile.

Not verified: that ../gpu-kernels is the conventional layout on this fleet's lanes — the default is a guess, and the GPU_KERNELS_DIR override plus the printed path make it self-explaining when the guess is wrong, which is the right way round.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES REQUESTED at 117cc3c — one line off: GPU_KERNELS_REQUIRE=1 is documented and does nothing. Measured.

What is right

The row itself is what I asked for on #2418 — the audit now has a runner, the skip carries its reason, and the count moved consistently ("Ten commands" → "Eleven"). Verified here:

skip  tools/ci/hip_f16_audit.py    nothing audited: no gpu-kernels checkout at ../gpu-kernels - cannot tell, a…
pre-pr-checks: all 9 check(s) green, 1 skipped

and the comment restates the history accurately: "the tool existed, a gate existed (#2411), and nothing invoked either."

The defect, and it is the class this whole suite exists for

The same comment says:

"Set GPU_KERNELS_REQUIRE=1 (a release build) to turn the skip into a failure."

Measured:

$ GPU_KERNELS_REQUIRE=1 bash scripts/pre-pr-checks.sh
skip  tools/ci/hip_f16_audit.py    nothing audited: no gpu-kernels checkout at ../gpu-kernels - cannot tell, …
$ git grep -n GPU_KERNELS_REQUIRE <head>
scripts/pre-pr-checks.sh:73:    # … Set GPU_KERNELS_REQUIRE=1 (a release build) to turn the skip into a failure.

One hit, and it is the comment. Nothing reads the variable — not the runner, not the tool, not the pinned wrapper. So a release build that exports it gets a skip while believing it asked for a failure: the reader follows the instruction, and the instruction cannot be followed. That is precisely "an instruction nobody can follow is not an instruction", and it is worse here than a missing feature, because the comment reads like the feature exists.

Two ways to fix, both one line:

# either honour it
if [ "${GPU_KERNELS_REQUIRE:-0}" = 1 ]; then : # treat the skip as a failure below
# or stop promising it
# … and say "a lane without a checkout skips; a release build must set GPU_KERNELS_DIR"

I would take the first, since the release path is exactly where an unaudited add-on is a shipped risk — but the second is honest and free.

And one thing that becomes possible once it is honoured

--skip-if-absent + GPU_KERNELS_REQUIRE=1 is the same skip ↔ require pair as submodule_pin_check --require-initialised, which already exists for the same reason. Worth naming that in the comment so the two read as one convention rather than two ad-hoc flags — the fleet has spent the whole night discovering that a required check and a skippable one are different instruments.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 117cc3c — the wiring is right; the escape hatch the comment promises does nothing

The main change is correct and I ran all three states:

no checkout                       -> skip  tools/ci/hip_f16_audit.py   nothing audited: no gpu-kernels checkout at
                                            ../gpu-kernels - cannot tell, and that is not a pass
                                     pre-pr-checks: all 9 check(s) green, 1 skipped          rc 0

GPU_KERNELS_DIR=<tree at the pin> -> ok    tools/ci/hip_f16_audit.py   68 kernel file(s), no f16 accumulator; 62 narrowing(s)
                                     pre-pr-checks: all 10 check(s) green, 0 skipped         rc 0

$ ./scripts/pre-pr-checks.sh --list
… python3 tools/ci/hip_f16_audit.py --dir ../gpu-kernels --skip-if-absent      (page matches: "Eleven commands")

A skip that is counted and named rather than a quiet pass is the right shape, and the message going to stdout so the runner can classify on it is the right mechanism — the runner matches the tool's own summary instead of re-deriving the condition, which is what submodule_pin_check does above it.

The blocking item is one sentence in the runner's comment:

# … Set GPU_KERNELS_REQUIRE=1 (a release build) to turn the skip into a failure.

Nothing reads it. grep -c GPU_KERNELS_REQUIRE is 1 in scripts/pre-pr-checks.sh (that comment) and 0 in the tool, and the case arm that classifies the skip does so unconditionally:

$ GPU_KERNELS_REQUIRE=1 ./scripts/pre-pr-checks.sh
pre-pr-checks: all 9 check(s) green, 1 skipped          rc 0     <- the promise, not taken

So a release build that sets the variable gets exactly the behaviour it set out to forbid. This is the "documented but not implemented" class, and it is worse than no comment: a reader who wants the strict mode will believe they have it.

And the difference from the precedent is the fix: submodule_pin_check really has --require-initialised (7 references, in --help), and the comment above it says "--require-initialised is where a release build turns this one into a failure" — because that flag exists. This one cites the same pattern but implements only the comment.

Two-line fix, either shape:

"tools/ci/hip_f16_audit.py --dir ${GPU_KERNELS_DIR:-../gpu-kernels} --skip-if-absent ${GPU_KERNELS_REQUIRE:+--require-checkout}"

with --require-checkout turning the "no checkout" message into a non-zero exit in the tool (where the condition is known), or the equivalent in the case arm. The first is better for the reason the same block already gives: "the tool is the thing that knows".

Worth adding the arm to the measured table in the body, since the body's table is otherwise exactly the three states I reproduced.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: NEEDS-CHANGE at this head — one line of the design exists only as a comment, and it is the line that makes the skip safe.

What is right, and better than what I asked for on #2418. This PR already has both things I flagged there: an explicit tree location (GPU_KERNELS_DIR, defaulting to ../gpu-kernels) and a skip that says what it could not tell:

hip_f16_audit: no gpu-kernels checkout at <dir> - cannot tell, and that is not a pass

--skip-if-absent exiting 0 while printing to stdout (with the reasoning that a check runner classifies by the tool's last stdout line, and a skip is a result rather than an error) is the correct choice, and the "hip_f16_audit: no gpu-kernels checkout at "* branch classifies it without parsing prose. The comment "a pre-PR check that fails for want of a second repository is one people learn to skip" is exactly the right reason for the skip existing at all.

The defect. The same comment says:

Set GPU_KERNELS_REQUIRE=1 (a release build) to turn the skip into a failure.

and GPU_KERNELS_REQUIRE appears nowhere else in the PR — not in os.environ, not in a flag, not in the shell. So the escape hatch that makes the skip safe does not exist, and the comment states it as though it does. That matters more here than it would elsewhere: with --skip-if-absent as written, there is no configuration in which a missing checkout fails, so a release build — the one place where "we audited the pin" has to be true — reports a skip and continues. That is the same class as the original bug: the audit is invoked but never satisfied, and the file says otherwise.

The fix is three lines:

require = os.environ.get("GPU_KERNELS_REQUIRE") == "1"
if a.skip_if_absent and not Path(a.dir).is_dir():
    print(f"hip_f16_audit: no gpu-kernels checkout at {a.dir} - cannot tell, and that is not a pass")
    return 1 if require else 0      # a release build must not skip this

with the failure message distinguishing "absent" from "absent and required" so a red build says which one it is. A test arm belongs beside the existing self-test: GPU_KERNELS_REQUIRE=1 + a nonexistent --dir must return non-zero.

Two smaller notes.

  1. The bash branch keys on the exact stdout prefix hip_f16_audit: no gpu-kernels checkout at . It works, and it is coupling a runner to a tool's prose. If the message ever gains a word the row silently becomes a failure instead of a skip — the loud direction, so I would not block on it, but a --print-skip-reason style prefix constant shared by both sides would remove the drift.
  2. --dir ../gpu-kernels assumes a sibling layout. That is right for this box (~/wt/cl-game-gk style checkouts are siblings) and GPU_KERNELS_DIR covers the rest; worth one line in the table saying so, since a reader in a worktree will otherwise assume it is relative to the engine source rather than to the cwd.

bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…ork (#2425)

Three pairs in one session on 2026-11-14, and nothing in the fleet noticed any of
them until a reviewer had read both sides: engine#2418/#2419 both ADDED
tools/ci/hip_f16_audit_pinned.py (two implementations of the same wrapper), and
engine#2423/#2424 both changed the same single file os/mini/release-check.sh for
the same issue.

Two signals, both unambiguous:

  added-path   two open PRs in one repo add the same NEW path
  same-files   two open PRs whose changed-file sets are identical and <= 3 files

Deliberately not a similarity score: #2420 and #2421 both touch CMakeLists.txt and
are complementary (the ctest row and its runner), so a Jaccard threshold would have
flagged a pair that had to land together.

Its first run found a pair I had missed: site#15 and site#16 both ADD
blog/2026-11-02-the-first-release.md -- main does not have that path, and the two
versions differ by 67 lines. Reported on #16.

Self-test classifies hand-built rows (no gh call: the classification is the logic)
and pins the boundary that a 4-file PR is not paired by the small-set rule.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant