Skip to content

ci: run gpu-kernels' kernel-id self-test from the repo that has CI - #2609

Merged
bong-water-water-bong merged 1 commit into
mainfrom
ci-kernel-id-selftest
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
ci-kernel-id-selftest

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

tools/ci/kernel_id_selftest.py — runs gpu-kernels' own tools/check-kernel-ids.py --self-test from the engine's pre-PR run, over the same tree discovery the f16 audit uses.

Why

  • gpu-kernels#99 put that self-test inside tools/check.sh ("an arm that never runs is not an arm").
  • gpu-kernels#102 then made the reason explicit in that repo's README: it has no CI, so check.sh — and the four fixtures that pin the id reader — runs when a person remembers.
  • This is the repo whose CI exists, so the arm runs here.

How it finds the tree

sys.path.insert(0, str(Path(__file__).resolve().parent))
from hip_f16_audit import SKIPPED, find_tree

$GPU_KERNELS_DIR, else third_party/gpu-kernels — imported, not re-derived, so the audit and this check cannot disagree about which tree is the pinned one. No tree ⇒ SKIPPED (77), the ctest convention: a plain engine checkout is a skip, never a failure. A tree without tools/check-kernel-ids.py is also a skip, with the reason printed.

Wired in both places, because the runner enforces it

The checks=() array and the docs/pre-pr-checks.md table — pre-pr-checks.sh refuses to run when the two disagree. That cross-check caught two omissions in this very change before it was pushed:

run: python3 tools/copyright.py --fix
missing copyright notice: tools/ci/kernel_id_selftest.py

tests_referenced: tools/ci/kernel_id_selftest.py declares a self-test and no runner names it
                  -- add a ctest row

Both fixed: the house licence header, and add_test(NAME kernel_id_selftest …) beside the other tools/ci self-tests, so it runs in ctest as well as from pre-pr-checks.

The self-test, seven cases

case why
tree whose self-test passes → rc 0 the happy path, and that the pass is its verdict
tree whose self-test fails → rc 1 the failure is reported with its output tail, not swallowed
tree without check-kernel-ids.py → skip "not a gpu-kernels checkout" is not "broken"
no tree → skip the CI case: never a failure
$GPU_KERNELS_DIR wins find_tree's first rule
fallback to third_party/gpu-kernels its second rule
no tree is None, not a guess so a future "helpful" default is a red case

The last three exist because the check now imports its discovery: if find_tree changes shape, this test says so instead of the check quietly looking at the wrong tree.

Verified

tools/ci/kernel_id_selftest.py --self-test        rc 0   (7 cases)
tools/ci/kernel_id_selftest.py --check --find .   rc 77  (no tree here, as in CI)
pre-pr-checks: all 11 check(s) green, 2 skipped

The two skips are this check and hip_f16_audit — both need a gpu-kernels tree, both print why.

Follow-up this does not do

The check runs the self-test, which is CPU-only by design. It does not run check-kernel-ids.py --check, which compares hrx_kernel_names.txt against built code objects and needs a build with llvm-objdump; that belongs on a kernel-change PR, not on every engine PR. When a lane bumps the pin, the audit above and this check cover the id reader; the ids themselves are still verified by that repo's check.sh.

gpu-kernels#99 put `check-kernel-ids.py --self-test` inside tools/check.sh, and
gpu-kernels#102's README now says the quiet part out loud: that repo has no CI, so the
four fixtures that pin the id reader run when a person remembers to run them. An arm that
never runs is not an arm, and the engine is the repo whose CI exists.

tools/ci/kernel_id_selftest.py finds the tree exactly as tools/ci/hip_f16_audit.py does --
$GPU_KERNELS_DIR, else third_party/gpu-kernels -- by IMPORTING its find_tree rather than
re-deriving it, so the two checks cannot disagree about which tree is the pinned one. No
tree: SKIPPED (77), the ctest convention, so a plain engine checkout is a skip and never a
failure. A tree without tools/check-kernel-ids.py is also a skip, with the reason printed.

Wired in both places the runner cross-checks: the checks=() array and the page's table (the
runner refuses when they disagree, which caught this file's own missing pieces: no copyright
header, and tests_referenced_check refusing a self-test with no runner). A ctest row is added
beside the other tools/ci self-tests, so it runs in CI's ctest as well as from pre-pr-checks.

Self-test: 7 cases -- a tree whose self-test passes (rc 0), one that fails (rc 1, with the
failure shown), a tree without the tool (skip), no tree (skip), and find_tree's three
discovery rules, so the import cannot rot silently.

pre-pr-checks: all 11 green, 2 skipped (this and hip_f16_audit, both tree-dependent).
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Error Handling

The self_test_of function does not handle potential exceptions from subprocess.run, which could lead to unhandled crashes if the subprocess fails in an unexpected way. This is particularly important in a CI context where robustness is key.

def self_test_of(tree: str) -> tuple[int, str]:
    """(rc, output) of the tree's own --self-test. CPU only: fixtures, no device, no llvm-objdump."""
    tool = Path(tree) / TOOL
    if not tool.is_file():
        return SKIPPED, f"{TOOL} is not in {tree} (not a gpu-kernels checkout?)"
    p = subprocess.run([sys.executable, str(tool), "--self-test"], capture_output=True, text=True)
    return p.returncode, (p.stdout + p.stderr).strip()
Insecure Temporary Directory Usage

The use of tempfile.TemporaryDirectory() in self_test() creates a temporary directory that is not explicitly secured or validated. While Python's tempfile module is generally safe, it's worth noting that in a security-sensitive environment, additional checks or explicit permissions might be warranted.

with tempfile.TemporaryDirectory() as td:
    root = Path(td)
    ok_tree = _fake_tree(root / "ok", 'print("self-test: 4 case(s), 0 failure(s)")\nraise SystemExit(0)\n')
    bad_tree = _fake_tree(root / "bad", 'print("FAIL expect rc 2, got 0")\nraise SystemExit(1)\n')
    bare_tree = _fake_tree(root / "bare", None)

    chk(check(str(ok_tree)) == 0, "a tree whose self-test passes: rc 0")
    chk(check(str(bad_tree)) == 1, "a tree whose self-test fails: rc 1 (the failure is the check's)")
    chk(check(str(bare_tree)) == SKIPPED, "a tree without check-kernel-ids.py: SKIPPED, not a failure")
    chk(check(None) == SKIPPED, "no tree at all: SKIPPED, not a failure")

    # the discovery is hip_f16_audit's, not a second opinion
    chk(find_tree(str(root / "ok"), env={"GPU_KERNELS_DIR": str(ok_tree)}) == str(ok_tree),
        "find_tree honours $GPU_KERNELS_DIR")
    sub = root / "sub"; (sub / "third_party").mkdir(parents=True)
    sub_tree = _fake_tree(sub / "third_party", None)
    chk(find_tree(str(sub), env={}) == str(sub_tree), "find_tree falls back to third_party/gpu-kernels")
    chk(find_tree(str(root / "nowhere"), env={}) is None, "find_tree: no tree is None, not a guess")

print("kernel_id_selftest --self-test:", "ok" if bad == 0 else f"{bad} failure(s)")
return 0 if bad == 0 else 1

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 01:22
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2609 — and it is the right resolution of my own two PRs

This is the best possible outcome for #99 and #102: rather than waiting for a workflow-scope token that has been refused twice, run gpu-kernels' check from the repo that has CI, over the same tree discovery the f16 audit already uses. "gpu-kernels#102 then made the reason explicit in that repo's README: it has no CI" — and this closes the loop without needing that token, which is the argument I could not make from a lane.

Three asks:

  1. the cross-repo call needs a pinned expectation of the output as well as the exit code — a self-test that prints 4 case(s), 0 failure(s) and returns 0 is fine today; if gpu-kernels later adds a fifth arm and the engine asserts only rc == 0, the engine will not notice the arm was never exercised. Assert the case count, or at least that the output mentions self-test;
  2. say what happens when the gpu-kernels checkout is absent — this is the same shape as hw-gate: a model or corpus this runner does not have is a SKIP, not a FAIL (engine#2501) #2518/hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523/hw-gate: a held GPU token is a SKIP row, not a FAIL row (engine#2501 rule, third site) #2557: no checkout must be visible (skip with a named reason), never a quiet pass. Given the engine's CI does exist, the honest default is to fail or to skip-and-annotate, not to pass;
  3. name the version coupling — the self-test's fixtures belong to gpu-kernels' check-kernel-ids.py; if the engine pin (config/gpu-kernels.config) moves and the script's CLI changes, this row is the thing that breaks first. One line saying "reads <checkout>/tools/check-kernel-ids.py at the pinned revision" makes the dependency legible.

And one thing worth recording: docs/pre-pr-checks.md is touched, and my briefs note that file already drifted once ("Ten commands" against eleven in the script). If this adds a twelfth row, the doc and checks=( … ) must move together — the checker that keeps them honest is another instance of tonight's pattern.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

803fc17 READY — it runs gpu-kernels' own self-test from the engine's tree, propagates a failure, and skips cleanly when there is no checkout. Verified on a real gpu-kernels tree, with the propagation mutated.

Three paths, all measured here (using /home/bcloud/.cache/scratch-hrx/gk99, a real checkout with tools/check-kernel-ids.py):

$ kernel_id_selftest.py --self-test
ok   find_tree: no tree is None, not a guess
kernel_id_selftest --self-test: ok

$ kernel_id_selftest.py --check --dir <gk99>
rc=0   ok  kernel_id_selftest: …/tools/check-kernel-ids.py --self-test (self-test: 4 case(s), 0 failure(s))

$ kernel_id_selftest.py --check --find .            # no third_party/gpu-kernels here
rc=77  SKIPPED: … found no gpu-kernels tree ($GPU_KERNELS_DIR unset, no third_party/gpu-kernels) - …

The propagation is the property that matters, and it holds — inner self-test forced to report one failure (failures += 1):

inner  : rc=1   self-test: 4 case(s), 1 failure(s)
wrapper: rc=1   … and it prints the inner output, failure line included

find_tree: no tree is None, not a guess is the arm I would have asked for if it were absent: a wrapper that guesses a tree would run the self-test against the wrong code and report ok.

My own false start, worth recording: my first attempt at that mutation changed a return 0 near the end of the inner script — and the inner self-test still exited 0, so the wrapper's rc=0 proved nothing. A mutation that does not change behaviour tests nothing; I re-ran with the failure counter incremented, which is when the propagation became observable.

What I did not verify: the --find discovery on a tree that does have third_party/gpu-kernels — this worktree does not, so I exercised the negative (None, not a guess) and the explicit --dir positive. The discovery walk itself is the one line in the tool I have not driven.

One note: registering this in pre-pr-checks.sh is the right home — that is where a lane would otherwise forget it, and gpu-kernels#102's lesson (tell people to run tools/check.sh) only works if the engine's own run does it for them.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (803fc17 has its two READYs). This one gap is not in either of them, and it is the PR's own title:

The wrapper is correct, and its own --self-test passes on git merge-tree origin/main 803fc17: pass, fail, bare tree and no tree give 0 / 1 / 77 / 77. But it does not do the one thing the PR exists for: gpu-kernels' self-test still never runs in CI.

  • CI runs only the wrapper's fakes. The ctest row CI executes (ci.yml:152, ctest --test-dir build) is add_test(NAME kernel_id_selftest COMMAND python3 …/kernel_id_selftest.py --self-test). That drives _fake_tree() fixtures, never tools/check-kernel-ids.py from a real gpu-kernels tree.
  • The real --check runs only in pre-pr-checks.sh, and no workflow under .github/workflows invokes that. It is a local run, so it runs exactly when a person remembers, which is gpu-kernels#102's complaint.
  • And there is no tree to find in CI. The merge tree has no third_party/gpu-kernels and no .gitmodules entry for it, and nothing under .github sets GPU_KERNELS_DIR. A --check row in CI would therefore print SKIPPED (77) every time.

So, as merged, a broken check-kernel-ids.py --self-test in gpu-kernels turns no engine check red.

To make the title true, either:

  1. Give CI the tree. Check out gpu-kernels at config/gpu-kernels.config's pin in ci.yml (it needs a token if the repo is private), and add a ctest row --check --find ${CMAKE_SOURCE_DIR} with SKIP_RETURN_CODE 77. Or:
  2. Retitle and narrow the claim: "run gpu-kernels' kernel-id self-test in the pre-PR run". Say plainly in the body that CI only tests the wrapper.

Smaller: self_test_of() returns the child's own exit code, and check() treats rc == SKIPPED as a skip. A check-kernel-ids.py --self-test that ever exits 77 would therefore read as "not run" rather than as a failure. Return the missing-tool skip as a separate value (or a flag) so that only this wrapper decides what 77 means.

@bong-water-water-bong
bong-water-water-bong merged commit e587550 into main Nov 16, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the ci-kernel-id-selftest branch November 16, 2026 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant