From 5acb6e7a1f26eb73fb7dd492d2360348b4d754d8 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Tue, 10 Nov 2026 23:58:23 -0400 Subject: [PATCH 01/28] bump-hrx: a mirror hold, so a rewrite of AMD's line that needs a hand merge is reported, not failed daily (Refs #1945) (#2110) config/llama-mirror-hold.config: while it names the mirror's commit, the sync reports the hold, moves nothing and succeeds. The run fails, naming the file, when the hold is stale (the mirror moved) or unneeded (AMD's pin contains the held commit again). The shipped file names no commit: it holds nothing until the owner adds the line. Co-authored-by: bong-water-water-bong Co-authored-by: Claude Fable 5.2 --- config/llama-mirror-hold.config | 33 +++++++++++++++++++++ docs/hrx.md | 12 ++++++++ scripts/bump-hrx-sync.sh | 51 +++++++++++++++++++++++++++++++++ tests/bump_hrx_sync_test.sh | 31 ++++++++++++++++++++ 4 files changed, 127 insertions(+) create mode 100644 config/llama-mirror-hold.config diff --git a/config/llama-mirror-hold.config b/config/llama-mirror-hold.config new file mode 100644 index 000000000..e53755909 --- /dev/null +++ b/config/llama-mirror-hold.config @@ -0,0 +1,33 @@ +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# The commit of 1bit-MONSTER/llama.cpp `1bit/hrx-vulkan` (our mirror of AMD's HRX line) that the daily +# bump-hrx run must not move away from. While a commit is named here, bump-hrx does not move the mirror, +# does not merge AMD's llama.cpp pin into `1bit/hrx-vulkan-patched`, reports the hold in the run summary and +# succeeds. Delete the commit line to lift the hold. +# +# The hold is for one situation only: AMD rewrote its line, so its pin no longer descends from the mirror, +# and taking it is a hand merge that has not been done yet. The run FAILS, naming this file, when the file +# is out of date (the mirror is no longer the named commit) or no longer needed (AMD's pin contains the +# named commit again), so a hold cannot outlive the state it describes. +# +# No commit is named below, so nothing is held. Naming one is the owner's decision. The state it would +# record, as of 2026-11-11 (engine#1945): +# mirror 1bit/hrx-vulkan 44ee5710d19f7f3af79b99bbcbe5fd13979d1f67 (AMD's line before its rebase) +# AMD pin 9bf5871310a8d7de74f4cb0c57009835fbf91c27 (the rebased line, 1777 upstream commits newer) +# merge into 1bit/hrx-vulkan-patched, against the last AMD commit our line took (43b0ec2c41cc): +# 149 conflicted files, 538 hunks (65 files / 83 hunks in the HRX backend) +# To hold, put the mirror's commit on a line of its own: +# 44ee5710d19f7f3af79b99bbcbe5fd13979d1f67 diff --git a/docs/hrx.md b/docs/hrx.md index 0b1ebecee..2faf2db4d 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -98,6 +98,18 @@ fails a pin that leaves a gate model's node off HRX. `ggml/src/ggml-hrx` keep our side; a conflict anywhere else fails the run and names the files; - merges AMD's hrx-system pin into our `1bit/main` the same way; - opens a PR here moving the submodules, listing the llama.cpp commits new to the engine. +- **The mirror hold.** A rewrite of AMD's line that the rules above cannot merge leaves the daily run with + nothing to do but fail, every day, until somebody does the merge by hand. While + [`config/llama-mirror-hold.config`](../config/llama-mirror-hold.config) names the mirror's commit, the run + instead reports the hold in its summary (`mirror 1bit/hrx-vulkan HELD at …; AMD's pin is …, on a line that + does not contain it`), moves nothing and succeeds; the engine's pin still follows our own line. The hold + describes one state and fails the run when that state ends, so it cannot be forgotten: **stale** when the + mirror is no longer the named commit, **unneeded** when AMD's pin contains the named commit again (an ordinary + sync would work). `scripts/bump-hrx-sync.sh llama-hold ` prints the decision alone + (`none`, `held`, `stale`, `unneeded`). The file ships naming no commit, so it holds nothing; naming one is + the owner's decision. It was added for AMD's 2026-11-09 rebase (engine#1945): taking that pin is a merge of + 149 conflicted files (538 hunks) into `1bit/hrx-vulkan-patched`, measured against the last AMD commit our + line took. - **The hold.** While [`config/hrx-system-hold.config`](../config/hrx-system-hold.config) names a commit, the run does not merge AMD's hrx-system, keeps `third_party/hrx-system` where it is, and takes only AMD's llama.cpp changes outside the HRX backend paths diff --git a/scripts/bump-hrx-sync.sh b/scripts/bump-hrx-sync.sh index c8beae0a9..3e9e1e80f 100755 --- a/scripts/bump-hrx-sync.sh +++ b/scripts/bump-hrx-sync.sh @@ -34,6 +34,8 @@ # A mirror commit that is on no AMD branch fails the run and is named, as one of two # things (off_branch_commits): AMD's own commit from a line AMD rebased and dropped, to be # merged by hand, or a commit that is not AMD's, to be moved to our line. +# - the mirror is held (config/llama-mirror-hold.config names its commit): the run reports the hold and +# moves nothing. For a rewrite that cannot be merged by rule (engine#1945); see llama_hold_decision. # - hrx-system is held (config/hrx-system-hold.config). AMD's ggml-hrx is written against # AMD's hrx-system (e.g. #139 moved the kernel corpus to the `multiple_of` predicate that # our held hrx-system cannot parse), so only AMD's changes OUTSIDE the HRX backend paths @@ -48,6 +50,7 @@ set -euo pipefail cmd=${1:-} DRY_RUN=${DRY_RUN:-0} HOLD_FILE=${HOLD_FILE:-config/hrx-system-hold.config} +LLAMA_HOLD_FILE=${LLAMA_HOLD_FILE:-config/llama-mirror-hold.config} ENGINE_DIR=$(pwd) # The HRX backend paths of the llama.cpp tree (what AMD's hrx-graph line adds on top of ggml). # tests/CMakeLists.txt belongs here: it is where the HRX test targets are registered, so it moves @@ -85,6 +88,34 @@ hold_sha() { grep -v '^[[:space:]]*#' "$ENGINE_DIR/$HOLD_FILE" | grep -oE '^[0-9a-f]{40}' | head -n1 || true } +# The commit 1bit/hrx-vulkan (our mirror of AMD's line) is held at, or empty. Same file shape as the hrx-system hold. +llama_hold_sha() { + local f=$LLAMA_HOLD_FILE + case "$f" in /*) ;; *) f=$ENGINE_DIR/$f ;; esac # relative to the engine checkout unless given whole + [ -f "$f" ] || return 0 + grep -v '^[[:space:]]*#' "$f" | grep -oE '^[0-9a-f]{40}' | head -n1 || true +} + +# What a mirror hold means for this run. Reads repo $1; $2 is the held commit (may be empty), $3 the mirror tip, +# $4 AMD's pin. One word: +# none no commit is held: the sync runs as usual +# held the mirror is the held commit and AMD's pin does not descend from it: report it, move nothing +# stale a commit is held but the mirror is somewhere else: somebody moved the mirror, the file is out of date +# unneeded the mirror is the held commit and AMD's pin contains it: an ordinary fast-forward would work +# 2026-11-10 (engine#1945): AMD re-committed its HRX series on a base 1777 upstream commits newer. Taking that +# pin is a merge of about 150 conflicted files into our line, which is days of work for the lanes that own +# those files, and until it is done the daily run can only fail. A failure that is known, sized and assigned +# is not news; a hold says so in the run's summary and lets the run stay green for everything else it checks. +# The hold is deliberately narrow: `stale` and `unneeded` both FAIL the run, so the file cannot outlive the +# state it describes -- the day AMD's pin is reachable again, or the mirror moves, the run says to delete it. +llama_hold_decision() { + local repo=$1 hold=$2 mirror=$3 pin=$4 + if [ -z "$hold" ]; then echo none; return; fi + if [ "$hold" != "$mirror" ]; then echo stale; return; fi + if [ "$mirror" = "$pin" ] || git -C "$repo" merge-base --is-ancestor "$mirror" "$pin" 2>/dev/null; then echo unneeded; return; fi + echo held +} + # Why a mirror commit is on no AMD branch. Reads repo $1 (commits only are enough), whose refs/amd/* # are AMD's branches; $2 is the mirror tip and $3 AMD's new pin. One line per such commit: # rebased the new pin holds a commit with the same author, author date and @@ -128,6 +159,24 @@ sync_llama() { patched=$(git rev-parse refs/remotes/fork/patched) echo "AMD pin $pin, mirror 1bit/hrx-vulkan $mirror, our line 1bit/hrx-vulkan-patched $patched" + # 0. A mirror hold (config/llama-mirror-hold.config): the run reports it and moves nothing. + local lhold; lhold=$(llama_hold_sha) + case "$(llama_hold_decision . "$lhold" "$mirror" "$pin")" in + stale) + die "$LLAMA_HOLD_FILE holds 1bit/hrx-vulkan at ${lhold:0:12}, but the mirror is ${mirror:0:12}: the hold is out of date. Name the mirror's commit in it, or delete the commit line to let the sync run." ;; + unneeded) + die "$LLAMA_HOLD_FILE holds 1bit/hrx-vulkan at ${lhold:0:12}, and AMD's pin ${pin:0:12} contains that commit: an ordinary sync would work. Delete the commit line." ;; + held) + note "1bit/hrx-vulkan is held at ${mirror:0:12} by $LLAMA_HOLD_FILE: AMD's pin ${pin:0:12} does not descend from it (AMD rewrote its line). Nothing was moved; the merge is tracked where the hold file says." + summary "- llama.cpp: mirror \`1bit/hrx-vulkan\` HELD at \`${mirror:0:12}\` ($LLAMA_HOLD_FILE); AMD's pin is \`${pin:0:12}\`, on a line that does not contain it. Not merged." + output "rebased=$patched" + output "llama_held=1bit/hrx-vulkan HELD at \`${mirror:0:12}\` by \`$LLAMA_HOLD_FILE\` (AMD's pin is \`${pin:0:12}\`)." + output "patches</dev/null || true) +EOF" + return 0 ;; + esac + # 1. The mirror. It holds AMD's commits only; ours live on 1bit/hrx-vulkan-patched. local rewritten=0 if [ "$mirror" != "$pin" ] && ! git merge-base --is-ancestor "$mirror" "$pin"; then @@ -293,5 +342,7 @@ case "$cmd" in hrx) sync_hrx ;; # For tests/bump_hrx_sync_test.sh: the classification alone, on a local repository, no network. off-branch) off_branch_commits "${2:?repo}" "${3:?mirror}" "${4:?pin}" ;; + # The mirror-hold decision alone, on a local repository: llama-hold , hold file from LLAMA_HOLD_FILE. + llama-hold) llama_hold_decision "${2:?repo}" "$(llama_hold_sha)" "${3:?mirror}" "${4:?pin}" ;; *) echo "usage: $0 llama|hrx" >&2; exit 2 ;; esac diff --git a/tests/bump_hrx_sync_test.sh b/tests/bump_hrx_sync_test.sh index 4108db8ce..4191ec487 100755 --- a/tests/bump_hrx_sync_test.sh +++ b/tests/bump_hrx_sync_test.sh @@ -102,5 +102,36 @@ other=$(g rev-parse HEAD) expect "the same subject and author at another author date is foreign" "$(run "$other" "$pin" | head -n1)" \ "foreign $(g rev-parse --short "$other") hrx: something new" +# ---- the mirror hold (config/llama-mirror-hold.config): what a named commit means for a run ------------------- +holdf=$work/hold.config +hold() { LLAMA_HOLD_FILE=$holdf bash "$script" llama-hold "$repo" "$1" "$2"; } +# a pin that descends from the mirror: build one +g checkout -q --detach "$mirror" +commit "$amd" "$amd_mail" 2026-11-10T00:00:00Z 2026-11-10T00:00:00Z "hrx: one more on the old line" +desc=$(g rev-parse HEAD) + +rm -f "$holdf" +expect "no hold file: the sync runs as usual" "$(hold "$mirror" "$pin")" "none" +printf '# a comment\n# %s\n' "$mirror" > "$holdf" +expect "a commit that is only mentioned in a comment holds nothing" "$(hold "$mirror" "$pin")" "none" +printf '# held\n%s\n' "$mirror" > "$holdf" +expect "the mirror is the held commit and AMD's pin is on a rewritten line: held" "$(hold "$mirror" "$pin")" "held" +expect "the mirror has moved away from the held commit: the file is stale" "$(hold "$ours" "$pin")" "stale" +expect "AMD's pin contains the held commit again: the hold is not needed" "$(hold "$mirror" "$desc")" "unneeded" +expect "the pin IS the held commit: not needed either" "$(hold "$mirror" "$mirror")" "unneeded" +printf 'not-a-sha\n%s extra words\n' "$mirror" > "$holdf" +expect "the commit line may carry a trailing note; a line that is not a commit is ignored" "$(hold "$mirror" "$pin")" "held" +printf '%s\n' "${mirror:0:12}" > "$holdf" +expect "a short hash is not a hold: forty hex digits or nothing" "$(hold "$mirror" "$pin")" "none" + +# The file shipped in the tree names no commit, so merging it holds nothing. +shipped=$(cd "$(dirname "$script")/.." && pwd)/config/llama-mirror-hold.config +if [ -f "$shipped" ]; then + expect "the shipped hold file holds nothing until a commit line is added" \ + "$(LLAMA_HOLD_FILE=$shipped bash "$script" llama-hold "$repo" "$mirror" "$pin")" "none" +else + fail "config/llama-mirror-hold.config is in the tree" +fi + if [ "$fails" -ne 0 ]; then echo "$fails check(s) failed"; exit 1; fi echo "bump_hrx_sync: ok" From 0f68f1ef1d11386107628f0a65539bfd40151452 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Tue, 10 Nov 2026 23:58:29 -0400 Subject: [PATCH 02/28] weekly: the runner preflight warns on a short restart policy and blocks only on a dead runner (Refs #1504) (#2109) ci-runner-health.sh exits 3 (not 1) when every runner unit is accepting jobs but a restart policy would leave one down after the next OOM kill, and prints those lines as WARN. weekly.sh's preflight blocks on exit 1 and warns on exit 3 (owner decision on #1504, 2026-11-11). A missing ceiling alone no longer reads 'stops for good'. docs/ci-runners.md carries the two drop-in files and the three owner commands (scripts/ci-runner-restart.conf), and says that exit 143 in a build step is the memory guard. tests/weekly_runner_preflight_test.sh runs the real health script and the real preflight function against a stub systemctl. Co-authored-by: bong-water-water-bong Co-authored-by: Claude Fable 5.2 --- CMakeLists.txt | 4 ++ docs/ci-runners.md | 75 ++++++++++++++++++++++- docs/hw-gate.md | 4 +- scripts/ci-runner-health.sh | 28 +++++++-- scripts/ci-runner-restart.conf | 29 +++++++++ scripts/weekly.sh | 42 +++++++------ tests/weekly_runner_preflight_test.sh | 88 +++++++++++++++++++++++++++ 7 files changed, 245 insertions(+), 25 deletions(-) create mode 100644 scripts/ci-runner-restart.conf create mode 100755 tests/weekly_runner_preflight_test.sh diff --git a/CMakeLists.txt b/CMakeLists.txt index 610142905..ce9aeeed1 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -374,6 +374,10 @@ add_test(NAME pack_version COMMAND ${CMAKE_SOURCE_DIR}/tests/pack_version_test.s # arm asserts the row's message, and a clean tree is the control. No package build, no GPU, runs in CI. add_test(NAME pack_check COMMAND ${CMAKE_SOURCE_DIR}/tests/pack_check_test.sh) add_test(NAME check_ci_runners COMMAND ${CMAKE_SOURCE_DIR}/tests/check_ci_runners_test.sh) +# engine #1504: the release preflight blocks on a runner unit that is down and only warns on a short restart policy. +# The real scripts/ci-runner-health.sh and weekly.sh's preflight_runner_units() against a stub systemctl; no systemd. +add_test(NAME weekly_runner_preflight COMMAND ${CMAKE_SOURCE_DIR}/tests/weekly_runner_preflight_test.sh) +add_test(NAME ci_runner_health_selftest COMMAND bash ${CMAKE_SOURCE_DIR}/scripts/ci-runner-health.sh --self-test) # engine #1750: the carried amdxdna series applies in order to the pinned driver, and the context-restart code it # produces fails a context's jobs (and logs it) when the restart fails, instead of leaving them parked or running # them at close on a half-built firmware context. The driver's own functions in a user-space harness: no NPU, no diff --git a/docs/ci-runners.md b/docs/ci-runners.md index f3195c82a..107760817 100644 --- a/docs/ci-runners.md +++ b/docs/ci-runners.md @@ -306,7 +306,9 @@ here: a new workflow changes what the repository's token may do, which is an RFC ## Two rows, one question: are the runners healthy? Since #1682 and #1546 both landed there are two checks, and neither alone is enough — each is blind to what the -other sees. Both run in `scripts/weekly.sh`'s preflight, so a release refuses to start on a runner it cannot trust. +other sees. Both run in `scripts/weekly.sh`'s preflight. A runner that is **not accepting jobs** stops the release +before it starts. A runner that is up but whose **restart policy is short** is a warning, and the release goes on +(owner decision on engine#1504, 2026-11-11): see [the next section](#a-short-restart-policy-warns-it-does-not-block). | | `scripts/check-ci-runners.sh` | `scripts/ci-runner-health.sh` | |---|---|---| @@ -315,7 +317,7 @@ other sees. Both run in `scripts/weekly.sh`'s preflight, so a release refuses to | notices | a registration that was revoked, expired or never made | `Result=oom-kill`, a restart window, a unit that stopped | | cannot see | *why* it died — no `oom-kill`, no restart count | a unit that is running fine while its registration is gone | | speed | after the heartbeat times out | the moment the unit dies | -| exit | 0 online · 1 an offender, named · **2 cannot tell** (unreadable or truncated list) | 0 all accepting · 1 one is not · 2 no such units on this host | +| exit | 0 online · 1 an offender, named · **2 cannot tell** (unreadable or truncated list) | 0 all accepting, policy right · 1 one is not accepting jobs · 2 no such units on this host · **3 all accepting, a restart policy is short** | Run either by hand — both are read-only and neither restarts anything: @@ -335,3 +337,72 @@ rather than report green. One case neither row covers: a runner that is `online` in the API *and* active locally, but wedged so it takes no jobs. That is a hung process, not a registration or unit state; the practical signal for it is a queue that stops moving while both rows stay green. + +## A short restart policy warns, it does not block + +`scripts/ci-runner-health.sh` prints two lines per unit: its state (`PASS`/`FAIL`) and its restart policy +(`PASS`/`WARN`). The exit code keeps them apart, because they call for different things: + +| exit | meaning | `scripts/weekly.sh` preflight | +|---|---|---| +| 0 | every unit is accepting jobs and would come back after an OOM kill | passes | +| 1 | a unit is not accepting jobs now (dead, or in its restart window) | **blocks**: the checks after the release would queue for ever | +| 2 | no `actions.runner.*` units on this host | a note | +| 3 | every unit is accepting jobs, but a `WARN` line names a policy setting | **warns** and goes on | + +A unit that is down outranks a policy line, so 1 wins over 3. The warning in the release log reads +`preflight: WARNING, not blocking: every runner unit is accepting jobs, but a restart policy would leave one down +after the next OOM kill`, with the `WARN` lines above it naming the unit and the setting. + +Why a warning is enough: the policy is a box configuration change, which the owner applies (below), and a release +that cannot start does not make the unit any safer. Meanwhile the heavy builds run under +`scripts/mem-guard.sh --term`, which ends the build before the kernel's OOM killer reaches the runner. **A build step +that ends with exit 143 was stopped by that guard**: the log has a `GUARD: MemAvailable N GB < floor GB` line, the +runner is still alive, and the job is re-run once memory frees. The guard covers only the steps that use it. A server +or a bench started without it can still take the unit down, which is why the drop-ins below are the real fix. + +`tests/weekly_runner_preflight_test.sh` (ctest `weekly_runner_preflight`) runs the real health script and the real +`preflight_runner_units()` from `scripts/weekly.sh` against a stub `systemctl`: the four exit codes, block on a dead +unit, warn on a short policy. + +### The two drop-ins (owner step) + +On strixhalo, `strixhalo-bench` and `strixhalo-org` still run the units that predate `scripts/install-ci-runner.sh`: +`Restart=no`, `OOMPolicy=stop`. One OOM kill of any process in the unit stops it for good. `strixhalo-engine` got +the same drop-in on 2026-11-10. This needs `sudo`, so it is the owner's to run. It does not re-register a runner, +change its user, or stop a job in flight. + +Both files have the same content, [`scripts/ci-runner-restart.conf`](../scripts/ci-runner-restart.conf): + +```ini +# /etc/systemd/system/actions.runner.1bit-MONSTER-engine.strixhalo-bench.service.d/restart.conf +# /etc/systemd/system/actions.runner.1bit-MONSTER.strixhalo-org.service.d/restart.conf +[Unit] +StartLimitIntervalSec=0 + +[Service] +Restart=always +RestartSec=15 +OOMPolicy=continue +``` + +The three commands, from the root of an engine checkout: + +```sh +sudo install -Dm644 scripts/ci-runner-restart.conf /etc/systemd/system/actions.runner.1bit-MONSTER-engine.strixhalo-bench.service.d/restart.conf +sudo install -Dm644 scripts/ci-runner-restart.conf /etc/systemd/system/actions.runner.1bit-MONSTER.strixhalo-org.service.d/restart.conf +sudo systemctl daemon-reload +``` + +Check it took (read-only, no `sudo`): + +```sh +systemctl show actions.runner.1bit-MONSTER-engine.strixhalo-bench.service actions.runner.1bit-MONSTER.strixhalo-org.service -p Id -p Restart -p OOMPolicy +scripts/ci-runner-health.sh; echo "exit $?" +``` + +Want `Restart=always` and `OOMPolicy=continue` on both. The health script still exits 3 afterwards, with one `WARN` +per unit naming only `MemoryMax=infinity(want a ceiling)`: the drop-in sets no memory ceiling. The ceiling this page +documents (`MemoryHigh=43G`, `MemoryMax=48G`) comes with a full `scripts/install-ci-runner.sh` install, or with +`sudo systemctl set-property MemoryHigh=43G MemoryMax=48G` per unit. To undo a drop-in, delete the +`restart.conf` and run `sudo systemctl daemon-reload`. diff --git a/docs/hw-gate.md b/docs/hw-gate.md index 1d7afdeef..67402e213 100644 --- a/docs/hw-gate.md +++ b/docs/hw-gate.md @@ -72,7 +72,9 @@ hrx-system) through `cmake/hrx.cmake`, then links `onebit` with its HRX wiring. - **Memory.** The box is shared, and on Strix Halo GTT is host RAM. Before it compiles, the job waits for 24 GB of `MemAvailable` (`scripts/mem-wait.sh`, polled once a minute for up to 15 minutes). If memory doesn't free up, the job fails with the reading, and a re-run is the fix. The compile itself runs under `scripts/mem-guard.sh --term 12`, - which stops the whole `thermal-run` scope if `MemAvailable` falls below 12 GB. On 2026-11-08 both self-hosted + which stops the whole `thermal-run` scope if `MemAvailable` falls below 12 GB. **Exit 143 in a build step is + that guard, not a compiler failure**: the step log has a `GUARD: MemAvailable N GB < 12 GB` line just above it, the + runner is still alive, and a re-run once memory frees is the fix. On 2026-11-08 both self-hosted runners were OOM-killed, the second time 76 s after a restart handed this job a compile (engine#1504). - **What it is not.** It is a compile and link check of the public plumbing only. It does not build the private HIP add-on (the gate builds that) and does not dispatch anything. Green means the pinned pair builds into the engine. It diff --git a/scripts/ci-runner-health.sh b/scripts/ci-runner-health.sh index 20f47c874..0cd0fa2cf 100755 --- a/scripts/ci-runner-health.sh +++ b/scripts/ci-runner-health.sh @@ -30,8 +30,12 @@ # scripts/ci-runner-health.sh [...] # named units, e.g. for a specific runner # scripts/ci-runner-health.sh --self-test # the verdict logic, with systemctl stubbed # -# Exit: 0 when every runner is active/running with no failure result; 1 when any is not; 2 when there is nothing to inspect -# (no such units on this host, so a silent pass would be the defect). +# Exit: 0 when every runner is active/running with a restart policy that survives an OOM kill; 1 when a runner is not +# accepting jobs NOW; 2 when there is nothing to inspect (no such units on this host, so a silent pass would be the defect); +# 3 when every runner is accepting jobs but a restart policy would leave one down after the next OOM kill. 1 and 3 are +# separate because they call for different things (engine#1504, owner 2026-11-11): a dead runner blocks a release, since +# its checks would queue for ever, while a policy line is a box configuration change the owner applies +# (docs/ci-runners.md "The two drop-ins"), so scripts/weekly.sh warns on 3 and goes on. The policy lines print as WARN. set -uo pipefail # -> one verdict line. The state checks above answer "is it alive", which is what the @@ -53,6 +57,13 @@ policy_verdict() { # an infinite ceiling means the cgroup never hits its own limit, so the box's OOM killer chooses - and the bench runner # is the unit whose jobs are memory-heavy by definition. [ -n "${_mmax:-}" ] && [ "$_mmax" != infinity ] || _bad="$_bad MemoryMax=${_mmax:-?}(want a ceiling);" + case "$_bad" in + "") ;; + *Restart=*|*OOMPolicy=*) ;; + # Restart and OOMPolicy are right and only the ceiling is missing: the unit does come back, so do not say it will not. + *) printf 'FAIL the runner would come back after an OOM kill, but its cgroup has no ceiling:%s the box-wide OOM killer picks the victim, not the limit of this unit' "$_bad" + return 0 ;; + esac if [ -n "$_bad" ]; then printf 'FAIL the restart policy cannot restart this runner after an OOM kill:%s one OOM kill and the unit stops for good, while jobs read as pending' "$_bad" return 0 @@ -109,7 +120,12 @@ self_test() { chk FAIL "$(policy_verdict always stop 51539607552)" # an OOM kill of a JOB stops the RUNNER chk FAIL "$(policy_verdict always continue infinity)" # no cgroup ceiling: the box's OOM killer chooses chk FAIL "$(policy_verdict '' '' '')" # nothing readable is not a pass - [ "$rc" = 0 ] && echo "ci-runner-health self-test: ok (15 cases)" || echo "ci-runner-health self-test: FAIL" + # a missing ceiling alone must not be reported as "the unit stops for good" (the live strixhalo-engine unit, 2026-11-11) + case "$(policy_verdict always continue infinity)" in + *'would come back'*) printf 'ok a missing ceiling alone says the runner comes back\n' ;; + *) printf 'FAIL a missing ceiling alone should not say the unit stops for good\n'; rc=1 ;; + esac + [ "$rc" = 0 ] && echo "ci-runner-health self-test: ok (16 cases)" || echo "ci-runner-health self-test: FAIL" return $rc } @@ -125,7 +141,7 @@ if [ ${#units[@]} -eq 0 ]; then exit 2 fi -rc=0 +rc=0 policy=0 for u in "${units[@]}"; do line=$("$SC" show "$u" -p ActiveState -p SubState -p Result -p NRestarts -p Restart -p OOMPolicy -p MemoryMax 2>/dev/null) st=$(printf '%s\n' "$line" | sed -n 's/^ActiveState=//p') @@ -144,7 +160,9 @@ for u in "${units[@]}"; do p=$(policy_verdict "$rst" "$oom" "$mmax") case "$p" in PASS*) printf 'PASS %-52s %s\n' "" "${p#PASS }" ;; - *) printf 'FAIL %-52s %s\n' "" "${p#FAIL }"; rc=1 ;; + *) printf 'WARN %-52s %s\n' "" "${p#FAIL }"; policy=1 ;; esac done +# a runner that is down outranks a policy line: 1 if any unit is not accepting jobs, else 3 if only a policy is short. +[ "$rc" = 0 ] && [ "$policy" = 1 ] && rc=3 exit "$rc" diff --git a/scripts/ci-runner-restart.conf b/scripts/ci-runner-restart.conf new file mode 100644 index 000000000..ee193a792 --- /dev/null +++ b/scripts/ci-runner-restart.conf @@ -0,0 +1,29 @@ +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# systemd drop-in for a self-hosted runner unit that predates scripts/install-ci-runner.sh (engine#1504). Installed as +# /etc/systemd/system/.d/restart.conf by the owner; docs/ci-runners.md "The two drop-ins" has the commands. +# It gives the unit the crash-recovery lines of the documented unit and changes nothing else (same user, same paths). +[Unit] +# never give up restarting: a runner that hit the start limit is as dead as one that was never restarted +StartLimitIntervalSec=0 + +[Service] +# always, not on-failure: runsvc.sh exits 0 when it is killed, so on-failure never fires +Restart=always +RestartSec=15 +# the kernel's OOM victim is one process; with the default (stop) that ends the unit. continue lets the job die and +# the runner keep running, so the job fails with a reason instead of the runner vanishing +OOMPolicy=continue diff --git a/scripts/weekly.sh b/scripts/weekly.sh index 468864481..d8ce116fc 100755 --- a/scripts/weekly.sh +++ b/scripts/weekly.sh @@ -635,6 +635,29 @@ stage_release() { say "released $tag" } +# The systemd row of the runner preflight (engine#1504): $1 is scripts/ci-runner-health.sh. Returns 1 only when a runner +# unit is not accepting jobs NOW (its exit 1): the checks after the release would queue for ever. Its exit 3 - every +# runner up, but a restart policy that would leave one down after the next OOM kill - is a WARNING and the release goes +# on (owner 2026-11-11): the policy is a box configuration change the owner applies (docs/ci-runners.md "The two +# drop-ins"), every heavy build already runs under scripts/mem-guard.sh --term, which ends the build (exit 143) before +# the kernel's OOM killer reaches the runner, and a release that cannot start does not make the unit any safer. +# A function of its own so tests/weekly_runner_preflight_test.sh can run it against a stub. +preflight_runner_units() { + local health=$1 health_out health_rc=0 + [ -x "$health" ] || return 0 + health_out=$("$health" 2>&1) || health_rc=$? + case "$health_rc" in + 0) printf '%s\n' "$health_out" | grep -c '^PASS' | xargs -r -I{} echo "preflight: runner units: {} PASS in systemd's view" ;; + 2) say "preflight: no actions.runner.* units on this host: systemd has nothing to inspect" ;; + 3) printf '%s\n' "$health_out" | sed 's/^/preflight: /' + echo "preflight: WARNING, not blocking: every runner unit is accepting jobs, but a restart policy would leave one down after the next OOM kill (scripts/ci-runner-health.sh exit 3; the WARN lines above name the unit and the setting, docs/ci-runners.md \"The two drop-ins\" has the fix)" ;; + *) printf '%s\n' "$health_out" | sed 's/^/preflight: /' + echo "preflight: a runner unit is not accepting jobs, so the checks after this release would queue for ever (scripts/ci-runner-health.sh exit $health_rc; the FAIL lines above say which; WEEKLY_SKIP_RUNNER_CHECK=1 to release anyway)" + return 1 ;; + esac + return 0 +} + # --- preflight ---------------------------------------------------------------------------- # What the stages need from the machine, checked before the first one runs, so that a missing # tool stops the run at the start rather than after an hour of build (2026-10-19 rehearsal). @@ -686,6 +709,7 @@ preflight() { # scripts/ci-runner-health.sh - systemd's view (#1546): Result=oom-kill, a restart window, a unit that is not # active. Says WHY, and fires the moment a unit dies. Exit 2 = no such units on # this host, i.e. a fresh box, which is a note rather than a failure. + # Exit 3 = all up but a restart policy is short: a warning (preflight_runner_units). # WEEKLY_SKIP_RUNNER_CHECK=1 is the release-during-an-outage escape, and it skips BOTH: a flag that skipped only # one of them would be useless in exactly the outage it exists for, because both fail for the same dead runner. if [ "${WEEKLY_SKIP_RUNNER_CHECK:-0}" != 1 ]; then @@ -697,23 +721,7 @@ preflight() { echo "preflight: a self-hosted runner is not online, so the checks after this release would queue for ever (WEEKLY_SKIP_RUNNER_CHECK=1 to release anyway)" bad=1 fi - if [ -x "$(cd "$(dirname "$0")" && pwd)/ci-runner-health.sh" ]; then - if health_out=$("$(cd "$(dirname "$0")" && pwd)/ci-runner-health.sh" 2>&1); then - printf '%s\n' "$health_out" | grep -c '^PASS' | xargs -r -I{} echo "preflight: runner units: {} PASS in systemd's view" - else - health_rc=$? - if [ "$health_rc" = 2 ]; then - say "preflight: no actions.runner.* units on this host: systemd has nothing to inspect" - else - printf '%s\n' "$health_out" | sed 's/^/preflight: /' - # Two kinds of FAIL reach here (engine#1504): a unit that is down NOW, and since #2082 a unit that is up - # but whose restart policy would leave it down after the next OOM kill. The FAIL lines above say which; - # this line must not claim "not accepting jobs" for the second, where every runner is active/running. - echo "preflight: a runner unit is down, or its restart policy would leave it down after the next OOM kill (scripts/ci-runner-health.sh exit $health_rc; the FAIL lines above say which, and scripts/install-ci-runner.sh fixes a policy line)" - bad=1 - fi - fi - fi + preflight_runner_units "$(cd "$(dirname "$0")" && pwd)/ci-runner-health.sh" || bad=1 else # a skip that says nothing is the failure mode these rows exist for: a release log where the runner checks # silently did not run reads exactly like one where they passed. diff --git a/tests/weekly_runner_preflight_test.sh b/tests/weekly_runner_preflight_test.sh new file mode 100755 index 000000000..381f590a5 --- /dev/null +++ b/tests/weekly_runner_preflight_test.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# weekly_runner_preflight_test.sh: the release preflight BLOCKS on a runner unit that is down and only WARNS on a restart +# policy that is short (engine#1504, owner 2026-11-11). Two halves, both the real code: +# 1. scripts/ci-runner-health.sh against a stub systemctl: exit 0 / 1 / 2 / 3, and a down unit outranks a policy line. +# 2. scripts/weekly.sh's preflight_runner_units(), cut out of the script and run against the real health script on the +# same stub: returns 0 with a WARNING line on exit 3, returns 1 on exit 1. +# No systemd, no network, no build. +set -u +root=$(cd "$(dirname "$0")/.." && pwd) +tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT +fails=0 +ok() { printf 'ok %s\n' "$1"; } +bad() { printf 'FAIL %s\n' "$1"; fails=$((fails + 1)); } + +# stub systemctl: STUB_UNITS = lines "unit state sub result restart oompolicy memorymax" +cat > "$tmp/systemctl" <<'STUB' +#!/usr/bin/env bash +case "$1" in + list-units) awk '{print $1, "loaded", $2, $3}' "$STUB_UNITS" ;; + show) awk -v u="$2" '$1 == u { + printf "ActiveState=%s\nSubState=%s\nResult=%s\nNRestarts=0\nRestart=%s\nOOMPolicy=%s\nMemoryMax=%s\n", $2, $3, $4, $5, $6, $7 }' "$STUB_UNITS" ;; +esac +STUB +chmod +x "$tmp/systemctl" +export SYSTEMCTL="$tmp/systemctl" STUB_UNITS="$tmp/units" +health="$root/scripts/ci-runner-health.sh" + +good="actions.runner.a.service active running success always continue 51539607552" +policy="actions.runner.b.service active running success no stop infinity" # the live bench unit, 2026-11-11 +dead="actions.runner.c.service failed failed oom-kill no stop infinity" # the 11-08 shape + +expect_rc() { # + local name=$1 want=$2; shift 2 + printf '%s\n' "$@" > "$STUB_UNITS"; [ $# -eq 0 ] && : > "$STUB_UNITS" + out=$(bash "$health" 2>&1); rc=$? + if [ "$rc" = "$want" ]; then ok "health: $name -> exit $rc"; else bad "health: $name: want exit $want, got $rc: $out"; fi +} +expect_rc "all healthy" 0 "$good" +expect_rc "a unit is down" 1 "$good" "$dead" +expect_rc "no units" 2 +expect_rc "all up, one policy short" 3 "$good" "$policy" +expect_rc "down outranks policy" 1 "$policy" "$dead" +printf '%s\n' "$good" "$policy" > "$STUB_UNITS" +out=$(bash "$health" 2>&1) +if printf '%s\n' "$out" | grep -q '^WARN .*Restart=no(want always)' && ! printf '%s\n' "$out" | grep -q '^FAIL'; then + ok "health: a short policy prints WARN with the setting, and no FAIL line" +else bad "health: a short policy should print WARN and no FAIL: $out"; fi + +# half 2: the function weekly.sh really runs +sed -n '/^preflight_runner_units() {$/,/^}$/p' "$root/scripts/weekly.sh" > "$tmp/fn.sh" +grep -q 'health_rc' "$tmp/fn.sh" || { bad "could not cut preflight_runner_units() out of scripts/weekly.sh"; echo "weekly_runner_preflight: FAIL"; exit 1; } +grep -q 'preflight_runner_units "' "$root/scripts/weekly.sh" || bad "scripts/weekly.sh's preflight no longer calls preflight_runner_units" +say() { echo "$*"; } +# shellcheck disable=SC1091 +. "$tmp/fn.sh" +row() { # + local name=$1 want=$2 text=$3; shift 3 + printf '%s\n' "$@" > "$STUB_UNITS"; [ $# -eq 0 ] && : > "$STUB_UNITS" + out=$(preflight_runner_units "$health" 2>&1); rc=$? + if [ "$rc" = "$want" ] && printf '%s\n' "$out" | grep -q -- "$text"; then ok "weekly: $name -> return $rc" + else bad "weekly: $name: want return $want with '$text', got $rc: $out"; fi +} +row "all healthy passes" 0 "PASS in systemd's view" "$good" +row "a short policy warns and does not block" 0 "WARNING, not blocking" "$good" "$policy" +row "the warning names the setting" 0 "OOMPolicy=stop(want continue)" "$good" "$policy" +row "a dead unit blocks" 1 "not accepting jobs" "$good" "$dead" +row "a dead unit blocks even beside a short policy" 1 "after an OOM kill" "$policy" "$dead" +row "no units is a note" 0 "nothing to inspect" +out=$(preflight_runner_units "$tmp/absent.sh" 2>&1); rc=$? +[ "$rc" = 0 ] && [ -z "$out" ] && ok "weekly: no health script, no row" || bad "weekly: a missing health script should be silent: $rc $out" + +[ "$fails" = 0 ] && { echo "weekly_runner_preflight: ok"; exit 0; } +echo "weekly_runner_preflight: FAIL ($fails)"; exit 1 From 697db7dc205e08251da98c23381c3ad7304c6354 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Tue, 10 Nov 2026 23:58:36 -0400 Subject: [PATCH 03/28] npu-pack: say what main does with a graph head (stored as is; an unaligned one is refused), review of #2104 (#2107) Co-authored-by: bong-water-water-bong Co-authored-by: Claude Fable 5.2 --- npu/pack_import.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/npu/pack_import.h b/npu/pack_import.h index 5fc37dc35..8977766f2 100644 --- a/npu/pack_import.h +++ b/npu/pack_import.h @@ -98,8 +98,10 @@ std::string qk_norm_refusal(bool declared, bool found); // How many rows a head read from the .fconst tail occupies in the container. The pack stores the head in whole 64-row // tiles, so a vocabulary that is not a multiple of 64 is carried with the pack's own padding rows (Hunyuan: 120,818 -> -// 120,832) while `vocab_size` and the plan's tensor shape stay the vocabulary. A head that is a graph initializer does -// not go through this (its own padding, to the 32-row tile quantum, is engine#1992's). This is the number a container's lm_head tile count follows from. +// 120,832) while `vocab_size` and the plan's tensor shape stay the vocabulary. A head that is a graph initializer is +// stored as it is: on main nothing pads it, and one whose row count is not a multiple of 32 is refused by the tile +// writer ("q4_1 tiles need N % 32 == 0"). If graph-head padding lands (engine#1992 was closed unmerged), its quantum +// belongs here. This is the number a container's lm_head tile count follows from. int64_t head_stored_rows(int64_t vocab, bool from_fconst_tail); // Why a pack's two layout witnesses cannot be reconciled, or "" when they can (engine#1985). From aa11d5127fbb9c333a0eac519af91a3b1877dd6d Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:09:22 -0400 Subject: [PATCH 04/28] ci: a skipped test is a test that did not run, and now something says so (refs #1946) (#2095) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * ci: a skipped test is a test that did not run, and now something says so (refs #1946) Five of the defects this session kept finding are one defect: **green because nothing ran.** * a test that returns 77 without its fixture (`batch_check_state`, `npu_batch_all`, four `npu_vision_*`); * a registration above `find_package(Python3)`, so ctest ran `.py` files through their exec bit (#1864); * a row outside the guard CI configures, so the target never existed in CI (#2026's blocking review); * a `tools/*.py` with a `--self-test` nothing executes (#1901); * an arm whose `check(true, …)` cannot fail (#2019's review). `ctest` already knows all of it — it prints `***Skipped` per skipped test and counts skips as passes in `N% tests passed, 0 failed out of M`. Nothing read it. `tools/ci/ctest_skips.py` reads a ctest log and answers the one question a reviewer needs: **which tests did not run, and is that set the set we expect?** * `***Skipped` (the test ran and returned 77) → against `tools/ci/ctest_skips.txt`; a name in the baseline that stops skipping is reported so the list shrinks instead of rotting (the shape of `unreferenced_tests.txt`, #1824); * **`(Not Run)` — the binary is missing — is never baselineable.** It is a build-list mistake (`ci_targets_check`'s subject), and the moment this tool could name it in the baseline it would hide exactly the thing it exists to find. `--write` prints those separately and refuses to put them in the file. ## Measured on the CI-shaped tree (`ONEBIT_NPU=OFF`, 192 tests) ``` $ ctest --output-on-failure --timeout 300 > log $ tools/ci/ctest_skips.py --log log 7 tests ***Skipped -> baselined, one line each, with the reason 53 tests (Not Run) -> reported as missing binaries, not baselined 56 failed -> the summary line, unchanged ``` The 53 are this developer tree's unbuilt targets, not CI's — in CI the Build step lists what it builds and `ci_targets_check` keeps that list honest. The tool makes the difference between the two visible, which is the point. ## Self-test Eight arms on synthetic logs: no skips, both skip shapes, a baselined skip accepted, an unbaselined skip refused, a baseline entry that stopped skipping reported, a missing binary refused with no way to baseline it, the two shapes yielding different verdicts, and a log without a summary being "cannot tell" (rc 2). `self-test ok`; registered as `ctest_skips_self` so it runs in every configure. ## What this PR does not do It does not wire itself into `ci.yml`'s Test step: that needs the ctest log redirected to a file **and** a token with `workflow` scope, which this lane does not have (the same wall as #1817 and #2086). The wiring is one line — `ctest … | tee build/ctest.log` then `tools/ci/ctest_skips.py --log build/ctest.log` — and it belongs to whoever can push a workflow change. Refs #1946, #1864, #2026, #1901, #2019, #1824. * ci: the skip baseline carries its notice, and the copyright gate now knows the name (CI found this on #2095) CI was right and the check did its job: ``` build Copyright notice on every file ##[error]Process completed with exit code 1 no comment style known (add one to LINE_STYLES/BLOCK_STYLES or EXEMPT): tools/ci/ctest_skips.txt ``` `.txt` has no comment style of its own, so `tools/copyright.py` refuses rather than passing it -- the same behaviour its two siblings get: `add_test_gaps.txt` (#1595) and `unreferenced_tests.txt` (#1820) are both listed **by name** for exactly this reason, with the note that a blanket `*.txt` would demand a notice in files that are data rather than code. So this does what those two do, and nothing more: * `ctest_skips.txt` joins the by-name list in `tools/copyright.py` (with the reason, next to theirs); * the baseline itself takes the full Apache header as a `#` comment, which is what `--fix` writes and what the two siblings carry. Verified: `tools/copyright.py --check` rc 0; `ctest_skips.py --self-test` ok; `ctest -R ctest_skips` 1/1 passed. Refs #2095 (this is the fix for its failing build), #1946, #1595, #1820. --------- Co-authored-by: t --- CMakeLists.txt | 5 + tools/ci/ctest_skips.py | 195 +++++++++++++++++++++++++++++++++++++++ tools/ci/ctest_skips.txt | 26 ++++++ tools/copyright.py | 4 + 4 files changed, 230 insertions(+) create mode 100755 tools/ci/ctest_skips.py create mode 100644 tools/ci/ctest_skips.txt diff --git a/CMakeLists.txt b/CMakeLists.txt index ce9aeeed1..b6b578068 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -358,6 +358,11 @@ add_test(NAME bench_preflight COMMAND bash ${CMAKE_SOURCE_DIR}/tools/tests/test- # two false conclusions about a pack in one night. `--self-test` drives the rule on a synthetic shard (no network): # the same offset from the data section returns the tensor, from byte 8 it does not. add_python_test(hf_tensor_self ${CMAKE_SOURCE_DIR}/tools/hf_tensor.py --self-test) +# engine#1946: a test that returns 77 is green, and green-because-it-did-not-run is the failure this fleet keeps paying +# for. `tools/ci/ctest_skips.py` reads a ctest log and reports every test that did not execute -- `***Skipped` (77) against +# `tools/ci/ctest_skips.txt`, and `(Not Run)` (a missing binary) as a build-list mistake that can never be baselined. +# Self-test only here: wiring it into ci.yml's Test step needs the log plus a `workflow`-scoped token (engine#1817). +add_python_test(ctest_skips_self ${CMAKE_SOURCE_DIR}/tools/ci/ctest_skips.py --self-test) add_python_test(moe_placement ${CMAKE_SOURCE_DIR}/tools/moe_placement.py --self-test) # engine#856: one HRX process's GTT split into the loader's copy and HRX's packed second copy, from its own -v log. add_python_test(hrx_gtt_split_self ${CMAKE_SOURCE_DIR}/tools/hrx_gtt_split.py --self-test) diff --git a/tools/ci/ctest_skips.py b/tools/ci/ctest_skips.py new file mode 100755 index 000000000..c5f088b99 --- /dev/null +++ b/tools/ci/ctest_skips.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""ctest_skips.py -- make "green because it did not run" visible in a ctest log. + +Two nights of reviewing turned up the same failure five times: a test that returns 77 without its fixture, a +registration above `find_package`, a row outside the guard CI configures, a `.py` nothing runs, an arm whose +`check(true, ...)` cannot fail. Each one is *green*, and each one is green because nothing executed. + +`ctest` already knows: it prints `***Skipped` per skipped test and a `N% tests passed, 0 failed out of M` line that +counts skips as passes. This tool reads a ctest log and answers the one question a reviewer needs -- **which tests did +not run, and is that set the set we expect?** -- against a checked-in baseline. + + tools/ci/ctest_skips.py --log build/ctest.log # report; exit 1 on a NEW skip + tools/ci/ctest_skips.py --log build/ctest.log --write # refresh the baseline + tools/ci/ctest_skips.py --self-test + +Exit: 0 every skip is in the baseline, 1 a test skipped that the baseline does not name (named), 2 cannot tell +(no log, or no summary line). + +The baseline is `tools/ci/ctest_skips.txt`: one test name per line with an optional `# reason`. An entry that no longer +skips is reported so the list shrinks instead of rotting -- the same shape as `unreferenced_tests.txt` (#1824). +""" + +import argparse +import pathlib +import re +import sys + +BASELINE = pathlib.Path("tools/ci/ctest_skips.txt") +# ctest writes "did not run" in two shapes, and the second is the worse one: +# " 16/192 Test #16: npu_repack_path .......***Skipped 0.72 sec" the test ran and returned 77 +# "\t182 - npu_private_route (Not Run)" the binary was not built at all +# Only the first is a fixture deciding it cannot work here; the second is a build-list mistake (ci_targets_check's +# subject) and has no legitimate baseline entry. +SKIP_LINE = re.compile(r"^\s*\d+/\d+ Test\s+#\d+:\s+(\S+)\s+\.*\*\*\*Skipped", re.MULTILINE) +NOT_RUN_LINE = re.compile(r"^\s*\d+ - (\S+) \(Not Run\)", re.MULTILINE) +SUMMARY = re.compile(r"^\s*(\d+)% tests passed, (\d+) tests failed out of (\d+)") + + +def skipped(log_text: str): + """{test name: 'skipped' or 'not-run'} -- every test that did not execute.""" + found = {name: "skipped" for name in SKIP_LINE.findall(log_text)} + for name in NOT_RUN_LINE.findall(log_text): + found[name] = "not-run" + return found + + +def summary(log_text: str): + """(failed, total) from ctest's summary line, or None when the log has no summary.""" + for line in log_text.splitlines(): + m = SUMMARY.match(line) + if m: + return int(m.group(2)), int(m.group(3)) + return None + + +def baseline(root: pathlib.Path): + """{test name: reason} from the baseline file; comments and blanks skipped.""" + path = root / BASELINE + out = {} + if not path.is_file(): + return out + for i, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + line = line.strip() + if not line or line.startswith("#"): + continue + name, _, reason = line.partition("#") + name = name.strip() + if name: + out[name] = reason.strip() + return out + + +def check(root: pathlib.Path, log_text: str) -> int: + total = summary(log_text) + if total is None: + print("ctest_skips: cannot tell: the log has no 'N% tests passed' summary line", file=sys.stderr) + return 2 + failed, count = total + did_not_run = skipped(log_text) + allow = baseline(root) + new = sorted(n for n in did_not_run if n not in allow and did_not_run[n] == "skipped") + not_run = sorted(n for n in did_not_run if did_not_run[n] == "not-run") + gone = sorted(n for n in allow if n not in did_not_run) + for name in not_run: + print(f"ctest_skips: {name} did NOT RUN because its binary is missing -- that is a build-list mistake, not a " + f"fixture decision: add the target to ci.yml's Build step (tools/ci/ci_targets_check.py is the same " + f"class), and it needs no baseline entry") + for name in new: + print(f"ctest_skips: {name} SKIPPED and the baseline does not name it -- it did not run, and a test that does " + f"not run cannot fail. Give it its fixture, move it where CI builds it, or add it to {BASELINE} with " + f"the reason it cannot run here") + for name in gone: + print(f"ctest_skips: {name} is in {BASELINE} but it no longer skips -- drop the line, the list should shrink") + if new or gone or not_run: + return 1 + if did_not_run: + print(f"ctest_skips: {len(did_not_run)} of {count} test(s) did not run, all named in the baseline " + f"({failed} failed)") + else: + print(f"ctest_skips: all {count} test(s) ran ({failed} failed)") + return 0 + + +def write_baseline(root: pathlib.Path, log_text: str) -> int: + did_not_run = skipped(log_text) + # Only a test that RAN and returned 77 can be baselined. "Not Run" means the binary is missing, which is a build-list + # mistake (ci_targets_check's subject) and would become invisible the moment this file named it. + skipped_only = sorted(n for n, kind in did_not_run.items() if kind == "skipped") + not_run = sorted(n for n, kind in did_not_run.items() if kind == "not-run") + path = root / BASELINE + lines = ["# Tests that legitimately do not run in the CI configuration, one name per line with the reason.", + "# Written by tools/ci/ctest_skips.py --write. A name here that stops skipping is reported, so the list", + "# shrinks rather than rots (the shape of tools/ci/unreferenced_tests.txt, engine#1824).", + "# A test whose BINARY is missing is never baselined here: that is a build-list mistake, not a fixture.", + ""] + for name in skipped_only: + lines.append(f"{name} # TODO: why this cannot run here") + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + print(f"ctest_skips: wrote {path} with {len(skipped_only)} skipped test(s)") + if not_run: + print(f"ctest_skips: {len(not_run)} test(s) did NOT RUN (binary missing) and are deliberately not baselined: " + f"{', '.join(not_run[:6])}{' …' if len(not_run) > 6 else ''}", file=sys.stderr) + return 0 + + +def self_test() -> int: + """A check that only ever sees a clean log proves nothing: drive all four verdicts.""" + import tempfile + + fails = 0 + + def ok(cond, what): + nonlocal fails + print(("ok " if cond else "FAIL ") + what) + fails += 0 if cond else 1 + + clean = "100% tests passed, 0 tests failed out of 3\n" + with_skip = (" 12/13 Test #12: npu_pack_import ..................***Skipped 0.01 sec\n" + "100% tests passed, 0 tests failed out of 13\n") + with_notrun = ("\t182 - npu_private_route (Not Run)\n" + "100% tests passed, 0 tests failed out of 13\n") + root = pathlib.Path(tempfile.mkdtemp()) + (root / BASELINE).parent.mkdir(parents=True, exist_ok=True) + (root / BASELINE).write_text("# reasons\nnpu_pack_import # needs a container\n") + + ok(not skipped(clean), "a log with no skip lines reports none") + ok(list(skipped(with_skip)) == ["npu_pack_import"], "a '***Skipped'-style line is read") + ok(check(root, clean) == 1, "a baseline entry that stopped skipping is drift (rc 1)") + ok(check(root, with_skip) == 0, "a skip the baseline names is accepted (rc 0)") + ok(summary(with_skip) == (0, 13), "the summary counts skips as passes, which is the point") + (root / BASELINE).write_text("# empty\n") + ok(check(root, with_skip) == 1, "a skip the baseline does NOT name is refused (rc 1)") + ok(check(root, "no summary here\n") == 2, "a log without a summary is 'cannot tell' (rc 2)") + ok(list(skipped(with_notrun)) == ["npu_private_route"], "a '(Not Run)' line is read") + ok(check(root, with_notrun) == 1, "a missing binary is refused with no way to baseline it (rc 1)") + ok(skipped(with_skip) != skipped(with_notrun), "... and the two shapes are not the same verdict") + print("self-test ok" if not fails else f"self-test FAILED ({fails})") + return 1 if fails else 0 + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--log", default="build/ctest.log", help="a ctest log (default: build/ctest.log)") + ap.add_argument("--root", default=".") + ap.add_argument("--write", action="store_true") + ap.add_argument("--self-test", action="store_true") + a = ap.parse_args() + if a.self_test: + return self_test() + path = pathlib.Path(a.log) + if not path.is_file(): + print(f"ctest_skips: cannot tell: no log at {path}", file=sys.stderr) + return 2 + text = path.read_text(encoding="utf-8", errors="replace") + root = pathlib.Path(a.root).resolve() + return write_baseline(root, text) if a.write else check(root, text) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/ci/ctest_skips.txt b/tools/ci/ctest_skips.txt new file mode 100644 index 000000000..b41bd8a6d --- /dev/null +++ b/tools/ci/ctest_skips.txt @@ -0,0 +1,26 @@ +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Tests that legitimately do not run in the CI configuration, one name per line with the reason. +# Written by tools/ci/ctest_skips.py --write. A name here that stops skipping is reported, so the list +# shrinks rather than rots (the shape of tools/ci/unreferenced_tests.txt, engine#1824). +# A test whose BINARY is missing is never baselined here: that is a build-list mistake, not a fixture. + +amdxdna_restart # returns 77: needs a fixture this configuration does not have (see the test's own skip message) +npu_import_contract # returns 77: needs a fixture this configuration does not have (see the test's own skip message) +npu_repack_path # returns 77: needs a fixture this configuration does not have (see the test's own skip message) +patch_sfs_atomic # returns 77: needs a fixture this configuration does not have (see the test's own skip message) +serve_client_hangup_cpu # returns 77: needs a fixture this configuration does not have (see the test's own skip message) +serve_context_limit_cpu # returns 77: needs a fixture this configuration does not have (see the test's own skip message) +serve_embed_vectors_cpu # returns 77: needs a fixture this configuration does not have (see the test's own skip message) diff --git a/tools/copyright.py b/tools/copyright.py index 840a92fa9..580a41350 100644 --- a/tools/copyright.py +++ b/tools/copyright.py @@ -44,6 +44,10 @@ # engine#1820: the other sweep's allow-list, same shape and the same reason -- `#`-comment data, `.txt` has no # style, and the header it wants is already at the top of the file. "unreferenced_tests.txt", + # engine#1946: the ctest skip baseline, the same shape and the same reason -- a '#'-comment data file whose header + # is the notice it would otherwise be asked for. (CI caught this on the PR: the tool refuses an unknown style rather + # than passing it, which is the behaviour worth having.) + "ctest_skips.txt", # engine#2018: the third test-registration baseline, same shape and the same reason -- data, `.txt` has no style. "unreferenced_py_tests.txt", "21serial-console", "90onboard-limits", "Dockerfile", "*.Dockerfile", From cad81fd2ce8cd74f7f483f2bbf0cbcccfc6b0865 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:31:23 -0400 Subject: [PATCH 05/28] tools: fit the routing head over --dump-encoder output, so before/after accuracy has an after (engine#1313 item 3) (#2112) Co-authored-by: t --- CMakeLists.txt | 3 + docs/laya-models.md | 13 ++ tools/laya_head_finetune.py | 335 ++++++++++++++++++++++++++++++++++++ 3 files changed, 351 insertions(+) create mode 100644 tools/laya_head_finetune.py diff --git a/CMakeLists.txt b/CMakeLists.txt index b6b578068..45a7daa0a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -364,6 +364,9 @@ add_python_test(hf_tensor_self ${CMAKE_SOURCE_DIR}/tools/hf_tensor.py --self-tes # Self-test only here: wiring it into ci.yml's Test step needs the log plus a `workflow`-scoped token (engine#1817). add_python_test(ctest_skips_self ${CMAKE_SOURCE_DIR}/tools/ci/ctest_skips.py --self-test) add_python_test(moe_placement ${CMAKE_SOURCE_DIR}/tools/moe_placement.py --self-test) +# engine#1313 item 3: the "after" of "before/after routing accuracy" -- a ridge head over --dump-encoder output, with the +# table's own logits as the before arm. Self-test only, so it needs no dump and no model. +add_python_test(laya_head_finetune ${CMAKE_SOURCE_DIR}/tools/laya_head_finetune.py --self-test) # engine#856: one HRX process's GTT split into the loader's copy and HRX's packed second copy, from its own -v log. add_python_test(hrx_gtt_split_self ${CMAKE_SOURCE_DIR}/tools/hrx_gtt_split.py --self-test) # engine#679: time per HRX dispatch from the backend's serial trace (the id hash and the per-kernel sums). diff --git a/docs/laya-models.md b/docs/laya-models.md index 6da158d8b..ffdb8d913 100644 --- a/docs/laya-models.md +++ b/docs/laya-models.md @@ -145,6 +145,19 @@ A line whose decision was cached still carries the state it was cached under, so same decisions serve made. A decision whose `state_len` exceeds `len(state)` must be dropped or re-sent with a shorter prompt: `--eval` would otherwise classify a prefix the demo never sent. +**The head itself: `tools/laya_head_finetune.py` (engine#1313 item 3).** `--dump-encoder` writes the encoder state and +`--eval` scores a classifier; this takes the dump and fits the classifier, so "before/after routing accuracy" has both +halves on the same cases, encoder and class order: + +```sh +1bit route --laya-models table.json --laya-model DIR --dump-encoder cases.jsonl --out dump/ +tools/laya_head_finetune.py dump/ --holdout 0.5 # before (the table's logits) vs after (a ridge head), held out +``` + +The split is by **case and stratified by label**, not by index: a decision log is written in arrival order and a demo's +arrival order is class-contiguous, so an index split holds out a class rather than a sample of each. `--holdout 0` fits +and scores on everything and reports `in-sample`, because a number is not a gate until you know which it is. + `--eval` prints accuracy, per-class support, recall and precision, the confusion matrix (rows true, columns chosen), serve's mean confidence, the mean top probability, ECE (10 equal bins over the top probability), log loss and the decision time p50 and p90. Size-gated cases are counted (`gated`), since serve would gate them too. diff --git a/tools/laya_head_finetune.py b/tools/laya_head_finetune.py new file mode 100644 index 000000000..d0bf6b6e2 --- /dev/null +++ b/tools/laya_head_finetune.py @@ -0,0 +1,335 @@ +#!/usr/bin/env python3 +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Fine-tune the routing head over `1bit route --dump-encoder` output, and report before/after accuracy +(engine#1313, item 3). + +The other two thirds of item 3 already exist: `--eval` prints accuracy, per-class recall/precision, the confusion +matrix, ECE and log loss over labelled cases, and `--dump-encoder` writes the head's own input. What was missing is +the middle: nothing took the dump and turned it into a head, so "before/after routing accuracy" had no "after". + +`before` is the table's decision, read from each case's own `logits` in the dump (argmax) -- so the two arms are +scored on the same cases, the same encoder and the same class order, and the only thing that changes is the head. +`after` is a ridge regression from the pooled encoder state to the class one-hot, which is closed-form, deterministic +and needs no framework: on a few hundred cases with a few thousand input rows the normal equations are exact. + +The split is by CASE, never by row: a case's encoder rows are all in one half, or a head could be scored on rows of a +case it was fitted on and the number would be in-sample while looking held out. `--holdout 0` fits and scores on +everything and says so (`in-sample`), because a gate must not read a number without knowing which it is. + +Usage: + tools/laya_head_finetune.py DUMP_DIR [--holdout F] [--ridge L] [--pool mean|last] [--json] + tools/laya_head_finetune.py --self-test + +The dump is what `1bit route --laya-models T --laya-model DIR --dump-encoder cases.jsonl --out DUMP` writes: + DUMP/h.f32 float32 encoder rows, concatenated + DUMP/cases.jsonl one line per case: {"src", "label", "options", "offset", "len", "logits", ...} + DUMP/dump.json {"d": encoder width, "rows", "cases", ...} + +A line whose `label` is -1 was written without a label; it is skipped, exactly as `--eval` skips it. +""" +import argparse +import json +import math +import os +import struct +import sys + + +def read_dump(root): + with open(os.path.join(root, "dump.json")) as f: + info = json.load(f) + d = int(info["d"]) + cases = [] + with open(os.path.join(root, "cases.jsonl")) as f: + for line in f: + line = line.strip() + if line: + cases.append(json.loads(line)) + raw = open(os.path.join(root, "h.f32"), "rb").read() + rows = len(raw) // (4 * d) if d else 0 + encoded = struct.unpack("<%df" % (rows * d), raw[: rows * d * 4]) + return d, encoded, cases + + +def pool_case(encoded, d, off, length, how): + """One feature vector per case: the mean (or the last) encoder row.""" + if length <= 0: + return None + if how == "last": + start = (off + length - 1) * d + return list(encoded[start : start + d]) + acc = [0.0] * d + for r in range(length): + base = (off + r) * d + for i in range(d): + acc[i] += encoded[base + i] + inv = 1.0 / length + return [v * inv for v in acc] + + +def argmax(v): + best = 0 + for i in range(1, len(v)): + if v[i] > v[best]: + best = i + return best + + +def fit_ridge(xs, ys, k, lam): + """Closed-form W (d x k) mapping features to one-hot, with a bias column. Solves (X'X + L)^-1 X'Y.""" + n = len(xs) + d = len(xs[0]) + D = d + 1 + # A = X'X + lam*I (bias not regularised), B = X'Y + A = [[0.0] * D for _ in range(D)] + B = [[0.0] * k for _ in range(D)] + for x, y in zip(xs, ys): + xb = list(x) + [1.0] + for i in range(D): + xi = xb[i] + if xi == 0.0: + continue + row = A[i] + for j in range(D): + row[j] += xi * xb[j] + B[i][y] += xi + for i in range(d): + A[i][i] += lam + # Gaussian elimination with partial pivoting on [A|B] + M = [A[i] + B[i] for i in range(D)] + for col in range(D): + piv = max(range(col, D), key=lambda r: abs(M[r][col])) + if abs(M[piv][col]) < 1e-12: + raise SystemExit("laya_head_finetune: the design matrix is singular at column %d (raise --ridge)" % col) + if piv != col: + M[col], M[piv] = M[piv], M[col] + pv = M[col][col] + M[col] = [v / pv for v in M[col]] + for r in range(D): + if r != col and M[r][col] != 0.0: + f = M[r][col] + Mr, Mc = M[r], M[col] + for c in range(col, D + k): + Mr[c] -= f * Mc[c] + # After elimination the top D rows are [I | A^-1 B], so the weights live in columns D..D+k-1. + # Transposed to one row per class (each D long, bias last) so predict() is a plain dot product. + return [[M[i][D + c] for i in range(D)] for c in range(k)] + + +def predict(W, x): + xb = list(x) + [1.0] + return [sum(w * v for w, v in zip(row, xb)) for row in W] + + +def evaluate(feats, cases, W, idxs, k): + """(correct, total, confusion) for one arm; W is None for the table's own decision.""" + conf = [[0] * k for _ in range(k)] + correct = 0 + for i in idxs: + want = cases[i]["label"] + got = argmax(cases[i]["logits"]) if W is None else argmax(predict(W, feats[i])) + if 0 <= want < k and 0 <= got < k: + conf[want][got] += 1 + correct += 1 if want == got else 0 + return correct, len(idxs), conf + + +def run(args): + d, encoded, cases = read_dump(args.dump) + usable, feats, labels = [], [], [] + for c in cases: + if c.get("label", -1) < 0: + continue + f = pool_case(encoded, d, int(c["offset"]), int(c["len"]), args.pool) + if f is None: + continue + usable.append(c) + feats.append(f) + labels.append(int(c["label"])) + if not usable: + raise SystemExit("laya_head_finetune: no labelled cases in the dump") + k = max(len(usable[0]["options"]), max(labels) + 1) + for c, y in zip(usable, labels): + if not (0 <= y < len(c["options"])): + raise SystemExit("laya_head_finetune: %s: label %d is outside its options" % (c.get("src", "?"), y)) + + # Split by CASE and STRATIFIED BY LABEL, deterministically (no RNG: a re-run on one dump reports one number). + # + # Not by index. A decision log is written in arrival order, and a demo's arrival order is class-contiguous -- + # a session of reflex steps, then a planning stretch. An index split then holds out a whole CLASS rather than a + # sample of each, and the "after" number is measured on classes the head never saw while the "before" arm (which + # needs no training) scores them fine. My own self-test hit exactly this: 20+20+20 cases, holdout 0.5, train = + # classes 0 and 1 only, so the ceiling for any head was 1/3. Stratifying by label is what makes "held out" mean + # held out from the same population. With --holdout 0 nothing is held out and the report says in-sample. + n = len(usable) + by_label = {} + for i, y in enumerate(labels): + by_label.setdefault(y, []).append(i) + train, test = [], [] + if args.holdout == 0: + train = list(range(n)) + test = list(range(n)) # in-sample: fit and score on everything, and say so + for y in (sorted(by_label) if args.holdout > 0 else []): + idxs = by_label[y] + hold = int(round(len(idxs) * args.holdout)) + if args.holdout > 0 and hold == 0: + raise SystemExit("laya_head_finetune: class %d has %d case(s), too few for a held-out split; " + "add cases or use --holdout 0 (in-sample)" % (y, len(idxs))) + test.extend(idxs[len(idxs) - hold:] if hold else []) + train.extend(idxs[: len(idxs) - hold] if hold else idxs) + if not train: + raise SystemExit("laya_head_finetune: the holdout left no training cases (use a smaller --holdout)") + + W = fit_ridge([feats[i] for i in train], [labels[i] for i in train], k, args.ridge) + b_correct, b_total, b_conf = evaluate(feats, usable, None, test, k) + a_correct, a_total, a_conf = evaluate(feats, usable, W, test, k) + b_acc = b_correct / b_total if b_total else 0.0 + a_acc = a_correct / a_total if a_total else 0.0 + out = { + "cases": n, + "train": len(train), + "test": len(test), + "holdout": args.holdout, + "arm": "in-sample" if args.holdout == 0 else "held-out", + "pool": args.pool, + "classes": [c["options"] for c in usable[:1]][0], + "before": {"correct": b_correct, "total": b_total, "accuracy": b_acc}, + "after": {"correct": a_correct, "total": a_total, "accuracy": a_acc}, + "delta": a_acc - b_acc, + "confusion_before": b_conf, + "confusion_after": a_conf, + } + return out + + +def self_test(): + """A dump can be synthesised exactly: rows in class-conditional clusters with a known boundary. The check is + that the fine-tune LEARNS it (the table's logits, made deliberately uninformative, cannot), and that the split + is by case -- a head fitted on a case's rows must not be scored on the same case.""" + import tempfile + + fails = 0 + + def check(ok, what): + nonlocal fails + print(("ok " if ok else "FAIL ") + what) + if not ok: + fails += 1 + + rng = 12345 + + def rnd(): + nonlocal rng + rng = (1103515245 * rng + 12345) % (1 << 31) + return rng / float(1 << 31) + + d, k, per_class, rows_per_case = 8, 3, 20, 3 + centres = [[0.0] * d for _ in range(k)] + for c in range(k): + for i in range(d): + centres[c][i] = 4.0 if (i % k) == c else -4.0 + with tempfile.TemporaryDirectory() as tmp: + h = open(os.path.join(tmp, "h.f32"), "wb") + meta = open(os.path.join(tmp, "cases.jsonl"), "w") + off = 0 + made = [] + for c in range(k): + for _ in range(per_class): + vals = [] + for _r in range(rows_per_case): + for i in range(d): + vals.append(centres[c][i] + (rnd() - 0.5)) + h.write(struct.pack("<%df" % len(vals), *vals)) + # Uninformative table logits: identical for every class, so "before" can only be class 0. + meta.write(json.dumps({"src": "s%d" % len(made), "label": c, "options": ["a", "b", "c"][:k], + "offset": off, "len": rows_per_case, "logits": [0.0] * k}) + "\n") + off += rows_per_case + made.append(c) + h.close() + meta.close() + json.dump({"d": d, "rows": off, "cases": len(made)}, open(os.path.join(tmp, "dump.json"), "w")) + + class A: + dump = tmp + holdout = 0.5 + ridge = 1e-3 + pool = "mean" + json = False + + out = run(A) + check(out["arm"] == "held-out", "the default run is scored held-out, and says so") + check(out["after"]["accuracy"] > out["before"]["accuracy"], + "the fine-tuned head beats the table on the same held-out cases (%.2f vs %.2f)" + % (out["after"]["accuracy"], out["before"]["accuracy"])) + check(out["after"]["accuracy"] >= 0.9, + "and it actually learns a separated boundary (%.2f)" % out["after"]["accuracy"]) + check(out["test"] + out["train"] == out["cases"], "every case lands in exactly one side of the split") + + # In-sample is reported as in-sample: same dump, holdout 0. + A.holdout = 0.0 + ins = run(A) + check(ins["arm"] == "in-sample", "holdout 0 is labelled in-sample rather than passing as a held-out number") + check(ins["before"]["total"] == ins["cases"] and ins["test"] == ins["cases"], + "in-sample scores every case (%d of %d)" % (ins["test"], ins["cases"])) + + # An unlabelled line is skipped, as `--eval` skips it. + with open(os.path.join(tmp, "cases.jsonl"), "a") as f: + f.write(json.dumps({"src": "unlabelled", "label": -1, "options": ["a", "b", "c"][:k], + "offset": 0, "len": rows_per_case, "logits": [0.0] * k}) + "\n") + A.holdout = 0.5 + out2 = run(A) + check(out2["cases"] == out["cases"], "a label -1 line is skipped, not counted as a class") + + if fails: + print("laya_head_finetune: %d FAILED" % fails) + return 1 + print("laya_head_finetune: all self-test arms passed") + return 0 + + +def main(argv): + ap = argparse.ArgumentParser(add_help=True) + ap.add_argument("dump", nargs="?") + ap.add_argument("--holdout", type=float, default=0.5, help="fraction of CASES held out (0 = in-sample)") + ap.add_argument("--ridge", type=float, default=1e-3, help="ridge strength on the feature weights") + ap.add_argument("--pool", choices=["mean", "last"], default="mean", help="how a case's encoder rows become one vector") + ap.add_argument("--json", action="store_true") + ap.add_argument("--self-test", action="store_true") + args = ap.parse_args(argv) + if args.self_test: + return self_test() + if not args.dump: + ap.error("a DUMP_DIR or --self-test is required") + if not (0.0 <= args.holdout < 1.0): + ap.error("--holdout must be in [0, 1)") + out = run(args) + if args.json: + print(json.dumps(out, indent=1)) + return 0 + print("cases %d (%d train, %d held out) arm=%s pool=%s" % + (out["cases"], out["train"], out["test"], out["arm"], out["pool"])) + print(" before (the table): %3d/%3d %.3f" % (out["before"]["correct"], out["before"]["total"], out["before"]["accuracy"])) + print(" after (fine-tune): %3d/%3d %.3f delta %+0.3f" % + (out["after"]["correct"], out["after"]["total"], out["after"]["accuracy"], out["delta"])) + for name, conf in (("before", out["confusion_before"]), ("after", out["confusion_after"])): + print(" %s confusion (rows true, cols chosen):" % name) + for r, row in enumerate(conf): + print(" %d: %s" % (r, " ".join("%3d" % v for v in row))) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) From 875272bc7a83b6e3d518cb3c460a126fdd125d5f Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:31:32 -0400 Subject: [PATCH 06/28] npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) (#2101) * npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) Two gemma-2-2b hybrid packs measured on this box import with every check green and serve garbage: pack corr rel_err serves Qwen-2.5_1.5B_Instruct_rai_1.9.0_npu 0.996901 0.000000 'The capital of France is Paris.' x3 gemma-2-2b-onnx-ryzenai-hybrid 0.991539 0.136140 'myself myself ...' x3 gemma-2-2b-awq-...-g32-fp16-onnx-hybrid 0.996053 0.094874 'myselfGetBytes...' x3 A head that is a SCALED copy of its own embedding still correlates, so the correlation-only check passes it -- and the import printed the number that distinguishes them (rel_err) as an advisory field next to the correlation it gated on. Both packs cost a device ticket to discover. - one predicate, selfcheck_head_ok(corr, rel), so the four sites that set selfcheck_tied cannot drift; - kSelfcheckRelMax = 0.02, the measured gap: 0.000000 on the pack that serves, 0.095/0.136 on the two that do not; - Report::selfcheck_scaled, set at all four sites next to selfcheck_tied; - main() refuses on it and NAMES both numbers and the threshold. --allow-incomplete does not override: that flag means 'write the tensors I could not represent', and this is 'the tensors I wrote do not decode'. Verified: both translation units pass g++ -fsyntax-only (the json header from a neighbour build), and the predicate's six arms pass against the three measured triples plus a control, a no-correlation case and the inclusive boundary (~ /wt/relcheck.cpp, run). NOT verified here: the full npu_pack_import_test binary and any container import -- that needs a configured build, which the disk floor did not allow at the time; the two test arms are written for CI. * npu-pack: name the partial output in the mis-scaled-head refusal (engine#1503) Demonstrated on the real pack, not argued from the fixture: importing ~/scratch-packs/gemma22b (gemma-2-2b-onnx-ryzenai-hybrid, the pack that served 'myself myself ...' for 16 tokens) now stops at refusing to import ...: the head correlates with the pack's own embedding (0.991539) but its relative error is 0.136140, which is a scaled or partially decoded head rather than a rounding difference (the threshold is 0.020). A pack in this state imports cleanly and answers garbage on the device; --allow-incomplete does not override this. That run also showed the check happens after import_pack() has already copied the text artifacts and written model.q4nx, so a refusal left five files behind. An importer must not delete a directory the caller named, so the message now says the partial output exists and is not servable -- the operator deletes it before retrying. Verified by rebuilding onebit and re-running the same import, and the test target still passes its six arms. --------- Co-authored-by: pi-mosaic --- npu/pack_import.cpp | 31 +++++++++++++++++++++++++++++++ npu/pack_import.h | 11 +++++++++++ tests/npu_pack_import_test.cpp | 16 ++++++++++++++++ 3 files changed, 58 insertions(+) diff --git a/npu/pack_import.cpp b/npu/pack_import.cpp index f2ad9c895..463c49a36 100644 --- a/npu/pack_import.cpp +++ b/npu/pack_import.cpp @@ -498,6 +498,11 @@ const uint8_t* at(const Fconst& f, int64_t off, int64_t len) { // field, the readout and the note all have to agree on it: three copies of 0.99 is how they drifted apart before. constexpr double kTieCorr = 0.99; +// engine#1503: tied AND not mis-scaled. Correlation alone passes a head that is a scaled copy of the embedding -- two +// packs on this box report corr 0.9915/0.9961 with rel_err 0.095/0.136 and serve garbage, while the pack that serves +// correctly reports rel_err 0.000000. One predicate, so the four sites that set selfcheck_tied cannot drift apart. + + // Every entry was measured by diffing the 1.9 pack against the 1.8 pack of the SAME model: the bundle's tile shape and // its rotation are not derivable from the pack, and neither is `prefix`. It was tempting to read the extra bytes as // Qwen3's per-head QK norm gains, but Llama-3.2-1B -- which has no QK norms at all -- carries the same +4,352 at hidden @@ -5016,6 +5021,8 @@ Report import_pack(const std::string& pack_dir, const std::string& out_dir, cons // `tied` is false there while the head plainly reproduces its embedding, and the readout used to call those // packs untied while the container it wrote said tied. Same threshold the container's own decision uses. r.selfcheck_tied = r.lmhead_corr > kTieCorr; + + r.selfcheck_scaled = r.selfcheck_tied && !selfcheck_head_ok(r.lmhead_corr, r.lmhead_rel); const Q4 swap = decode_matmulnbits(bytes_of(wt, qw_need), scales.data(), zp, int(mm.N), int(mm.K), mm.block, /*swap_nibbles*/ true, false); const Q4 nozp = decode_matmulnbits(bytes_of(wt, qw_need), scales.data(), nullptr, int(mm.N), int(mm.K), @@ -5103,6 +5110,8 @@ Report import_pack(const std::string& pack_dir, const std::string& out_dir, cons } r.selfcheck_reference = true; r.selfcheck_tied = c32 > kTieCorr; + + r.selfcheck_scaled = r.selfcheck_tied && !selfcheck_head_ok(r.lmhead_corr, r.lmhead_rel); double sxy = 0, sx = 0, sy = 0, sxx = 0, syy = 0; const int nsh = std::min(2048, int(V) - 1); for (int rr = 0; rr < nsh; ++rr) @@ -5316,6 +5325,8 @@ Report import_pack(const std::string& pack_dir, const std::string& out_dir, cons r.lmhead_corr = corr(q, embed_tbl, q.N); r.selfcheck_reference = true; r.selfcheck_tied = r.lmhead_corr > kTieCorr; + + r.selfcheck_scaled = r.selfcheck_tied && !selfcheck_head_ok(r.lmhead_corr, r.lmhead_rel); double sxy = 0, sx = 0, sy = 0, sxx = 0, syy = 0; const int nsh = std::min(2048, q.N - 1); for (int rr = 0; rr < nsh; ++rr) @@ -5428,6 +5439,8 @@ Report import_pack(const std::string& pack_dir, const std::string& out_dir, cons r.lmhead_corr = corr(q, embed_tbl, q.N); r.selfcheck_reference = true; r.selfcheck_tied = r.lmhead_corr > kTieCorr; + + r.selfcheck_scaled = r.selfcheck_tied && !selfcheck_head_ok(r.lmhead_corr, r.lmhead_rel); // The control this path can actually run: the same tiles and the same decoder, with each // head row matched against the NEXT embedding row. If a misaligned head still correlated, // the number above would be measuring the vocabulary's mean brightness rather than the @@ -6025,6 +6038,20 @@ int run(int argc, char** argv) { } const Report r = import_pack(dir, out, Options{allow, layout}); + // engine#1503: a head that correlates but is mis-scaled imports cleanly and answers nonsense on the device (two + // packs measured: rel_err 0.095 and 0.136 against 0.000000 on the pack that serves). `--allow-incomplete` is not an + // override here: that flag means "write the tensors I could not represent", and this is "the tensors I wrote do not + // decode", which no amount of writing fixes. + if (r.selfcheck_scaled) { + std::fprintf(stderr, + "1bit npu-pack: refusing to import %s: the head correlates with the pack's own embedding (%.6f) but " + "its relative error is %.6f, which is a scaled or partially decoded head rather than a rounding " + "difference (the threshold is %.3f). A pack in this state imports cleanly and answers garbage on " + "the device; --allow-incomplete does not override this. The partial output this run already\n" + "wrote under %s is not servable -- delete it before retrying.\n", + dir.c_str(), r.lmhead_corr, r.lmhead_rel, kSelfcheckRelMax, out.c_str()); + return 3; + } if (!r.refused.empty() && !allow) { std::fprintf(stderr, "1bit npu-pack: refusing to import %s: %zu of the pack's tensors cannot be " @@ -6278,5 +6305,9 @@ void sweep_fconst_scratch_dirs(const std::vector& dirs, const std:: } } +// engine#1503: the tied-head rule, defined here so it is the namespace-scope function the header declares (a file-local +// copy is ambiguous with it at every call site). Correlated AND not a scaled copy of the embedding -- the two states this +// box measures are rel_err 0.000000 (serves) and 0.095/0.136 (imports cleanly, answers garbage). +bool selfcheck_head_ok(double corr, double rel) { return corr > kTieCorr && rel <= kSelfcheckRelMax; } } // namespace onebit::npu::pack_import diff --git a/npu/pack_import.h b/npu/pack_import.h index 8977766f2..812903775 100644 --- a/npu/pack_import.h +++ b/npu/pack_import.h @@ -450,6 +450,11 @@ struct Report { // check; an untied pack's head and embedding are separate matrices by construction, and the same number is // meaningless -- reported as such rather than left looking like a failed import. bool selfcheck_tied = false; + // engine#1503: tied, correlated -- but MIS-SCALED. Correlation alone passes a head that is a scaled copy of the + // embedding: two packs measured on this box report corr 0.9915/0.9961 with rel_err 0.095/0.136 and serve garbage + // ("myself" forever), while the pack that serves correctly reports rel_err 0.000000. A pack in this state imports + // cleanly and answers nonsense, so it is refused rather than reported. + bool selfcheck_scaled = false; double lmhead_corr = 0, lmhead_rel = 0, ctrl_corr_nibble = 0, ctrl_corr_nozp = 0; // The tail (.fconst) path has no MatMulNBits bytes to mis-decode, so its control is an ALIGNMENT one: the same // tiles, the same decoder, rows matched against the neighbouring embedding row. A correct decode correlates ~1 @@ -780,6 +785,12 @@ bool fconst_scratch_ready(const std::string& dir); const Fconst19Rule* select_fconst19_rule_from(const Fconst19Rule* rules, int count, int hidden, int heads, int layers, std::string* why); +// engine#1503: the tied-head rule, as ONE predicate so the four call sites cannot drift: the head must reproduce the +// embedding (corr) AND not be a scaled copy of it (rel_err). The constant is the measured gap between the two states -- +// 0.000000 for the pack that serves, 0.095/0.136 for the two that do not -- so 0.02 sits far from both. +inline constexpr double kSelfcheckRelMax = 0.02; +bool selfcheck_head_ok(double corr, double rel); + int64_t attn_tile_bytes(int64_t k); // Pearson correlation between a quantised head and a bf16 embedding table (engine#1690; the self-check's own statistic, // exported so a test can pin the buffer width it decodes). diff --git a/tests/npu_pack_import_test.cpp b/tests/npu_pack_import_test.cpp index 0dcad1ff0..dfceaa69a 100644 --- a/tests/npu_pack_import_test.cpp +++ b/tests/npu_pack_import_test.cpp @@ -5683,7 +5683,23 @@ void test_header_witness_refusal() { std::printf(" header witness: 8 arms (one-header vote vs declaration refuses; 3/1-without-declaration/0 allowed)\n"); } +// engine#1503: the tied-head rule as ONE predicate -- correlated AND not mis-scaled. Two packs on this box correlate +// (0.9915 / 0.9961) while being scaled copies of their own embedding (rel_err 0.1361 / 0.0949): they import cleanly and +// serve garbage, where the pack that serves correctly reports rel_err 0.000000. Correlation alone cannot tell them apart, +// which is why the correlation-only check passed both and cost two device tickets. +static void selfcheck_head_rule_arms() { + check(pi::selfcheck_head_ok(0.996901, 0.000000), "head rule: the pack that serves correctly passes (corr .9969, rel 0)"); + check(!pi::selfcheck_head_ok(0.991539, 0.136140), "head rule: a correlated but scaled head is refused (corr .9915, rel .1361)"); + check(!pi::selfcheck_head_ok(0.996053, 0.094874), "head rule: the second measured pack is refused too (corr .9961, rel .0949)"); + check(pi::selfcheck_head_ok(0.996053, 0.000000), + "head rule: correlated and unscaled passes -- the rule is not 'refuse everything'"); + check(!pi::selfcheck_head_ok(0.50, 0.000000), "head rule: an uncorrelated head is still refused on correlation"); + check(pi::selfcheck_head_ok(0.996053, pi::kSelfcheckRelMax), "head rule: exactly at the threshold passes (inclusive bound)"); + std::printf(" head rule: 6 arms (two measured bad packs, the serving one, the unscaled control, no-corr, boundary)\n"); +} + int main(int argc, char** argv) { + selfcheck_head_rule_arms(); // engine#1907: every crafted 1.9 import writes a canonical copy into the scratch cache. Without this the suite left 0-byte // copies in the user's real ~/.cache/1bit/fconst19 and in /tmp on every run. Give the run its own directory, removed at exit. const fs::path run_scratch = ::onebit::test::tmp_root() / ("1bit-pack-test-scratch-" + std::to_string(::getpid())); From afb46403be6066c85327db44537fa00e53db7fb6 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Wed, 11 Nov 2026 01:03:12 -0400 Subject: [PATCH 07/28] forward-serve: a text request to a decision-only pack is a 400 that names the decision route (Refs #1786) (#2121) A pack with non-causal attention has no decode to run. /v1/chat/completions and /v1/completions tried anyway and failed inside the forward with HTTP 500 and its internal wording (measured on FastFlowLM's D3-Nano-NPU2). Co-authored-by: bong-water-water-bong Co-authored-by: Claude Fable 5.2 --- app/forward_serve.cpp | 6 ++++++ app/npu_systemone.h | 16 ++++++++++++++++ docs/npu.md | 5 +++++ tests/npu_chatml_test.cpp | 15 +++++++++++++++ 4 files changed, 42 insertions(+) diff --git a/app/forward_serve.cpp b/app/forward_serve.cpp index 1b4249bec..b5da1fa66 100644 --- a/app/forward_serve.cpp +++ b/app/forward_serve.cpp @@ -807,6 +807,12 @@ void handle_generate_impl(ForwardEngine& e, const httplib::Request& req, httplib // this thread waits for the first frame or the end, and only then decides between an ordinary response -- the 400s // and 500s a request can earn before it has produced anything keep their status codes -- and a stream. void handle_generate(ForwardEngine& e, const httplib::Request& req, httplib::Response& res, bool chat) { + // A non-causal decision pack has no decode to run (npu::decision_only_refusal): say so before any work. + if (e.fwd && e.fwd->noncausal()) { + res.status = 400; + res.set_content(npu::decision_only_refusal(e.id, e.head != nullptr).dump(), "application/json"); + return; + } bool wants_stream = false; try { const json body = json::parse(req.body); diff --git a/app/npu_systemone.h b/app/npu_systemone.h index 22623f7cc..2a6c0bfd4 100644 --- a/app/npu_systemone.h +++ b/app/npu_systemone.h @@ -124,6 +124,22 @@ inline nlohmann::json systemone_question_error(const std::string& kind, const st {"message", message}}; } +// What a text route answers on a pack that cannot generate text. A decision pack that declares +// `attention_mode: noncausal_full_attention` encodes a whole prompt in one pass and reads an answer off the last +// position; there is no token-by-token decode to run, so `/v1/chat/completions` and `/v1/completions` can never +// succeed on it. They used to try, and failed inside the forward with HTTP 500 and the forward's own wording +// ("forward step failed: attention.mode noncausal_full_attention encodes a prompt in one batched pass, but this +// call starts at position 23 ..."; measured on FastFlowLM's D3-Nano-NPU2, engine#1786). That is the client's +// request against the wrong route, so it is a 400 that names the right one. `has_head` says whether the decision +// route is actually up on this serve, because pointing a client at a route that answers 503 is no help. +inline nlohmann::json decision_only_refusal(const std::string& model, bool has_head) { + const std::string where = has_head ? "send decisions to POST /v1/systemone" + : "and this serve has no decision head loaded, so it has no route for it"; + return {{"error", + {{"message", model + " is a decision model (non-causal attention): it does not generate text; " + where}, + {"type", "decision_model_error"}}}}; +} + inline std::string systemone_length_message(size_t tokens, size_t limit) { return "the question prompt has " + std::to_string(tokens) + " tokens, over the maximum context length of " + std::to_string(limit) + " tokens; nothing was truncated"; diff --git a/docs/npu.md b/docs/npu.md index 85857e81f..922997b93 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -2107,6 +2107,11 @@ forward-serve` serves that too, from the same process and the same weights: | `POST /v1/systemone` (alias `POST /v1/decisions`) | `{"state": …, "questions": {id: question}}` → `{"model", "answers": {id: answer}, "usage"}` | | `GET /v1/systemone/models` | the decision model this build serves: name, description, `max_input_tokens`, `modalities` | +A pack that declares `attention_mode: noncausal_full_attention` (below) is a decision model and nothing else: +it encodes a prompt in one pass and has no token-by-token decode. On such a pack `/v1/chat/completions` and +`/v1/completions` answer **400** with `type: decision_model_error` and a message that names `/v1/systemone` +(or says there is no decision head, when the serve loaded none), before any work is done. + `questions` is an **object**, and its ids come back as the keys of `answers` in request order. Question types are `choice` (1–255 options), `noul` and `score` (2–10 ordered levels); a decision runs one backbone prefill per question, pools the **last position's hidden state after the final norm** (`pooling: last`), diff --git a/tests/npu_chatml_test.cpp b/tests/npu_chatml_test.cpp index d43909570..dd407c1d1 100644 --- a/tests/npu_chatml_test.cpp +++ b/tests/npu_chatml_test.cpp @@ -154,6 +154,21 @@ int main(int argc, char** argv) { check(pin_probe_messages(oreq).size() == 2, "the same for the ordered JSON the template path uses"); } + // A text route on a decision-only pack (engine#1786, D3-Nano-NPU2): a 400 that names the right route + { + const nlohmann::json up = onebit::npu::decision_only_refusal("D3-Nano-NPU2", true); + const std::string m = up["error"]["message"].get(); + check(m.find("D3-Nano-NPU2") == 0 && m.find("does not generate text") != std::string::npos && + m.find("POST /v1/systemone") != std::string::npos, + "the refusal names the model, says it does not generate text, and names the decision route"); + check(up["error"]["type"] == "decision_model_error", "and carries a type a client can match on"); + const std::string none = onebit::npu::decision_only_refusal("x", false)["error"]["message"].get(); + check(none.find("POST /v1/systemone") == std::string::npos && none.find("no decision head") != std::string::npos, + "with no head loaded it does not send the client to a route that would answer 503"); + check(m.find("forward step failed") == std::string::npos && m.find("position") == std::string::npos, + "none of the forward's internal wording reaches the client"); + } + if (failures == 0) std::printf("npu_chatml: ok\n"); return failures == 0 ? 0 : 1; } From a97e39855190d0250dd4ff4187b4dbb53c36d101 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Wed, 11 Nov 2026 01:03:18 -0400 Subject: [PATCH 08/28] tools: npu_serve_smoke, the release gate's check that npu serve is on the fast route at speed (refs #2106, #2102) (#2119) * tools: npu_serve_smoke, the release gate's check that npu serve is on the fast route at speed (refs #2106, #2102) A serve that falls back to the per-op forward still answers, so a check that asks 'did it answer' passes on the wrong route. This starts 1bit serve --device npu, sends three prompts with thinking off, and asserts the route's own log line, a decode floor (60 tok/s), the first token (600 ms) and non-empty, non-degenerate content. Measured, Qwen3-0.6B, one npu-hold ticket each: the 2026-11-07 release candidate PASSes (median 84.4 tok/s, 277 ms); a main build without the fast route FAILs route, rate (1.9 tok/s) and first-token while its answers are fine. --self-test drives the judging on 15 shapes (the silent fallback, thinking left on, a degenerate reply, the median, missing timings); registered in ctest as npu_serve_smoke_self. Co-Authored-By: Claude Fable 5.2 * tools: npu_serve_smoke's self-test pins the median (a mean mutant passed all 15 cases) Co-Authored-By: Claude Fable 5.2 * tools: npu_serve_smoke takes the two route texts as options, so it can follow the serve's wording Co-Authored-By: Claude Fable 5.2 --------- Co-authored-by: bong-water-water-bong Co-authored-by: Claude Fable 5.2 --- CMakeLists.txt | 4 + docs/hw-gate.md | 40 ++++++ tools/npu_serve_smoke.py | 280 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 324 insertions(+) create mode 100755 tools/npu_serve_smoke.py diff --git a/CMakeLists.txt b/CMakeLists.txt index 45a7daa0a..0a8719d47 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -363,6 +363,10 @@ add_python_test(hf_tensor_self ${CMAKE_SOURCE_DIR}/tools/hf_tensor.py --self-tes # `tools/ci/ctest_skips.txt`, and `(Not Run)` (a missing binary) as a build-list mistake that can never be baselined. # Self-test only here: wiring it into ci.yml's Test step needs the log plus a `workflow`-scoped token (engine#1817). add_python_test(ctest_skips_self ${CMAKE_SOURCE_DIR}/tools/ci/ctest_skips.py --self-test) +# engine#2106 / #2102: the release gate's NPU serve check asserts the fast route, a decode floor, the first token and a +# non-empty answer, because a serve on the slow per-op route still answers. --self-test drives the judging on the shape +# of each failure (the silent fallback at 1.9 tok/s, thinking left on, a degenerate reply); no device, no binary. +add_python_test(npu_serve_smoke_self ${CMAKE_SOURCE_DIR}/tools/npu_serve_smoke.py --self-test) add_python_test(moe_placement ${CMAKE_SOURCE_DIR}/tools/moe_placement.py --self-test) # engine#1313 item 3: the "after" of "before/after routing accuracy" -- a ridge head over --dump-encoder output, with the # table's own logits as the before arm. Self-test only, so it needs no dump and no model. diff --git a/docs/hw-gate.md b/docs/hw-gate.md index 67402e213..c8cb13604 100644 --- a/docs/hw-gate.md +++ b/docs/hw-gate.md @@ -252,6 +252,46 @@ The settings are environment variables at the top of the script: `HW_GATE_STORE` `$HW_GATE_OUT` (default `~/.cache/hw-gate/runs/