Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions cmake/tests-tmp-path.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# engine#2021: a hardcoded `/tmp` path is RAM on this box, not disk.
#
# `/tmp` is tmpfs here, so a dump, a staging copy or a benchmark log written there is anonymous memory: swap fills
# and every lane on the box stalls (measured 2026-11-10 21:15Z: /tmp 52 G of 62 G, swap 71/71 GiB). engine#1907/#1952
# bounded the fconst scratch; the literals themselves are still in the tree, 14 of them across 8 files.
#
# Both rows are cheap and device-free: the check is a grep over npu/ app/ tools/ scripts/ against
# tools/ci/tmp_paths.txt, and the ratchet may only shrink.
add_python_test(tmp_path_check ${CMAKE_SOURCE_DIR}/tools/ci/tmp_path_check.py --check --root ${CMAKE_SOURCE_DIR})
add_python_test(tmp_path_check_self ${CMAKE_SOURCE_DIR}/tools/ci/tmp_path_check.py --self-test)
254 changes: 254 additions & 0 deletions tools/ci/tmp_path_check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
#!/usr/bin/env python3
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""A hardcoded `/tmp` path is RAM on this box, not disk.

`/tmp` is tmpfs on both development boxes, so a dump, a staging copy or a server log written there is
anonymous memory: it cannot be reclaimed to disk under pressure, it goes to swap, and when swap fills every
lane on the box stalls. Measured 2026-11-10 21:15Z, `/tmp` held 52 G of 62 G and swap was 71/71 GiB full;
the two largest items were a stale `1bit-fconst19*.fconst` scratch copy (3.5 G) and five `q17-*.json` dumps
(9.2 G). engine#1907/#1952 bounded the fconst scratch; the *paths* are still literals in the tree.

This check keeps the literals from spreading. It is a ratchet, not a gate: `tools/ci/tmp_paths.txt` holds the
sites that exist today and may only shrink. A new site fails, and so does a baseline line whose site is gone -
that second arm matters because a file rename would otherwise move a site out of the baseline silently, which
is the `tools/ci/ci_targets_check.py` lesson from engine#1728 (a ratchet that only counts upward cannot see a
rename).

Usage:
tools/ci/tmp_path_check.py --check [--root .] # rc 1 if a site was added or a baseline line went stale
tools/ci/tmp_path_check.py --self-test # rc 1 unless the arms below behave
tools/ci/tmp_path_check.py --write-baseline # rewrite tools/ci/tmp_paths.txt from the tree
"""
from __future__ import annotations

import argparse
import os
import pathlib
import re
import sys

# A `/tmp` path in C, C++ or shell. The first version of this required a LEADING QUOTE, and a reviewer measured
# what that missed: `echo y > /tmp/x.log` -- a bare path, the plain shell idiom, and arguably the most likely form
# in `scripts/*.sh` -- was invisible, while the same path in quotes was caught. So the leading quote is gone and the
# anchor is "not the tail of another path": `(?<![A-Za-z0-9_./-])` keeps `/var/tmp/x` out (the `a` before `tmp`
# blocks it) and lets `/tmp/x` in whether it follows a space, `>`, `=`, a quote or nothing at all. A trailing quote
# is included when present so the baseline names the literal as a reader finds it.
LITERAL = re.compile(r"""(?<![A-Za-z0-9_./-])/tmp/[^"'\s;|)]*["']?""")

# The paths this check owns. `npu/` and `app/` are where a dump lands; `tools/` and `scripts/` are where a
# benchmark log lands. Everything else is out of scope on purpose: a test may use a temp dir (and pytest's
# `tmp_path` is the right instrument there), and `docs/` quotes paths rather than writing them.
SCAN = ("npu", "app", "tools", "scripts")
SUFFIXES = (".c", ".cc", ".cpp", ".h", ".hpp", ".sh", ".py")

# This checker's own fixture text is not a site: its self-test writes `"/tmp/..."` literals into a scratch tree to
# prove they are found. Excluded BY NAME rather than by pattern so the exclusion is one line a reader can check, and
# the run prints it (--check counts excluded files) so a second such file cannot appear unnoticed.
SELF = "tools/ci/tmp_path_check.py"

# Test files are excluded: a test that writes to a temp dir it created is not the defect, and the fixtures would
# otherwise have to be baselined too.
def is_test(rel: str) -> bool:
parts = rel.split(os.sep)
if any(p in ("tests", "test") for p in parts[:-1]):
return True
name = parts[-1]
return name.endswith(("_test.c", "_test.cpp", "_test.py", "_test.sh")) or name.startswith("test_")


# A commented-out path is not a path. engine#1983's review took `# was include(cmake/...)` for an include and
# lost seven registrations with every gate green; the same substring trap applies here, so comments are removed
# before the match rather than filtered after it.
def strip_comments(text: str, suffix: str) -> str:
if suffix == ".py":
return re.sub(r"#[^\n]*", "", text)
if suffix == ".sh":
# A `#` starts a comment when it begins a word; `$#`, `${x#y}` and a `#` inside a string are not.
return re.sub(r"(?<![$\w{])#(?!\{)[^\n]*", "", text)
text = re.sub(r"/\*.*?\*/", "", text, flags=re.S)
return re.sub(r"(?<!:)//[^\n]*", "", text)


def rel_literal(lit: str, rel: str) -> str:
"""The baseline form: the literal as written, so a line stays recognisable next to its file."""
return f"{rel}\t{lit}"


def scan(root: pathlib.Path) -> dict[str, list[str]]:
"""{file: [literal, ...]} for every in-scope site, sorted and de-duplicated."""
found: dict[str, list[str]] = {}
excluded_self: set[str] = set()
scan.excluded_self = excluded_self # read by --check so the exclusion is printed, not implicit
for top in SCAN:
base = root / top
if not base.is_dir():
continue
for path in sorted(base.rglob("*")):
if not path.is_file() or path.suffix not in SUFFIXES:
continue
rel = path.relative_to(root).as_posix()
if rel == SELF:
excluded_self.add(rel)
if is_test(rel) or rel == SELF:
continue
try:
text = path.read_text(encoding="utf-8", errors="replace")
except OSError:
continue
# A trailing quote is stripped so the baseline records the PATH: re-quoting a site (`"/tmp/x"` ->
# `/tmp/x`) is not a new site, and the two spellings of one path would otherwise be two baseline lines.
hits = [m.group(0).rstrip("\"'") for m in LITERAL.finditer(strip_comments(text, path.suffix))]
if hits:
found[rel] = sorted(set(hits))
return found


def read_baseline(path: pathlib.Path) -> set[str]:
if not path.is_file():
return set()
out = set()
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if line and not line.startswith("#"):
out.add(line)
return out


def write_baseline(path: pathlib.Path, found: dict[str, list[str]]) -> None:
lines = [
"# Hardcoded /tmp paths under npu/, app/, tools/, scripts/ - the sites that exist today.",
"# A ram on this box is tmpfs, so each of these is anonymous memory when it is written.",
"# This list may only SHRINK: fix a site, then delete its line. A new site fails",
"# tools/ci/tmp_path_check.py --check, and so does a line whose site is gone.",
"# Regenerate with: tools/ci/tmp_path_check.py --write-baseline",
"",
]
for rel in sorted(found):
for lit in found[rel]:
lines.append(rel_literal(lit, rel))
path.write_text("\n".join(lines) + "\n", encoding="utf-8")


def main(argv: list[str]) -> int:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
g = ap.add_mutually_exclusive_group(required=True)
g.add_argument("--check", action="store_true")
g.add_argument("--self-test", action="store_true")
g.add_argument("--write-baseline", action="store_true")
ap.add_argument("--root", default=".")
args = ap.parse_args(argv)
root = pathlib.Path(args.root).resolve()
baseline_path = root / "tools" / "ci" / "tmp_paths.txt"

found = scan(root)
have = {rel_literal(lit, rel) for rel, lits in found.items() for lit in lits}

if args.write_baseline:
write_baseline(baseline_path, found)
print(f"tmp_path_check: wrote {len(have)} site(s) to {baseline_path}")
return 0

if args.self_test:
return self_test()

base = read_baseline(baseline_path)
new = sorted(have - base)
gone = sorted(base - have)
for line in new:
print(f"tmp_path_check: NEW hardcoded /tmp path: {line.replace(chr(9), ' ')}")
for line in gone:
print(
"tmp_path_check: baseline line has no site any more (fixed or renamed): "
f"{line.replace(chr(9), ' ')} - delete it if fixed"
)
if new or gone:
print(
f"tmp_path_check: FAIL ({len(new)} new, {len(gone)} stale of {len(base)} baselined) - /tmp is tmpfs "
"on this box, so route the path through ONEBIT_DUMP_DIR, $XDG_CACHE_HOME/1bit/scratch or a "
"$TMPDIR the caller set, instead"
)
return 1
excluded = len(getattr(scan, "excluded_self", ()))
print(f"tmp_path_check: {len(base)} baselined site(s), none new, none stale "
f"({excluded} file(s) excluded: this checker's own fixtures)")
return 0


def self_test() -> int:
import tempfile

bad = 0

def check(ok: bool, what: str) -> None:
nonlocal bad
print((" ok " if ok else " FAIL ") + what)
if not ok:
bad += 1

with tempfile.TemporaryDirectory() as td:
root = pathlib.Path(td)
(root / "npu" / "forward").mkdir(parents=True)
(root / "tools" / "ci").mkdir(parents=True)
(root / "tests").mkdir()
# A live site, a site in a comment, a test file, and a path that merely contains "tmp".
(root / "npu" / "forward" / "dump.cpp").write_text(
'void f() { open("/tmp/npu_layer.bin"); }\n'
"// was: open(\"/tmp/old.bin\");\n"
'/* open("/tmp/block.bin"); */\n'
'const char* other = "/var/tmp/not_ours.bin";\n',
encoding="utf-8",
)
(root / "tests" / "dump_test.cpp").write_text('void t() { open("/tmp/x.bin"); }\n', encoding="utf-8")
(root / "tools" / "run.sh").write_text(
'LOG="${TMPDIR:-/tmp}/run.log"\n' "echo hi > /tmp/run2.log\n" "echo $# # a comment /tmp/nope\n",
encoding="utf-8",
)
found = scan(root)
check(found.get("npu/forward/dump.cpp") == ["/tmp/npu_layer.bin"],
"a live literal is found and the two commented copies are not")
check("/tmp/old.bin" not in str(found), "a commented-out path is not a site")
check("tests/dump_test.cpp" not in found, "a test file is out of scope")
check("/var/tmp/not_ours.bin" not in str(found), "/var/tmp is not /tmp")
# engine#2027 review: the bare shell spelling, which the first pattern missed
(root / "tools" / "bare.sh").write_text('echo y > /tmp/bare.log\ncp a /tmp/b2\n', encoding="utf-8")
bare = scan(root)
check(bare.get("tools/bare.sh") == ["/tmp/b2", "/tmp/bare.log"],
"a bare /tmp path is a site (no quotes): the engine#2027 review's hole")
check(all('"' not in s and "'" not in s for s in bare.get("tools/bare.sh", [])),
"and the bare form is recorded without a quote it never had")

# The ratchet: a new site fails, a stale baseline line fails, and an unchanged tree passes.
write_baseline(root / "tools" / "ci" / "tmp_paths.txt", found)
base = read_baseline(root / "tools" / "ci" / "tmp_paths.txt")
have = {rel_literal(lit, rel) for rel, lits in found.items() for lit in lits}
check(base == have, f"the baseline round-trips ({len(base)} site(s))")
check(not (have - base) and not (base - have), "an unchanged tree has no drift")

(root / "npu" / "forward" / "added.cpp").write_text('void g() { open("/tmp/new.bin"); }\n', encoding="utf-8")
added = {rel_literal(lit, rel) for rel, lits in scan(root).items() for lit in lits}
check(bool(added - base), "a new site is drift")
check(not (base - added), "and adding one leaves no stale line")

(root / "npu" / "forward" / "dump.cpp").unlink()
renamed = {rel_literal(lit, rel) for rel, lits in scan(root).items() for lit in lits}
check(bool(base - renamed), "a renamed file leaves a stale baseline line (the engine#1728 lesson)")

print("tmp_path_check self-test:", "ok" if bad == 0 else f"{bad} FAILED")
return 0 if bad == 0 else 1


if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
64 changes: 64 additions & 0 deletions tools/ci/tmp_paths.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Hardcoded /tmp paths under npu/, app/, tools/, scripts/ - the sites that exist today.
# A ram on this box is tmpfs, so each of these is anonymous memory when it is written.
# This list may only SHRINK: fix a site, then delete its line. A new site fails
# tools/ci/tmp_path_check.py --check, and so does a line whose site is gone.
# Regenerate with: tools/ci/tmp_path_check.py --write-baseline

app/npu_device_lock.h /tmp/1bit-npu-device.lock
app/serve.cpp /tmp/1bit-hybrid-XXXXXX
npu/forward/src/q4nx_forward.cpp /tmp/a0-XXXXXX
npu/forward/src/q4nx_forward.cpp /tmp/npu_layer.bin
npu/forward/src/q4nx_forward.cpp /tmp/q4nx_hidden_%d.bin
npu/ort/onebit_npu_gemm.cpp /tmp/1bit-npu-device.lock
npu/target/xdna/xdna_target.cpp /tmp/a0-XXXXXX
scripts/check-ubuntu-2604-lemond.sh /tmp/backend.log
scripts/check-ubuntu-2604-lemond.sh /tmp/engine
scripts/check-ubuntu-2604-lemond.sh /tmp/engine.tar.zst
scripts/check-ubuntu-2604-lemond.sh /tmp/engine.tar.zst:ro
scripts/check-ubuntu-2604-lemond.sh /tmp/engine/*/install-lemonade-ppa.sh
scripts/check-ubuntu-2604-lemond.sh /tmp/lemond.log
scripts/npu-window11.sh /tmp/window11-$LABEL.log
scripts/pkg-proof.sh /tmp/b1.log
scripts/pkg-proof.sh /tmp/b2.log
scripts/pkg-proof.sh /tmp/ecmod.ko
scripts/pkg-proof.sh /tmp/engine.log
scripts/pkg-proof.sh /tmp/engv.log
scripts/pkg-proof.sh /tmp/hb.log
scripts/pkg-proof.sh /tmp/i1.log
scripts/pkg-proof.sh /tmp/i2.log
scripts/pkg-proof.sh /tmp/install.log
scripts/pkg-proof.sh /tmp/k.log
scripts/pkg-proof.sh /tmp/kr.log
scripts/pkg-proof.sh /tmp/r.log
scripts/pkg-proof.sh /tmp/re.log
scripts/pkg-proof.sh /tmp/rr.log
tools/npu-arena/ceiling.sh /tmp/ceiling.XXXXXX
tools/npu-arena/make_probe.sh /tmp/arena-probe.\$\$.XXXXXX
tools/npu-arena/make_task.sh /tmp/arena-build.$$.XXXXXX
tools/npu-arena/oracle.sh /tmp/arena-$TASK-$$.last
tools/repro-hostgemm-lock.sh /tmp/1bit-npu-device.lock
tools/repro-hostgemm-lock.sh /tmp/hg-lock-$LABEL.log
tools/showcase-612-npu-streams.sh /tmp/1bit-npu-device.lock
tools/showcase-612.sh /tmp/1bit-npu-device.lock
tools/spec_decode_acceptance_hrx.sh /tmp/hrxacc-$PORT_D.log
tools/spec_decode_acceptance_hrx.sh /tmp/hrxacc-$PORT_T.log
tools/spec_decode_acceptance_hrx.sh /tmp/hrxacc-$port.log
tools/spec_decode_acceptance_hrx.sh /tmp/hrxacc-prompt.txt
tools/spec_decode_speedup.sh /tmp/specdec-$label.acc
tools/spec_decode_speedup.sh /tmp/specdec-$label.log
tools/spec_decode_speedup.sh /tmp/specdec-B.log
tools/spec_decode_speedup.sh /tmp/specdec-{lab}.acc
9 changes: 6 additions & 3 deletions tools/copyright.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,16 @@
# engine#1820: the other sweep's allow-list, same shape and the same reason -- `#`-comment data, `.txt` has no
# style, and the header it wants is already at the top of the file.
"unreferenced_tests.txt",
# engine#1946: the ctest skip baseline, the same shape and the same reason -- a '#'-comment data file whose header
# is the notice it would otherwise be asked for. (CI caught this on the PR: the tool refuses an unknown style rather
# engine#1946: the ctest skip baseline, the same shape and the same reason -- a '#'-comment data file whose
# is the notice it would otherwise be asked for. (CI caught this on the PR: the tool refuses an unknown sty
# than passing it, which is the behaviour worth having.)
"ctest_skips.txt",
# engine#2018: the third test-registration baseline, same shape and the same reason -- data, `.txt` has no style.
# engine#2018: the test-registration baseline of `tests/*_test.py`, same shape and the same reason -- data,
"unreferenced_py_tests.txt",
"hrx_f16_narrowing.txt",
# engine#2027: the `#`-comment baseline for hardcoded /tmp paths, same shape and the same reason --
# data, `.txt` has no style, and the notice it wants is already at the top of the file.
"tmp_paths.txt",
"21serial-console", "90onboard-limits", "Dockerfile", "*.Dockerfile",
# cycle 17: Puppy's switch that skips its first-run Quick Setup is an empty marker file whose only content
# is this notice - Puppy tests for its existence, so a comment is free
Expand Down
Loading