From 7d242c885e90f3f162748ff6100f09aed0f89e37 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Fri, 13 Nov 2026 01:06:13 -0400 Subject: [PATCH 1/2] tools/ci/lane_syntax_check.py: the brace error CI cannot see, because CI never compiles the file engine#2330 said main did not compile. Checking it, **every commit in the window reported `build: success`**, including `273f6e9df`, whose own subject is that sentence. The two are both right, and the reason is the lane: CMakeLists.txt:2003 if(ONEBIT_NPU) CMakeLists.txt:2108 target_sources(onebit PRIVATE app/forward_serve.cpp) `app/forward_serve.cpp` is added to `onebit` **only inside the lane guard**, and CI builds with `ONEBIT_NPU=OFF`. Measured in CI's own build log: `app/serve.cpp`, `app/main.cpp` and `app/embed_serve.cpp` each have a `Building CXX object` line; **`app/forward_serve.cpp` has none**, while `[298/331] Linking CXX executable 1bit` succeeds. So CI was correct about the configuration it builds, and main was red **only for the lane** -- invisible until someone built on a device. A brace imbalance needs no headers, no runtime and no link to detect, which is the one class that does survive that guard. This adds: lane_syntax_check.py # 181 files checked on main, 0 imbalanced, 5 named as not checked lane_syntax_check.py --self-test # 9 arms **Proven both ways against the real break**, with a compiler as the oracle: * the pre-fix tree `40643dd4e`: `app/forward_serve.cpp: 337 '{' vs 336 '}' first unclosed '{' at line 81` -> **rc 1**, and `g++` on that tree says `app/forward_serve.cpp:282:76: error: expected '}' before '.' token` -- the same file, the same class, without needing the NPU runtime; * current main: **rc 0**, the file balanced. Four decisions: 1. **A raw string runs to `)delim"`, not to the next quote.** `npu_decl_check.strip_code` counts `R"(a"b{"` as a string ending at the inner quote; measured here, **44 of the 54 raw literals under `app/` and `npu/` contain a `"`**, and with `strip_code` alone `app/serve.cpp` reports 2 opens and 0 closes. The lexer tracks the delimiter. (The naive count is not an option either: `app/serve.cpp` reads 654/651 that way and 591/591 lexed.) 2. **Preprocessor-aware, and honest about what it cannot resolve.** With the lane defines, `app/serve.cpp` read 614/612 only because mutually exclusive `#if` branches were both counted. Inactive branches are now blanked; the 5 files whose conditions need a bare macro (`#if defined(ONEBIT_HRX_ALLOW_NO_SVM) && ONEBIT_HRX_ALLOW_NO_SVM`) are **named as not checked**, rc 0 -- a wrong guess would either pass a broken file or fail a good one. 3. **9 arms, and the filter arm is mutation-proven**: keeping inactive branches fails `an inactive #if 0 branch is skipped` with `2/1`. 4. **Self-test caught my own malformed arm.** `R"(a"b{d"` is not a valid raw string (it must close `)"`), so the lexer was right and the test was wrong -- the arm now closes properly. Offline, python-only, no build input: one ctest row `lane_syntax_selftest` **Passed 0.03 s**, `copyright --check` rc 0, `ci_targets_check --check` rc 0. --- CMakeLists.txt | 6 + tools/ci/lane_syntax_check.py | 255 ++++++++++++++++++++++++++++++++++ 2 files changed, 261 insertions(+) create mode 100644 tools/ci/lane_syntax_check.py diff --git a/CMakeLists.txt b/CMakeLists.txt index 7a45c29b4..b5f07155c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -241,6 +241,12 @@ add_test(NAME config_pin_check_selftest COMMAND python3 ${CMAKE_SOURCE_DIR}/tool # branch's side dropped a section that had merged that morning (#2314). Offline, git-only, no build input. add_test(NAME doc_section_loss_selftest COMMAND python3 ${CMAKE_SOURCE_DIR}/tools/ci/doc_section_loss_check.py --self-test WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}) +# engine#2330: CI builds with ONEBIT_NPU=OFF, so a source added to a target only inside `if(ONEBIT_NPU)` is never +# compiled by it -- `app/forward_serve.cpp` was one `}` short after #2327 and main did not compile for the lane, while +# every commit reported `build: success`. A brace imbalance needs no headers, no runtime and no link, so this catches +# the class in CI. Files whose `#if` conditions cannot be resolved are named as not checked, never guessed at. +add_test(NAME lane_syntax_selftest COMMAND python3 ${CMAKE_SOURCE_DIR}/tools/ci/lane_syntax_check.py --self-test + WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}) add_dependencies(onebit npu_decode_note_test) # engine#2106: a shipped kernel set's own consistency -- manifest vs bytes, one form or none, name vs directory. The # refusal this pre-empts ("differ in size") needed private-repo archaeology to diagnose; the symptom is checkable here. diff --git a/tools/ci/lane_syntax_check.py b/tools/ci/lane_syntax_check.py new file mode 100644 index 000000000..eb0bcb611 --- /dev/null +++ b/tools/ci/lane_syntax_check.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Copyright 2026 The 1bit-MONSTER Authors. +"""Balance the braces of every C++ source CI cannot compile. + +`CI` builds with `ONEBIT_NPU=OFF`, and a source added to a target only inside +`if(ONEBIT_NPU)` is therefore never compiled by it: `app/forward_serve.cpp` +appears exactly once in `CMakeLists.txt`, at `target_sources(onebit PRIVATE +app/forward_serve.cpp)` inside that block, and the CI build log of main contains +zero `Building CXX object .../forward_serve.cpp.o` lines while it does compile +`app/serve.cpp`, `app/main.cpp` and `app/embed_serve.cpp`. + +That is intended (the lane needs the private NPU runtime), but it means a +**syntax** error in those files is invisible until someone builds on a device. +engine#2330 is exactly that: `app/forward_serve.cpp` was one `}` short after +#2327 and main did not compile for the lane until the fix-forward; every commit +in the window still reported `build: success`, because CI was right about the +configuration it builds. + +A brace imbalance needs no headers, no runtime and no link to detect, so this +check closes that class cheaply. It reports; it does not judge a file it cannot +read (rc 2). + +**Why not `npu_decl_check.strip_code`:** measured on this tree, 44 of the 54 raw +string literals under `app/` and `npu/` contain a `"`, and `strip_code` treats +`R"(a"b)"` as a string ending at the inner quote - `app/serve.cpp` then looks +like 2 opens and 0 closes. A raw string runs to `)delim"`, so the lexer below +tracks its delimiter. + +A file whose `#if` conditions this cannot resolve is **named as not checked**, never +guessed at: a wrong guess here would either pass a broken file or fail a good one. +""" +from __future__ import annotations + +import argparse +import os +import re +import sys + +ROOTS = ("app", "npu") +EXTS = (".cpp", ".cc", ".c", ".h", ".hpp") + + +LANE_DEFINES = ("ONEBIT_NPU", "ONEBIT_MOE", "ONEBIT_LAYA", "ONEBIT_HRX_SERVER", "__linux__") + +COND = re.compile(r"^\s*#\s*(ifdef|ifndef|if|elif|else|endif)\b(.*)$") + + +def _truth(expr: str, defines): + """Return True/False for a simple `#if` expression, or None when unresolved.""" + e = expr.strip() + while e.startswith("(") and e.endswith(")"): + e = e[1:-1].strip() + if e in ("0", "1"): + return e == "1" + m = re.fullmatch(r"(!?)\s*defined\s*\(?\s*([A-Za-z_]\w*)\s*\)?", e) + if m: + inside = m.group(2) in defines + return (not inside) if m.group(1) else inside + parts = re.split(r"\s*(&&|\|\|)\s*", e) + if len(parts) == 1: + return None + vals = [] + for i, part in enumerate(parts): + if i % 2: + vals.append(part) + continue + v = _truth(part, defines) + if v is None: + return None + vals.append(v) + out = vals[0] + for i in range(1, len(vals), 2): + out = (out and vals[i + 1]) if vals[i] == "&&" else (out or vals[i + 1]) + return out + + +def active_lines(text: str, defines): + """Blank inactive preprocessor branches; return (text, unresolved line or None).""" + out, stack, active = [], [], True + for lineno, line in enumerate(text.split("\n"), 1): + m = COND.match(line) + if not m: + out.append(line if active else " " * len(line)) + continue + kind, rest = m.group(1), m.group(2).strip() + if kind in ("ifdef", "ifndef"): + val = (rest in defines) + if kind == "ifndef": + val = not val + else: + val = _truth(rest, defines) if kind in ("if", "elif") else True + if val is None: + return None, lineno + if kind in ("ifdef", "ifndef", "if"): + stack.append((active, val)) + active = active and val + elif kind == "elif": + parent, taken = stack[-1] + if taken: + active = False + else: + active = parent and bool(val) + stack[-1] = (parent, bool(val)) + elif kind == "else": + parent, taken = stack[-1] + active = parent and not taken + stack[-1] = (parent, True) + else: # endif + active = stack.pop()[0] if stack else True + out.append(" " * len(line)) + return "\n".join(out), None + + +def tokens(text: str): + """Yield (offset, char) for code characters only, tracking line numbers.""" + i, n = 0, len(text) + while i < n: + c = text[i] + two = text[i:i + 2] + if two == "//": + j = text.find("\n", i) + i = n if j < 0 else j + elif two == "/*": + j = text.find("*/", i + 2) + i = n if j < 0 else j + 2 + elif text.startswith('R"', i) and (i == 0 or not (text[i - 1].isalnum() or text[i - 1] == "_")): + # R"delim( ... )delim" + k = text.find("(", i + 2) + if k < 0: + i += 2 + continue + delim = text[i + 2:k] + close = ")" + delim + '"' + j = text.find(close, k + 1) + i = n if j < 0 else j + len(close) + elif c in "\"'": + j = i + 1 + while j < n and text[j] != c and text[j] != "\n": + j += 2 if text[j] == "\\" else 1 + i = min(j + 1, n) + else: + yield i, c + i += 1 + + +def imbalance(text: str): + """Return (opens, closes, first-unmatched-open-line) after lexing.""" + depth = 0 + stack = [] + opens = closes = 0 + for off, ch in tokens(text): + if ch == "{": + opens += 1 + stack.append(off) + depth += 1 + elif ch == "}": + closes += 1 + depth -= 1 + if stack: + stack.pop() + line = text.count("\n", 0, stack[-1]) + 1 if stack else None + return opens, closes, line + + +def check(root: str): + bad, unreadable, checked, skipped = [], [], 0, [] + for top in ROOTS: + for dirpath, _, files in os.walk(os.path.join(root, top)): + for name in sorted(files): + if not name.endswith(EXTS): + continue + path = os.path.join(dirpath, name) + try: + text = open(path, encoding="utf-8", errors="replace").read() + except OSError as exc: + unreadable.append((path, str(exc))) + continue + filtered, unknown = active_lines(text, LANE_DEFINES) + if filtered is None: + skipped.append((os.path.relpath(path, root), unknown)) + continue + checked += 1 + opens, closes, line = imbalance(filtered) + if opens != closes: + bad.append((os.path.relpath(path, root), opens, closes, line)) + return bad, unreadable, checked, skipped + + +def self_test() -> int: + arms = [ + ("balanced", "int f(){ if(x){ return 1; } return 0; }", True), + ("one short", "int f(){ if(x){ return 1; }", False), + ("brace in a string does not count", 'int f(){ const char* s = "{"; }', True), + ("brace in a comment does not count", "int f(){ } // one more }", True), + ("a raw string with a quote does not count", 'int f(){ auto s = R"(a"b{c)d)"; }', True), + ("a delimited raw string does not count", 'int f(){ auto s = R"j(}{(})j"; }', True), + ("char literal brace does not count", "int f(){ char c = '{'; }", True), + ("an inactive #if 0 branch is skipped", "#if 0\nint broken(){\n#endif\nint f(){ return 0; }", True), + ("the ONEBIT_NPU branch is the one counted", "#ifdef ONEBIT_NPU\nint f(){\n#endif\nint g(){ return 0; }", False), + ] + failed = 0 + for name, src, ok in arms: + filtered, unknown = active_lines(src, LANE_DEFINES) + if filtered is None: + print(f" FAIL {name}: unexpectedly unresolved at line {unknown}") + failed += 1 + continue + opens, closes, _ = imbalance(filtered) + got = opens == closes + if got != ok: + print(f" FAIL {name}: expected balanced={ok}, got {got} ({opens}/{closes})") + failed += 1 + else: + print(f" ok {name}") + return 1 if failed else 0 + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--root", default=".", help="repository root (default: .)") + ap.add_argument("--self-test", action="store_true", help="check the lexer's arms and exit") + args = ap.parse_args(argv) + if args.self_test: + return self_test() + bad, unreadable, checked, skipped = check(args.root) + for path, err in unreadable: + print(f"lane_syntax: cannot read {path}: {err}") + if bad: + for path, opens, closes, line in bad: + where = f" first unclosed '{{' at line {line}" if line else "" + print(f"lane_syntax: {path}: {opens} '{{' vs {closes} '}}'{where}") + print(f"lane_syntax: {len(bad)} of {checked} file(s) are brace-imbalanced") + return 1 + for path, line in skipped: + print(f"lane_syntax: not checked {path}: unresolved #if condition at line {line}") + print(f"lane_syntax: {checked} file(s) balanced, {len(skipped)} not checked") + return 2 if unreadable else 0 + + +if __name__ == "__main__": + sys.exit(main()) From 9e584a3bfa651a9147fa46880ea5f09c5b1147f3 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Fri, 13 Nov 2026 01:17:05 -0400 Subject: [PATCH 2/2] lane_syntax_check: cover the other lane-only trees too (181 -> 338 files, still 0 imbalanced) `app/` and `npu/` are not where the lane-only sources end: measured from CMakeLists, **124 sources outside those two trees** sit inside an `if(ONEBIT_NPU)` block -- `moe/`, `laya/`, `tools/` (e.g. mtp_head_bench.cpp) and `tests/` (e.g. moe_gguf_index_test.cpp). The same argument applies to them, so they are checked: before: 181 file(s) balanced, 5 not checked (app, npu) after: 338 file(s) balanced, 8 not checked (app, npu, moe, laya, tools, tests) Still rc 0 on main, so the wider scope finds nothing today -- which is the point of running it before the next break, not after. The 3 extra unresolved files (`tests/attn_math_test.cpp:49`, `tests/quant_i8_row_test.cpp:63`, one more) are named, not guessed. self-test 0.10 s in ctest, copyright rc 0. --- tools/ci/lane_syntax_check.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tools/ci/lane_syntax_check.py b/tools/ci/lane_syntax_check.py index eb0bcb611..6997f9d39 100644 --- a/tools/ci/lane_syntax_check.py +++ b/tools/ci/lane_syntax_check.py @@ -50,7 +50,9 @@ import re import sys -ROOTS = ("app", "npu") +# Every tree that holds a source added to a target inside `if(ONEBIT_NPU)`: measured from CMakeLists, +# 124 such sources live outside app/ and npu/ (moe/, laya/, tools/, tests/). +ROOTS = ("app", "npu", "moe", "laya", "tools", "tests") EXTS = (".cpp", ".cc", ".c", ".h", ".hpp")