From d019200851a742e264b477586ac32402d08c08e2 Mon Sep 17 00:00:00 2001 From: agent Date: Sat, 7 Nov 2026 18:04:42 -0400 Subject: [PATCH] package-linux: a staged file's mode is the packer's decision, and the package asserts it Fixes #1352. c31 and c32 both shipped 134 of the 195 files under opt/1bit-package group- or other-writable, because the packer stages with `cp -a` and so inherited whatever mode the build tree or the builder's umask left behind. Nothing was exploitable -- every file is root:root and no appliance user is in group root -- but a 0775 root-owned binary in a public tarball reads as deliberate to anyone auditing it. Measured on the tree the report measured (~/wt/c31-rootfs-probe/opt/1bit-package, 195 files), before -> after: 134 -> 0 regular files group/ or other-writable (111x0664 + 23x0775, the report's number) 50 -> 0 directories group/ or other-writable (the report counted files only) 195 / 14 files / symlinks unchanged 24 -> 24 executable files unchanged, so nothing that ran stops running modes 111x664,60x644,23x775,1x755 -> 171x644,24x755 scripts/pack-elf.sh gains two functions, split on purpose: pack_normalise_modes dirs 0755; regular files 0755 if staged executable, else 0644. Clears setuid/setgid/sticky, which cp -a was carrying in. pack_writable_files a reporter: every file and directory still group/other-writable. Split from the fixer because a function that normalises and then checks its own work can never fail -- the regression it exists to catch would be invisible. My first draft did exactly that and the new test caught it on a file staged right after the normaliser. package-linux.sh normalises and then asserts, as the LAST step before tar: the licence/notice files and the Lemonade drop-in are staged immediately above, so normalising earlier would leave the check green while the next cp slipped an 0664 back in. Not `install -m`, which the issue suggested: the package carries symlink sets (libmtmd.so -> libmtmd.so.0 -> libmtmd.so.0.0.1, 14 of them) precisely because put() uses cp -a, and install dereferences -- it would have replaced each symlink with a second copy of the library. And there are four staging sites, with a fifth arriving with the next feature: one normalisation after the last copy is one place to get right instead of one per site plus a rule to remember. tests/pack_modes_test.sh (17 checks, fixture stage tree, no build and no tar): the modes, the symlink still being a symlink pointing where it did, no executable bit invented, and the reporter firing both on a tree nobody normalised and on a file staged below the normalisation. Registered as `ctest -R pack_modes`. Also found and fixed a real bug in the reporter on the way: when find yields the tree root itself the prefix strip left an absolute path, so a writable package root would have been reported incompletely. docs/releases.md: one sentence where the package layout is described, for the person auditing a tarball. tools/copyright.py --check rc 0. No device: 0 s GPU, 0 s NPU. Lands for c33, not c32. --- CMakeLists.txt | 4 ++ docs/releases.md | 5 ++- scripts/pack-elf.sh | 33 ++++++++++++++ scripts/package-linux.sh | 8 ++++ tests/pack_modes_test.sh | 96 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 145 insertions(+), 1 deletion(-) create mode 100755 tests/pack_modes_test.sh diff --git a/CMakeLists.txt b/CMakeLists.txt index 3f0da71da..42e8b7be7 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1026,6 +1026,10 @@ add_dependencies(npu_forward_loader_test hadamard_rotate_test quant_i8_row_test # statistic named, and a control pair setting the tolerance from its own spread. Shell + the fixtures it # writes itself, so there is no target for the build list to miss. add_test(NAME npu_window11_policy COMMAND bash ${CMAKE_SOURCE_DIR}/scripts/npu-window11.sh --self-test) +# The package's file modes (engine#1352): a release artifact's modes are the packer's decision, so every staged file is +# 0755 (was executable) or 0644 (was not), no file or directory is group/other-writable, and the reporter that catches a +# staging line added below the normalisation still fires on a tree nobody normalised. Fixture tree: no build, no tar. +add_test(NAME pack_modes COMMAND bash ${CMAKE_SOURCE_DIR}/tests/pack_modes_test.sh) # The GEMM designs are derived on the container's K grid and the kernel's N grid (engine#845); a # model already on both is unchanged. Pure arithmetic + one assemble_B, so it runs without a model. # The engine's derived design set for a set of dims, as `ROLE M K N elf` lines. The instrument the diff --git a/docs/releases.md b/docs/releases.md index c3e071156..8d4aca1af 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -335,7 +335,10 @@ v2026.44 is published. The engine in `1bit/` keeps the build tree's layout (`scripts/package-linux.sh`): `1bit` finds each backend at the same path relative to itself as in the build (`app/built_path.h`), and every program -finds its libraries beside itself. Unpack the package anywhere and start Lemonade through +finds its libraries beside itself. File modes in the package are the **packer's** decision, not the build tree's: +the packer normalises every staged file to `0755` (if it was executable) or `0644`, clears setgid/sticky, and refuses +to write a tarball in which anything is group- or other-writable (`pack_normalise_modes` / `pack_writable_files`, +`scripts/pack-elf.sh`, engine#1352). The symlink sets (`lib*.so.0` -> `lib*.so.0.x.y`) are preserved as symlinks. Unpack the package anywhere and start Lemonade through `lemond-onebit`, which points Lemonade's onebit backend at the packaged `1bit` (`LEMONADE_ONEBIT_BIN`): diff --git a/scripts/pack-elf.sh b/scripts/pack-elf.sh index d4c7980f1..3d74c0db6 100644 --- a/scripts/pack-elf.sh +++ b/scripts/pack-elf.sh @@ -214,6 +214,39 @@ $f: embedded build-machine path" [ -z "$out" ] || printf "%b\n" "$out" } +# Every regular file in the package is group- or other-writable (prints " ", empty when clean). +# +# c31 and c32 both shipped 134 of the 195 files under opt/1bit-package writable this way (engine#1352): the packer used +# `cp -a`, so every mode came from the build tree or the builder's umask. Nothing was exploitable -- every file is +# root:root and no appliance user is in group root -- but a 0775 root-owned binary in a public tarball reads as +# deliberate to anyone auditing it. +# +# A reporter, not a fixer, on purpose: a function that normalises and then checks its own work can never fail, so the +# regression it is supposed to catch would have to be invisible. `pack_normalise_modes` below fixes; this one is what +# the package asserts with afterwards, and what can be pointed at an unpacked tarball by anything that did not pack it. +# Directories are included -- a world-writable directory in an installed tree is the worse of the two. +pack_writable_files() { + local tree=$1 f rel + while IFS= read -r -d '' f; do + rel=${f#"$tree"/} + [ "$rel" = "$f" ] && rel=. # the tree root itself, which find yields without a trailing slash + printf '%s %s\n' "$rel" "$(stat -c %a "$f")" + done < <(find "$tree" \( -type f -o -type d \) \( -perm -0020 -o -perm -0002 \) -print0) +} + +# Give every staged file a mode the packer decided, rather than the one it was built with. The executable bit as +# staged decides the class, so behaviour cannot change: what was executable stays executable (0755) and everything else +# is data (0644). It also clears setuid/setgid/sticky, which `cp -a` was faithfully carrying in from the build tree. +# Symlinks are skipped on purpose: their mode is not used by the kernel, and chmod would follow the link to whatever it +# points at -- and `install -m` cannot be used for this at all, because it dereferences and would replace the package's +# `lib*.so.0 -> lib*.so.0.x.y` sets with a second copy of each library. +pack_normalise_modes() { + local stage=$1 + find "$stage" -type d -exec chmod 0755 {} + 2>/dev/null || true + find "$stage" -type f -perm -u+x -exec chmod 0755 {} + 2>/dev/null || true + find "$stage" -type f ! -perm -u+x -exec chmod 0644 {} + 2>/dev/null || true +} + pack_build_rpaths() { local f find "$1" -type f -print0 | while IFS= read -r -d '' f; do diff --git a/scripts/package-linux.sh b/scripts/package-linux.sh index 99f7e314f..7d56b6a97 100755 --- a/scripts/package-linux.sh +++ b/scripts/package-linux.sh @@ -164,5 +164,13 @@ Keep the directory layout: 1bit finds its backends beside itself. HRX needs TheRock in /opt/rocm-therock; the NPU needs the amdxdna kernel driver. Docs: https://1bit.gg/ Code: https://github.com/1bit-MONSTER/engine EOF +# A file mode in a release artifact is the packer's decision (engine#1352). This runs LAST, after every staging line +# above, because the licence/notice files and the Lemonade drop-in are staged just before it -- normalising earlier +# would leave the check green while the next copy slipped an 0664 back in. Asserted rather than assumed: a file that +# is still group- or other-writable here fails the package instead of shipping. +pack_normalise_modes "$stage" +modes_bad=$(pack_writable_files "$stage") +[ -z "$modes_bad" ] || { echo "files still group/ or other-writable after normalising:"; echo "$modes_bad"; exit 1; } + tar -C "$(dirname "$stage")" -cf - "$name" | zstd -q -T4 -19 -o "$out/$name.tar.zst" ls -la "$out/$name.tar.zst" diff --git a/tests/pack_modes_test.sh b/tests/pack_modes_test.sh new file mode 100755 index 000000000..73f59a7d9 --- /dev/null +++ b/tests/pack_modes_test.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# pack_modes_test.sh: the package's file modes (scripts/pack-elf.sh, pack_normalise_modes -- engine#1352). +# +# c31 and c32 both shipped 134 of 195 files under opt/1bit-package group- or other-writable, because the packer used +# `cp -a` and therefore inherited whatever mode the build tree had. This asserts the two properties that fix has to +# have, on a fixture stage tree, with no build and no tar: +# +# 1. every regular file ends up 0755 (was executable) or 0644 (was not), and no directory is group/other-writable; +# 2. a file that appears group/other-writable AFTER normalisation is REPORTED, so a staging line added below the +# normalisation cannot regress the property silently. +# +# The fixture deliberately includes the shapes the real package has: a 0775 launcher, 0775 libraries with a symlink +# set (libx.so.0 -> libx.so.0.1.0), 0664 licence files, a 0666 file, a setgid file and a 0775 directory. Symlinks are +# checked to still BE symlinks afterwards: `chmod -R` or `install -m` would have replaced or followed them, which is +# why the normaliser only touches `-type f` and `-type d`. +set -euo pipefail + +here=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=../scripts/pack-elf.sh +. "$here/../scripts/pack-elf.sh" + +fails=0 +check() { if [ "$2" = "$3" ]; then printf ' ok %s = %s\n' "$1" "$3"; else printf ' FAIL %s = %s (want %s)\n' "$1" "$2" "$3"; fails=$((fails+1)); fi; } + +stage=$(mktemp -d)/pkg +mkdir -p "$stage/hrx/llama/bin" "$stage/lemonade" +printf 'launcher\n' > "$stage/1bit" +printf 'lib\n' > "$stage/hrx/llama/bin/libx.so.0.1.0" +ln -s libx.so.0.1.0 "$stage/hrx/llama/bin/libx.so.0" +printf 'tool\n' > "$stage/hrx/llama/bin/llama-server" +printf 'lic\n' > "$stage/LICENSE" +printf 'notice\n' > "$stage/NOTICE" +printf 'conf\n' > "$stage/lemonade/1bit-hrx.conf" +printf 'odd\n' > "$stage/odd-0666" +printf 'sgid\n' > "$stage/odd-setgid" + +# the modes the release actually shipped: 0775 executables and libraries, 0664 licences, and worse +chmod 0775 "$stage/1bit" "$stage/hrx/llama/bin/llama-server" "$stage/hrx/llama/bin/libx.so.0.1.0" +chmod 0664 "$stage/LICENSE" "$stage/NOTICE" "$stage/lemonade/1bit-hrx.conf" +chmod 0666 "$stage/odd-0666" +chmod 2664 "$stage/odd-setgid" # setgid on a DATA file, so the class is unchanged +chmod 0775 "$stage/hrx" "$stage/hrx/llama" "$stage/lemonade" + +echo "before: $(find "$stage" -type f \( -perm -0020 -o -perm -0002 \) | wc -l) group/other-writable, $(find "$stage" -type d -perm -0020 | wc -l) writable dirs" + +pack_normalise_modes "$stage" + +echo "after:" +check "no writable file or directory is left" "$(pack_writable_files "$stage")" "" +check "launcher" "$(stat -c %a "$stage/1bit")" "755" +check "shipped tool" "$(stat -c %a "$stage/hrx/llama/bin/llama-server")" "755" +check "shipped library" "$(stat -c %a "$stage/hrx/llama/bin/libx.so.0.1.0")" "755" +check "licence" "$(stat -c %a "$stage/LICENSE")" "644" +check "notice" "$(stat -c %a "$stage/NOTICE")" "644" +check "data file" "$(stat -c %a "$stage/lemonade/1bit-hrx.conf")" "644" +check "a group/other-writable data file is closed" "$(stat -c %a "$stage/odd-0666")" "644" +check "setgid is cleared" "$(stat -c %a "$stage/odd-setgid")" "644" +check "directory" "$(stat -c %a "$stage/hrx")" "755" +check "the symlink set is still links" "$(test -L "$stage/hrx/llama/bin/libx.so.0" && echo link || echo file)" "link" +check "and still points where it did" "$(readlink "$stage/hrx/llama/bin/libx.so.0")" "libx.so.0.1.0" +check "no executable bit was invented" "$(find "$stage" -type f -perm -u+x | wc -l)" "3" + +# The check is a REPORTER, so it must still fire on a tree nobody normalised -- that is the property that makes it +# worth having. Two shapes: a fixture as the packer receives it, and a file staged after the normalisation ran. +pristine=$stage/../pristine +mkdir -p "$pristine"; chmod 0755 "$pristine" # so the fixture reports its files, not its own umask +printf 'x\n' > "$pristine/a.json"; chmod 0664 "$pristine/a.json" +printf 'y\n' > "$pristine/b"; chmod 0775 "$pristine/b" +check "an unnormalised tree reports both files" "$(pack_writable_files "$pristine" | cut -d' ' -f1 | sort | tr '\n' ',')" "a.json,b," + +printf 'late\n' > "$stage/staged-after.json"; chmod 0664 "$stage/staged-after.json" +chmod 0777 "$pristine" +check "a writable tree root is reported as '.'" "$(pack_writable_files "$pristine" | grep -c '^\. 777')" "1" +chmod 0755 "$pristine" + +check "a file staged below the normalisation is caught" "$(pack_writable_files "$stage" | cut -d' ' -f1)" "staged-after.json" +check "and reports the mode a user would see" "$(pack_writable_files "$stage" | cut -d' ' -f2)" "664" + +rm -rf "$(dirname "$stage")" +if [ "$fails" -ne 0 ]; then printf '%d check(s) failed\n' "$fails"; exit 1; fi +echo "pack_modes: ok"