From f7d13d638728efbda2d782a565c34999948d69f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Mon, 28 Sep 2026 03:52:56 +0000 Subject: [PATCH 1/9] feat: make the three-build comparison of decision 0010 runnable Handbook decision 0010 step 3 requires a three-build comparison, control against unit against control again, for every component taken out of the shared tree. Nothing ran it. The capture half was /root/measure/build.sh on the build host, a scratch file in root's home directory on one machine, and the comparison half was never committed anywhere, so keel-mariadb#11 and keel-postgresql#7 were approved on a number only their author could produce. bt-layer-measure does both halves. capture builds one layer under the build lock and records its package list, file tree, symlinks, layer manifest and state paths under a directory named on the command line; compare reports what two captures differ by; attribute reports what the unit build differs from the control by that the control does not differ from itself by, and prints a block a pull request quotes whole, so a claim names the command behind it. Three things the hand-run measurement got wrong, and this does not: Paths are keyed on pathhash, not on sha256sum's " ". A Debian rootfs holds setuptools/_vendor/jaraco/text/Lorem ipsum.txt, which a whitespace split turns into two fields that never match themselves; that phantom cost the LAMP measurement of 2026-09-27 one file in each of its two counts (docs/traps.md, "A path with a space in it"). A path carrying a tab, a newline or a backslash is refused rather than mis-counted. A path inside the noise floor is no longer subtracted on the strength of its name. For mariadb the floor is 179 files of which 173 are ./var/lib/mysql/**, and that directory holds mysql/global_priv and mysql/user.*, so a component whose build-time job includes deleting accounts was measured blind exactly where the accounts live. share/layer-state-paths lists the paths that can hold accounts, credentials, keys or database content; their bytes are kept at capture time and compared per line for a text file and per byte offset for a binary one, in both pairs. A clock that moves in every build touches the same line or offset in both and cancels; a deleted account row leaves one the control pair never produces and is reported as real, with a three-way diff. A file that could not be read is not a file that did not change. A state path above the sample cap is recorded as not sampled and fails the run, so no report can say "0 attributable differences" about bytes it never saw. Logic in bin/layer-measure-lib, thin main in bt-layer-measure, tests in tests/measure, documented in docs/layer-measure.md. tests/coverage.sh runs the new suite: 379 of 379 lines in the library and 64 of 64 in the executable, every subcommand, every exit code and every error path, gate unchanged at 99. Closes #10 --- COVERAGE.md | 49 ++- bin/layer-measure-lib | 741 ++++++++++++++++++++++++++++++++++++++++ bt-layer-measure | 195 +++++++++++ docs/layer-measure.md | 142 ++++++++ share/layer-state-paths | 46 +++ tests/coverage.sh | 5 +- tests/measure | 507 +++++++++++++++++++++++++++ 7 files changed, 1676 insertions(+), 9 deletions(-) create mode 100644 bin/layer-measure-lib create mode 100755 bt-layer-measure create mode 100644 docs/layer-measure.md create mode 100644 share/layer-state-paths create mode 100755 tests/measure diff --git a/COVERAGE.md b/COVERAGE.md index 8ac0c2c..bbd6e8e 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -37,13 +37,6 @@ installed: tests/coverage.sh # threshold 95, or COVERAGE_THRESHOLD, or the first argument -## Not measured - -`tests/appname-version` exercises the inherited `bin/appname-version` by hand -and is not run under kcov; the other `bt-*` scripts (about 40 shell files) -have no test. They follow the 0004 treatment when the project touches them, -and the repository total is remeasured then. - ## 2026-09-26: the build audit `layer_audit_packages` reads the built rootfs's dpkg status and refuses a @@ -116,3 +109,45 @@ carries the mark and one that does not. | bin/generate-signature | 87 of 87 | 100 percent | The gate stays at 99. + +## 2026-09-28: the three-build comparison + +`bt-layer-measure` and `bin/layer-measure-lib` make handbook decision 0010 +step 3 executable: a capture subcommand that builds a layer and records its +package list, file tree, symlinks and state paths, a compare subcommand over +two captures and an attribute subcommand over the three the decision asks +for. `tests/coverage.sh` now runs `tests/measure` as a fourth suite and +measures eight files: + +| File | Lines covered | Cover | +|------|---------------|-------| +| bin/layer-lib | 331 of 332 | 99 percent | +| bt-layer | 110 of 111 | 99 percent | +| bin/aplinfo-lib | 186 of 186 | 100 percent | +| bt-aplinfo | 47 of 47 | 100 percent | +| bin/signature-lib | 76 of 76 | 100 percent | +| bin/generate-signature | 87 of 87 | 100 percent | +| bin/layer-measure-lib | 379 of 379 | 100 percent | +| bt-layer-measure | 64 of 64 | 100 percent | + +The gate stays at 99, the lowest file rounded down. Both new files are at +100 percent line and branch: every subcommand, every exit code (0, 1, 2 and +3) and every error path is exercised, including the comparison itself, where +`tests/measure` covers a difference inside the noise floor that is real and +has to be reported (an account row gone from a text state file, and an edited +account record in a binary one, both at paths whose clocks move in every +build) as well as a run where nothing was sampled and the measurement fails +rather than reporting a clean number it could not see. + +`tests/measure` sets `pipefail` the way `bt-layer-measure` does. Without it +the suite passed while the executable did not: `measure_signature` inherited +`cmp`'s "the files differ" status from a pipeline and reported a failure +instead of a signature, which silently dropped one state path's verdict from +every report. + +## Not measured + +`tests/appname-version` exercises the inherited `bin/appname-version` by hand +and is not run under kcov; the other `bt-*` scripts have no test. They follow +the 0004 treatment when the project touches them, and the repository total is +remeasured then. diff --git a/bin/layer-measure-lib b/bin/layer-measure-lib new file mode 100644 index 0000000..03d5157 --- /dev/null +++ b/bin/layer-measure-lib @@ -0,0 +1,741 @@ +#!/bin/bash +# Copyright (c) 2026 KeelLinux +# +# This file is part of buildtasks. +# +# Buildtasks is free software; you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published by the +# Free Software Foundation; either version 3 of the License, or (at your +# option) any later version. +# +# Functions shared by bt-layer-measure and tests/measure. Meant to be +# sourced. Every function reads its inputs from its arguments, prints its +# result on stdout and returns non-zero with a message on stderr instead of +# exiting, so the caller decides what is fatal. +# +# The vocabulary, which handbook decision 0010 uses without defining: +# +# capture one build of one layer, recorded as a directory of +# tab separated files keyed on the path +# noise floor the paths two builds of the *same* recipe differ at; a build +# of a changed recipe is allowed to differ there too +# attributable a difference the noise floor does not account for, which is +# the only kind that says anything about the change +# state path a path that can hold accounts, credentials, keys or database +# content; a difference at one of these is never subtracted on +# the strength of its name, its bytes are compared three ways + +# Paths are keyed on field 1 and separated by a tab, never by a space: +# a Debian rootfs contains setuptools/_vendor/jaraco/text/Lorem ipsum.txt, +# and sha256sum's default " " turns it into two fields that +# never match themselves (handbook docs/traps.md, "A path with a space in +# it"). A tab, a newline or a backslash in a path would break the format in +# the same way, and sha256sum escapes the last two, so a capture refuses +# them rather than mis-counting. +MEASURE_TSV_FIELDS_LINKS=2 + +# How many items a one-line summary names before it says "and N more". +MEASURE_LIST_MAX=8 + +# How many lines of a diff the evidence block prints per pair. +MEASURE_EVIDENCE_LINES=20 + +# Default ceiling on a single state file's sampled copy. Above it the bytes +# are not kept, the comparison has nothing to compare and says so; it never +# silently treats an unread file as unchanged. 16 MiB covers a MariaDB +# system table and a PostgreSQL global directory with room to spare. +MEASURE_STATE_MAX_BYTES_DEFAULT=16777216 + +# Files every capture directory carries. +MEASURE_CAPTURE_FILES="meta.tsv pkgs.tsv tree.tsv links.tsv state.tsv" + +measure_error() { echo "ERROR [layer-measure]: $*" 1>&2; } + +# measure_lines PREFIX TEXT [STREAM] +# One line at a time, never through word splitting: this whole tool exists +# because a path with a space in it was split into two fields once already. +measure_lines() { + local prefix=$1 text=$2 line + [[ -n "$text" ]] || return 0 + while IFS= read -r line; do + [[ -n "$line" ]] || continue + printf '%s%s\n' "$prefix" "$line" + done <<<"$text" + return 0 +} + +# --------------------------------------------------------------------------- +# path safety +# --------------------------------------------------------------------------- + +# measure_path_safe PATH +# A path this format can key on: no tab (the separator), no newline (the +# record terminator), no backslash (which sha256sum escapes). +measure_path_safe() { + [[ "$1" != *$'\t'* ]] || return 1 + [[ "$1" != *$'\n'* ]] || return 1 + [[ "$1" != *\\* ]] || return 1 + return 0 +} + +# measure_paths_safe ROOT +# The same rule over a whole tree, before anything is written. +measure_paths_safe() { + local root=$1 + if [[ ! -d "$root" ]]; then + measure_error "no such directory: $root" + return 1 + fi + local bad + local tab; tab=$(printf '\t') + bad=$(cd "$root" && find . -xdev \( -name "*$tab*" -o -name '*\\*' \) -printf '%p\n' 2>/dev/null | head -n "$MEASURE_LIST_MAX") + if [[ -n "$bad" ]]; then + measure_error "a path with a tab or a backslash in it, which this format cannot key on:" + measure_lines " " "$bad" 1>&2 + return 1 + fi + local entries lines + entries=$(cd "$root" && find . -xdev -printf 'x\n' | wc -l) + lines=$(cd "$root" && find . -xdev -printf '%p\n' | wc -l) + if [[ "$entries" != "$lines" ]]; then + measure_error "a path with a newline in it under $root" + return 1 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# capture: turning a built rootfs into files a second person can re-read +# --------------------------------------------------------------------------- + +# measure_tree_record ROOT -> pathsha256sizemode, C sorted +measure_tree_record() { + local root=$1 + measure_paths_safe "$root" || return 1 + local meta hash rc=0 + meta=$(mktemp) + hash=$(mktemp) + (cd "$root" && find . -xdev -type f -printf '%p\t%s\t%m\n') > "$meta" || rc=1 + (cd "$root" && find . -xdev -type f -print0 | xargs -0 -r sha256sum --) > "$hash" || rc=1 + if [[ $rc -eq 0 ]]; then + # sha256sum prints "" then two spaces then the path; the path + # starts at column 67 and needs no unescaping, because a path that + # sha256sum would escape was refused above. + LC_ALL=C awk -F'\t' -v OFS='\t' 'NR == FNR { size[$1] = $2; mode[$1] = $3; next } { p = substr($0, 67); print p, substr($0, 1, 64), size[p], mode[p] }' "$meta" "$hash" | LC_ALL=C sort + fi + rm -f "$meta" "$hash" + [[ $rc -eq 0 ]] || measure_error "could not read the tree under $root" + return $rc +} + +# measure_links_record ROOT -> pathtarget, C sorted +measure_links_record() { + local root=$1 + measure_paths_safe "$root" || return 1 + local out + out=$(cd "$root" && find . -xdev -type l -printf '%p\t%l\n' | LC_ALL=C sort) + if [[ -n "$out" ]] && ! awk -F'\t' -v n="$MEASURE_TSV_FIELDS_LINKS" \ + 'NF != n { exit 1 }' <<<"$out"; then + measure_error "a symlink target with a tab in it under $root" + return 1 + fi + [[ -z "$out" ]] || printf '%s\n' "$out" + return 0 +} + +# measure_pkgs_record ROOT -> nameversion for the installed packages +# only, C sorted. Packages left as config-files are not in the layer. +measure_pkgs_record() { + local root=$1 + local dpkg_query=${MEASURE_DPKG_QUERY:-dpkg-query} + if ! command -v "$dpkg_query" >/dev/null; then + measure_error "$dpkg_query not available, cannot read the package list" + return 1 + fi + if [[ ! -d "$root/var/lib/dpkg" ]]; then + measure_error "no dpkg database under $root" + return 1 + fi + "$dpkg_query" --admindir="$root/var/lib/dpkg" -W \ + -f '${Package}\t${Version}\t${Status}\n' 2>/dev/null \ + | awk -F'\t' -v OFS='\t' '{ split($3, s, " "); if (s[3] == "installed") print $1, $2 }' \ + | LC_ALL=C sort +} + +# --------------------------------------------------------------------------- +# state paths +# --------------------------------------------------------------------------- + +# measure_state_patterns FILE -> one glob per line, comments and blanks gone +measure_state_patterns() { + local file=$1 + if [[ ! -f "$file" ]]; then + measure_error "no state path list at $file" + return 1 + fi + local globs + globs=$(sed -e 's/[[:space:]]*#.*$//' -e '/^[[:space:]]*$/d' -e 's/[[:space:]]*$//' "$file") + if [[ -z "$globs" ]]; then + measure_error "the state path list $file declares no pattern" + return 1 + fi + printf '%s\n' "$globs" +} + +# measure_state_match PATH GLOBS +# GLOBS is the newline separated output of measure_state_patterns. The match +# is bash pattern matching, where * crosses a slash, so ./var/lib/mysql/** +# covers the whole directory. +measure_state_match() { + local path=$1 globs=$2 glob + [[ -n "$globs" ]] || return 1 + while IFS= read -r glob; do + [[ -n "$glob" ]] || continue + # shellcheck disable=SC2053 + if [[ "$path" == $glob ]]; then + return 0 + fi + done <<<"$globs" + return 1 +} + +# measure_state_sample ROOT TREE_TSV DEST CAP GLOBS +# Copies the bytes of every state path at or below CAP into DEST/state/, +# and records every state path in DEST/state.tsv as sampled or too-large. +# The bytes are what lets the comparison say what differs rather than that +# something differs. +measure_state_sample() { + local root=$1 tree=$2 dest=$3 cap=$4 globs=$5 + local path size dst rows=() row + : > "$dest/state.tsv" + mapfile -t rows < "$tree" + for row in "${rows[@]}"; do + IFS=$'\t' read -r path _ size _ <<<"$row" + measure_state_match "$path" "$globs" || continue + if [[ "$size" -gt "$cap" ]]; then + printf '%s\t%s\t%s\n' "$path" too-large "$size" >> "$dest/state.tsv" + continue + fi + dst=$dest/state/${path#./} + mkdir -p "$(dirname "$dst")" + cp -- "$root/${path#./}" "$dst" + printf '%s\t%s\t%s\n' "$path" sampled "$size" >> "$dest/state.tsv" + done + return 0 +} + +# measure_sample_path CAPTURE PATH -> where that capture kept the bytes +measure_sample_path() { + printf '%s/state/%s\n' "$1" "${2#./}" +} + +# --------------------------------------------------------------------------- +# reading a capture +# --------------------------------------------------------------------------- + +# measure_capture_check DIR +measure_capture_check() { + local dir=$1 f + if [[ ! -d "$dir" ]]; then + measure_error "no capture at $dir" + return 1 + fi + for f in $MEASURE_CAPTURE_FILES; do + if [[ ! -f "$dir/$f" ]]; then + measure_error "$dir is not a complete capture: $f is missing" + return 1 + fi + done + return 0 +} + +# measure_meta_get DIR KEY +measure_meta_get() { + measure_value "$1/meta.tsv" "$2" +} + +# measure_count FILE -> the number of records, 0 when the file is not there +measure_count() { + if [[ ! -f "$1" ]]; then + echo 0 + return 0 + fi + wc -l < "$1" | tr -d ' ' +} + +# measure_value FILE KEY -> everything after the first field, or nothing +measure_value() { + LC_ALL=C awk -F'\t' -v k="$2" \ + '$1 == k { i = index($0, "\t"); print substr($0, i + 1); exit }' "$1" +} + +# measure_only_in A B -> the keys of A that B does not have +measure_only_in() { + LC_ALL=C awk -F'\t' 'NR == FNR { seen[$1]; next } !($1 in seen) { print $1 }' "$2" "$1" +} + +# measure_changed A B -> the keys both have, whose remaining fields differ +measure_changed() { + LC_ALL=C awk -F'\t' '{ i = index($0, "\t"); v = substr($0, i + 1) } NR == FNR { a[$1] = v; next } ($1 in a) && a[$1] != v { print $1 }' "$1" "$2" +} + +# --------------------------------------------------------------------------- +# the nature of one difference +# --------------------------------------------------------------------------- + +# measure_is_text FILE +measure_is_text() { + local f=$1 + if [[ ! -f "$f" ]]; then + measure_error "no such file: $f" + return 1 + fi + local all stripped + all=$(wc -c < "$f") + stripped=$(tr -d '\0' < "$f" | wc -c) + [[ "$all" == "$stripped" ]] +} + +# measure_signature A B -> the tokens by which A and B differ, C sorted: +# o line n of A is not in B where it was (text) +# n line n of B is not in A where it was (text) +# @ byte n differs (binary) +# len the two files are not the same length (binary) +# +# The tokens, and not the bytes themselves, are what the noise floor is +# subtracted on: a clock that moves in both pairs touches the same line or +# the same offset in both, so its tokens cancel, while a deleted account +# leaves a token the control pair never produces. +measure_signature() { + local a=$1 b=$2 + if [[ ! -f "$a" ]] || [[ ! -f "$b" ]]; then + measure_error "cannot compare $a with $b: one of them is not a file" + return 1 + fi + if measure_is_text "$a" && measure_is_text "$b"; then + diff --old-line-format=$'o%dn\n' --new-line-format=$'n%dn\n' --unchanged-line-format='' -- "$a" "$b" | LC_ALL=C sort -u + return 0 + fi + local sa sb tokens + sa=$(wc -c < "$a") + sb=$(wc -c < "$b") + # cmp exits 1 when the files differ, which is the normal case here, so + # its status is collected rather than inherited: under pipefail an + # inherited 1 made this function report a failure instead of a signature. + tokens=$(cmp -l -- "$a" "$b" 2>/dev/null | awk '{ print "@" $1 }' || true) + [[ "$sa" == "$sb" ]] || tokens="len"$'\n'"$tokens" + [[ -n "$tokens" ]] || return 0 + LC_ALL=C sort -u <<<"$tokens" | grep . + return 0 +} + +# measure_set_minus A B -> the lines of A that are not lines of B +measure_set_minus() { + [[ -n "$1" ]] || return 0 + LC_ALL=C comm -23 <(LC_ALL=C sort -u <<<"$1") <(LC_ALL=C sort -u <<<"$2") +} + +# measure_nature_list TOKENS -> the same tokens as a sentence a reviewer reads +measure_nature_list() { + local token out=() n=0 total sorted=() + total=$(grep -c . <<<"$1" || true) + mapfile -t sorted < <(LC_ALL=C sort -V <<<"$1") + for token in "${sorted[@]}"; do + [[ -n "$token" ]] || continue + n=$((n + 1)) + if [[ $n -gt $MEASURE_LIST_MAX ]]; then + out+=("and $((total - MEASURE_LIST_MAX)) more") + break + fi + case "$token" in + len) out+=("the length") ;; + o*) out+=("control line ${token#o}") ;; + n*) out+=("other line ${token#n}") ;; + @*) out+=("byte ${token#@}") ;; + *) out+=("$token") ;; + esac + done + [[ ${#out[@]} -gt 0 ]] || return 0 + local IFS=, + printf '%s\n' "${out[*]}" | sed 's/,/, /g' +} + +# measure_state_verdict CONTROL UNIT CONTROL_AGAIN PATH -> verdictwhy +# +# noise the unit build differs from the control only where the +# control differs from itself +# real it differs somewhere the control pair does not, so the change +# being measured is what differs, whatever the path is called +# unsampled no bytes were kept on some side, so neither can be said +measure_state_verdict() { + local c=$1 u=$2 c2=$3 path=$4 + local sc su sc2 side + sc=$(measure_sample_path "$c" "$path") + su=$(measure_sample_path "$u" "$path") + sc2=$(measure_sample_path "$c2" "$path") + for side in "$sc" "$su" "$sc2"; do + if [[ ! -f "$side" ]]; then + printf 'unsampled\t%s\n' \ + "not sampled in every build (no bytes kept at $side), so nothing can be said about it" + return 0 + fi + done + local ra rn residue + ra=$(measure_signature "$sc" "$su") || return 1 + rn=$(measure_signature "$sc" "$sc2") || return 1 + residue=$(measure_set_minus "$ra" "$rn") + if [[ -z "$residue" ]]; then + printf 'noise\t%s\n' \ + "every difference here is one the control pair shows against itself ($(grep -c . <<<"$ra" || true) in total)" + return 0 + fi + printf 'real\t%s\n' \ + "differences the control pair does not show against itself ($(grep -c . <<<"$residue" || true) of them): $(measure_nature_list "$residue")" +} + +# measure_state_evidence CONTROL UNIT CONTROL_AGAIN PATH +# What actually differs, both ways round, so the verdict above can be +# checked rather than believed. +measure_state_evidence() { + local c=$1 u=$2 c2=$3 path=$4 + local sc su sc2 + sc=$(measure_sample_path "$c" "$path") + su=$(measure_sample_path "$u" "$path") + sc2=$(measure_sample_path "$c2" "$path") + echo " $path" + local side + for side in "$sc" "$su" "$sc2"; do + if [[ ! -f "$side" ]]; then + echo " not sampled: $side is absent, raise --state-max-bytes and capture again" + return 0 + fi + done + echo " control vs unit:" + measure_pair_evidence "$sc" "$su" + echo " control vs control again (the noise floor at this path):" + measure_pair_evidence "$sc" "$sc2" + return 0 +} + +# measure_pair_evidence A B +measure_pair_evidence() { + local a=$1 b=$2 + if measure_is_text "$a" && measure_is_text "$b"; then + diff -u --label control --label other -- "$a" "$b" \ + | tail -n +3 | head -n "$MEASURE_EVIDENCE_LINES" | sed 's/^/ /' + return 0 + fi + local all offsets n + all=$(cmp -l -- "$a" "$b" 2>/dev/null | awk '{ print $1 }' || true) + n=$(grep -c . <<<"$all" || true) + offsets=$(head -n "$MEASURE_LIST_MAX" <<<"$all" | paste -sd, - || true) + printf ' sizes %s and %s; differing bytes: %s; first offsets: %s\n' \ + "$(wc -c < "$a")" "$(wc -c < "$b")" "$n" "${offsets:-none}" + return 0 +} + +# --------------------------------------------------------------------------- +# arguments +# --------------------------------------------------------------------------- + +MEASURE_SUBCOMMANDS="capture compare attribute" + +# measure_parse_args ARGS... +# Sets MEASURE_SUB and one variable per option. Returns 1 on anything it +# does not recognise, so the caller can exit 2 for a usage error. The +# variables are read by bt-layer-measure, not here. +# shellcheck disable=SC2034 +measure_parse_args() { + MEASURE_SUB= + MEASURE_DIR= + MEASURE_LAYER= + MEASURE_PARENT= + MEASURE_TAG= + MEASURE_A= + MEASURE_B= + MEASURE_CONTROL= + MEASURE_UNIT= + MEASURE_CONTROL2= + MEASURE_HELP= + MEASURE_CAP=$MEASURE_STATE_MAX_BYTES_DEFAULT + local opt val + while [[ $# -gt 0 ]]; do + opt=$1 + case "$opt" in + -h|--help) + MEASURE_HELP=yes + shift + continue + ;; + --dir|--layer|--parent|--tag|--a|--b|--control|--unit|--control-again|--state-max-bytes) + if [[ $# -lt 2 ]]; then + measure_error "$opt needs a value" + return 1 + fi + val=$2 + shift 2 + ;; + -*) + measure_error "unknown option: $opt" + return 1 + ;; + *) + if [[ -n "$MEASURE_SUB" ]]; then + measure_error "one subcommand at a time, not '$MEASURE_SUB' and '$opt'" + return 1 + fi + MEASURE_SUB=$opt + shift + continue + ;; + esac + case "$opt" in + --dir) MEASURE_DIR=$val ;; + --layer) MEASURE_LAYER=$val ;; + --parent) MEASURE_PARENT=$val ;; + --tag) MEASURE_TAG=$val ;; + --a) MEASURE_A=$val ;; + --b) MEASURE_B=$val ;; + --control) MEASURE_CONTROL=$val ;; + --unit) MEASURE_UNIT=$val ;; + --control-again) MEASURE_CONTROL2=$val ;; + --state-max-bytes) MEASURE_CAP=$val ;; + esac + done + return 0 +} + +# measure_require WHAT VALUE +measure_require() { + if [[ -z "$2" ]]; then + measure_error "$MEASURE_SUB needs --$1" + return 1 + fi + return 0 +} + +# measure_check_args +measure_check_args() { + case "$MEASURE_SUB" in + capture) + measure_require dir "$MEASURE_DIR" || return 1 + measure_require layer "$MEASURE_LAYER" || return 1 + measure_require tag "$MEASURE_TAG" || return 1 + if [[ ! "$MEASURE_TAG" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]]; then + measure_error "a tag names a directory: it starts with a letter or a digit and holds only letters, digits, dot, dash and underscore" + return 1 + fi + if [[ ! "$MEASURE_CAP" =~ ^[0-9]+$ ]]; then + measure_error "--state-max-bytes takes a number of bytes, not '$MEASURE_CAP'" + return 1 + fi + ;; + compare) + measure_require dir "$MEASURE_DIR" || return 1 + measure_require a "$MEASURE_A" || return 1 + measure_require b "$MEASURE_B" || return 1 + ;; + attribute) + measure_require dir "$MEASURE_DIR" || return 1 + measure_require control "$MEASURE_CONTROL" || return 1 + measure_require unit "$MEASURE_UNIT" || return 1 + measure_require control-again "$MEASURE_CONTROL2" || return 1 + ;; + *) + measure_error "unknown subcommand '$MEASURE_SUB' (one of: $MEASURE_SUBCOMMANDS)" + return 1 + ;; + esac + return 0 +} + +# --------------------------------------------------------------------------- +# reporting +# --------------------------------------------------------------------------- + +# How many paths a report names before it says how many it left out. +MEASURE_PRINT_MAX=40 + +# measure_print_set LABEL SET +measure_print_set() { + local label=$1 set=$2 n + n=$(grep -c . <<<"$set" || true) + [[ "$n" -gt 0 ]] || return 0 + printf ' %s (%s):\n' "$label" "$n" + measure_lines " " "$(head -n "$MEASURE_PRINT_MAX" <<<"$set")" + if [[ "$n" -gt "$MEASURE_PRINT_MAX" ]]; then + printf ' and %s more\n' "$((n - MEASURE_PRINT_MAX))" + fi + return 0 +} + +# measure_describe DIR TAG -> the one line a report prints per capture +measure_describe() { + printf '%s %s on %s, epoch %s, captured %s, %s packages, %s files, %s symlinks\n' \ + "$2" \ + "$(measure_meta_get "$1/$2" layer)" \ + "$(measure_meta_get "$1/$2" parent)" \ + "$(measure_meta_get "$1/$2" epoch)" \ + "$(measure_meta_get "$1/$2" captured)" \ + "$(measure_count "$1/$2/pkgs.tsv")" \ + "$(measure_count "$1/$2/tree.tsv")" \ + "$(measure_count "$1/$2/links.tsv")" +} + +# measure_compare_one A B KIND -> the report for one kind, 1 if it differs +measure_compare_one() { + local a=$1 b=$2 kind=$3 + local onlya onlyb changed na nb nc + onlya=$(measure_only_in "$a/$kind.tsv" "$b/$kind.tsv") + onlyb=$(measure_only_in "$b/$kind.tsv" "$a/$kind.tsv") + changed=$(measure_changed "$a/$kind.tsv" "$b/$kind.tsv") + na=$(grep -c . <<<"$onlya" || true) + nb=$(grep -c . <<<"$onlyb" || true) + nc=$(grep -c . <<<"$changed" || true) + printf '%s: %s differing, %s only in the first, %s only in the second\n' \ + "$kind" "$nc" "$na" "$nb" + measure_print_set "differing" "$changed" + measure_print_set "only in the first" "$onlya" + measure_print_set "only in the second" "$onlyb" + [[ $((na + nb + nc)) -eq 0 ]] +} + +# measure_compare_report DIR A B -> 0 when the two captures are identical +measure_compare_report() { + local dir=$1 a=$2 b=$3 kind differs=0 + echo "=== bt-layer-measure compare ===============================================" + printf 'command bt-layer-measure compare --dir %s --a %s --b %s\n' "$dir" "$a" "$b" + printf 'first %s\n' "$(measure_describe "$dir" "$a")" + printf 'second %s\n' "$(measure_describe "$dir" "$b")" + for kind in pkgs links tree; do + measure_compare_one "$dir/$a" "$dir/$b" "$kind" || differs=1 + done + if [[ $differs -eq 0 ]]; then + echo "the two captures are identical: same packages, same symlinks, same files" + fi + echo "============================================================================" + return $differs +} + +# measure_attributable A B C2_BASE C2_OTHER KIND WHICH +# The differences between A and B that the control pair does not produce. +# changed keys both have whose record differs +# gone keys A has and B does not +# added keys B has and A does not +measure_attributable() { + local a=$1 b=$2 c=$3 c2=$4 kind=$5 which=$6 + local pair floor + case "$which" in + changed) + pair=$(measure_changed "$a/$kind.tsv" "$b/$kind.tsv") + floor=$(measure_changed "$c/$kind.tsv" "$c2/$kind.tsv") + ;; + gone) + pair=$(measure_only_in "$a/$kind.tsv" "$b/$kind.tsv") + floor=$(measure_only_in "$c/$kind.tsv" "$c2/$kind.tsv") + ;; + added) + pair=$(measure_only_in "$b/$kind.tsv" "$a/$kind.tsv") + floor=$(measure_only_in "$c2/$kind.tsv" "$c/$kind.tsv") + ;; + esac + measure_set_minus "$pair" "$floor" +} + +# measure_state_paths C U GLOBS -> every state path the unit pair differs at +measure_state_paths() { + local c=$1 u=$2 globs=$3 path all=() sorted=() + mapfile -t all < <(measure_changed "$c/tree.tsv" "$u/tree.tsv"; measure_only_in "$c/tree.tsv" "$u/tree.tsv"; measure_only_in "$u/tree.tsv" "$c/tree.tsv") + [[ ${#all[@]} -gt 0 ]] || return 0 + mapfile -t sorted < <(printf '%s\n' "${all[@]}" | LC_ALL=C sort -u) + for path in "${sorted[@]}"; do + [[ -n "$path" ]] || continue + measure_state_match "$path" "$globs" && printf '%s\n' "$path" + done + return 0 +} + +# measure_attribute_report DIR CONTROL UNIT CONTROL_AGAIN GLOBS GLOBS_FILE +# The three-build comparison handbook decision 0010 step 3 asks for. Returns +# 0 only when nothing is attributable to the change and nothing inside the +# noise floor was either real or left unread. +measure_attribute_report() { + local dir=$1 ctl=$2 unit=$3 ctl2=$4 globs=$5 globs_file=$6 + local C=$dir/$ctl U=$dir/$unit C2=$dir/$ctl2 + local kind which set n attributable=0 + echo "=== bt-layer-measure attribution ===========================================" + printf 'command bt-layer-measure attribute --dir %s --control %s --unit %s --control-again %s\n' \ + "$dir" "$ctl" "$unit" "$ctl2" + printf 'state paths %s (sha256 %s)\n' "$globs_file" \ + "$(sha256sum < "$globs_file" | cut -c1-16)" + printf 'control %s\n' "$(measure_describe "$dir" "$ctl")" + printf 'unit %s\n' "$(measure_describe "$dir" "$unit")" + printf 'control again %s\n' "$(measure_describe "$dir" "$ctl2")" + echo + echo "noise floor, the control against itself:" + for kind in pkgs links tree; do + printf ' %-6s %s differing, %s only in the first, %s only in the second\n' "$kind" \ + "$(grep -c . <<<"$(measure_changed "$C/$kind.tsv" "$C2/$kind.tsv")" || true)" \ + "$(grep -c . <<<"$(measure_only_in "$C/$kind.tsv" "$C2/$kind.tsv")" || true)" \ + "$(grep -c . <<<"$(measure_only_in "$C2/$kind.tsv" "$C/$kind.tsv")" || true)" + done + echo "the control against the unit:" + for kind in pkgs links tree; do + printf ' %-6s %s differing, %s only in the first, %s only in the second\n' "$kind" \ + "$(grep -c . <<<"$(measure_changed "$C/$kind.tsv" "$U/$kind.tsv")" || true)" \ + "$(grep -c . <<<"$(measure_only_in "$C/$kind.tsv" "$U/$kind.tsv")" || true)" \ + "$(grep -c . <<<"$(measure_only_in "$U/$kind.tsv" "$C/$kind.tsv")" || true)" + done + echo + local detail="" + for kind in pkgs links tree; do + for which in changed gone added; do + set=$(measure_attributable "$C" "$U" "$C" "$C2" "$kind" "$which") + n=$(grep -c . <<<"$set" || true) + attributable=$((attributable + n)) + [[ "$n" -gt 0 ]] || continue + detail="$detail$(measure_print_set "$kind $which" "$set")"$'\n' + done + done + printf 'attributable differences: %s\n' "$attributable" + echo " (the control-unit pair minus the control-control pair, over packages, symlinks and files)" + [[ -z "$detail" ]] || printf '%s' "$detail" + echo + + local paths verdict why path noise=0 real=0 unsampled=0 evidence="" + paths=$(measure_state_paths "$C" "$U" "$globs") + printf 'state paths the unit pair differs at: %s\n' "$(grep -c . <<<"$paths" || true)" + echo " (a path that can hold accounts, credentials, keys or database content is" + echo " never subtracted because it fell inside the noise floor; its bytes are" + echo " compared in both pairs and the difference is named)" + while IFS= read -r path; do + [[ -n "$path" ]] || continue + IFS=$'\t' read -r verdict why < <(measure_state_verdict "$C" "$U" "$C2" "$path") + case "$verdict" in + noise) noise=$((noise + 1)) ;; + real) real=$((real + 1)) ;; + unsampled) unsampled=$((unsampled + 1)) ;; + esac + printf ' %-10s %s\n' "$verdict" "$path" + printf ' %s\n' "$why" + if [[ "$verdict" != noise ]]; then + evidence="$evidence$(measure_state_evidence "$C" "$U" "$C2" "$path")"$'\n' + fi + done <<<"$paths" + printf ' totals: noise %s, real %s, not sampled %s\n' "$noise" "$real" "$unsampled" + if [[ -n "$evidence" ]]; then + echo + echo "evidence for every state path that is not plain noise:" + printf '%s' "$evidence" + fi + echo + if [[ $((attributable + real + unsampled)) -eq 0 ]]; then + echo "verdict: PASS" + echo "============================================================================" + return 0 + fi + printf 'verdict: FAIL (%s attributable, %s real inside the noise floor, %s not sampled)\n' \ + "$attributable" "$real" "$unsampled" + echo "============================================================================" + return 1 +} diff --git a/bt-layer-measure b/bt-layer-measure new file mode 100755 index 0000000..20be1ef --- /dev/null +++ b/bt-layer-measure @@ -0,0 +1,195 @@ +#!/bin/bash -e +# Copyright (c) 2026 KeelLinux +# +# This file is part of buildtasks. +# +# Buildtasks is free software; you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published by the +# Free Software Foundation; either version 3 of the License, or (at your +# option) any later version. + +[[ -z "$BT_DEBUG" ]] || set -x +set -o pipefail + +APP=$(basename "$0") +BT=${BT:-$(dirname "$(readlink -f "$0")")} +# shellcheck source=bin/layer-measure-lib +source "$BT/bin/layer-measure-lib" + +EXIT_DIFFERENCES=1 +EXIT_USAGE=2 +EXIT_INPUT=3 + +warning() { echo "WARNING [$APP]: $*"; } +info() { echo "INFO [$APP]: $*"; } +die() { local code=$1; shift; echo "FATAL [$APP]: $*" 1>&2; exit "$code"; } + +usage() { + cat< "$dest/build.log" 2>&1; then + tail -n 20 "$dest/build.log" 1>&2 + die "$EXIT_INPUT" "the build failed; its output is $dest/build.log" + fi + + local rootfs=$layers/$MEASURE_LAYER.rootfs + if [[ ! -x "$rootfs/bin/bash" ]]; then + die "$EXIT_INPUT" "no built rootfs at $rootfs" + fi + measure_tree_record "$rootfs" > "$dest/tree.tsv" \ + || die "$EXIT_INPUT" "could not record the file tree of $rootfs" + measure_links_record "$rootfs" > "$dest/links.tsv" \ + || die "$EXIT_INPUT" "could not record the symlinks of $rootfs" + measure_pkgs_record "$rootfs" > "$dest/pkgs.tsv" \ + || die "$EXIT_INPUT" "could not record the package list of $rootfs" + measure_state_sample "$rootfs" "$dest/tree.tsv" "$dest" "$MEASURE_CAP" "$globs" + + if [[ -f "$layers/$MEASURE_LAYER.manifest" ]]; then + cp -- "$layers/$MEASURE_LAYER.manifest" "$dest/manifest.txt" + else + warning "no layer manifest at $layers/$MEASURE_LAYER.manifest, so this capture records no provenance for the layer" + fi + + printf '%s\t%s\n' \ + tag "$MEASURE_TAG" \ + layer "$MEASURE_LAYER" \ + parent "${MEASURE_PARENT:-none}" \ + rootfs "$rootfs" \ + epoch "${SOURCE_DATE_EPOCH:-none}" \ + captured "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + builder "$builder" \ + buildtasks_commit "$(git -C "$BT" rev-parse HEAD 2>/dev/null || echo none)" \ + state_max_bytes "$MEASURE_CAP" \ + state_paths "$globs_file" \ + state_paths_sha256 "$(sha256sum < "$globs_file" | cut -d' ' -f1)" \ + > "$dest/meta.tsv" + + info "$MEASURE_TAG: $(measure_count "$dest/pkgs.tsv") packages, $(measure_count "$dest/tree.tsv") files, $(measure_count "$dest/links.tsv") symlinks, $(measure_count "$dest/state.tsv") state paths ($(grep -c "$(printf '\tsampled\t')" "$dest/state.tsv" || true) of them sampled)" + info "$MEASURE_TAG: recorded in $dest" +} + +do_compare() { + local tag + for tag in "$MEASURE_A" "$MEASURE_B"; do + measure_capture_check "$MEASURE_DIR/$tag" || die "$EXIT_INPUT" "no complete capture $tag in $MEASURE_DIR" + done + measure_compare_report "$MEASURE_DIR" "$MEASURE_A" "$MEASURE_B" \ + || exit "$EXIT_DIFFERENCES" +} + +do_attribute() { + local tag + for tag in "$MEASURE_CONTROL" "$MEASURE_UNIT" "$MEASURE_CONTROL2"; do + measure_capture_check "$MEASURE_DIR/$tag" || die "$EXIT_INPUT" "no complete capture $tag in $MEASURE_DIR" + done + local globs_file globs + globs_file=$(state_paths_file) + globs=$(measure_state_patterns "$globs_file") || die "$EXIT_INPUT" "no usable state path list" + measure_attribute_report "$MEASURE_DIR" "$MEASURE_CONTROL" "$MEASURE_UNIT" \ + "$MEASURE_CONTROL2" "$globs" "$globs_file" || exit "$EXIT_DIFFERENCES" +} + +measure_parse_args "$@" || { usage 1>&2; exit "$EXIT_USAGE"; } +if [[ -n "$MEASURE_HELP" ]]; then + usage + exit 0 +fi +if [[ -z "$MEASURE_SUB" ]]; then + usage 1>&2 + exit "$EXIT_USAGE" +fi +measure_check_args || exit "$EXIT_USAGE" + +case "$MEASURE_SUB" in + capture) do_capture ;; + compare) do_compare ;; + attribute) do_attribute ;; +esac diff --git a/docs/layer-measure.md b/docs/layer-measure.md new file mode 100644 index 0000000..6dcb019 --- /dev/null +++ b/docs/layer-measure.md @@ -0,0 +1,142 @@ +# bt-layer-measure: the three-build comparison + +Handbook decision 0010 accepts splitting components out of the shared `common` +tree on one condition, stated in step 3 of its order of work: components move +one at a time, "re-running this three-build comparison for each, including the +control-vs-control run". This is that comparison. Before this script existed +the capture half was a scratch file in root's home directory on the build host +and the comparison half was never written down, so two component extractions +(keel-mariadb#11, keel-postgresql#7) were approved on a number only their +author could produce. + +## Why three builds + +Two builds of the *same* recipe are not identical. They differ in install-time +state: clocks, log lines, Perl hash order, a database data directory created +when a conf script starts the server. That set is the **noise floor**, and it +is the only reason a build of a *changed* recipe is allowed to differ from the +control at all. Without a second control build there is nothing to subtract, +so no difference can be attributed to the change, which is what the third +build is for. + +## Usage + + bt-layer-measure capture --dir DIR --layer NAME [--parent NAME] --tag TAG + [--state-max-bytes N] + bt-layer-measure compare --dir DIR --a TAG --b TAG + bt-layer-measure attribute --dir DIR --control TAG --unit TAG --control-again TAG + +`--dir` is required by every subcommand and defaults to nothing, so the +directory holding the measurement is named in the command and a second person +runs the same command against the same directory. Nothing is kept in a home +directory on one machine. + +## The whole procedure, as run on the build host + +Three builds at one `SOURCE_DATE_EPOCH`, against one `common` commit and one +parent layer, or the comparison means nothing. `bt-layer-measure capture` +takes `/run/lock/keel-build.lock` with a bounded wait, so it queues behind +another build instead of racing it (handbook `docs/traps.md`, "Two builds on +one host, and one lock"). + + export FAB_PATH=/turnkey/fab-keel RELEASE=debian/trixie FAB_ARCH=amd64 + export BT_PATH=/turnkey/buildtasks-keel BT_PRODUCTS=$FAB_PATH/products + export TKLBAM_PATH=/turnkey/tklbam-profiles-keel BT_GPGKEY= + export SOURCE_DATE_EPOCH=1700000000 + M=/mnt/builds/measure/mariadb-$(date +%Y%m%d) + + # 1. the control: the recipe as it is on the default branch + $BT_PATH/bt-layer-measure capture --dir $M --layer mariadb --parent core --tag control + + # 2. the unit: the same recipe with the component supplied through unit.d + # (check out the branch that does that, then) + $BT_PATH/bt-layer-measure capture --dir $M --layer mariadb --parent core --tag unit + + # 3. the control again, from the same tree as step 1 + $BT_PATH/bt-layer-measure capture --dir $M --layer mariadb --parent core --tag control-again + + # 4. the result the pull request quotes + $BT_PATH/bt-layer-measure attribute --dir $M \ + --control control --unit unit --control-again control-again + +Step 4 prints one block. Paste it into the pull request whole: it names the +command, the three captures with their epoch and capture time, the noise +floor, the attributable count and the verdict, so a claim like "0 attributable +differences" says which command produced it and a reviewer can run that +command again. + +Exit status: 0 when the verdict is PASS, 1 when it is FAIL, 2 on a usage +error, 3 when a capture, a build or a tool is missing. + +## What a capture holds + +`DIR/TAG/`, all tab separated and keyed on the path in field 1: + +| File | Content | +|------|---------| +| `meta.tsv` | `keyvalue`: layer, parent, epoch, capture time, builder, buildtasks commit, the state path list and its digest | +| `pkgs.tsv` | `nameversion` for the installed packages, from the rootfs's own dpkg database | +| `tree.tsv` | `pathsha256sizemode` for every regular file | +| `links.tsv` | `pathtarget` for every symlink | +| `state.tsv` | `pathsampled\|too-largesize` for every state path | +| `state/` | the bytes of every sampled state path | +| `manifest.txt` | the layer manifest `bt-layer` wrote | +| `build.log` | the builder's output | + +A tag is never overwritten. Re-measuring means a new tag or a new directory, +so the numbers a pull request quoted stay on disk next to the ones that +replaced them. + +### Why a tab, and why the path comes first + +`sha256sum` prints `` then two spaces then the path, and a Debian +rootfs contains `setuptools/_vendor/jaraco/text/Lorem ipsum.txt`. Anything +splitting that on whitespace gets two fields out of one path, which never +matches itself, so it is reported as differing in every pair including the +control against itself. That is handbook `docs/traps.md`, "A path with a space +in it"; it cost the LAMP measurement of 2026-09-27 one phantom file in each of +its two counts. A capture therefore keys on `path...`, and refuses a path +containing a tab, a newline or a backslash rather than mis-counting it. + +## State paths: the part that is not a subtraction + +For the mariadb component the noise floor is 179 files, of which 173 are +`./var/lib/mysql/**`. That directory holds `mysql/global_priv` and +`mysql/user.*`, and the component's build-time job includes deleting accounts. +Subtracting the whole floor by path therefore made the measurement blind +exactly where the thing being measured would show up. A path is put inside the +noise floor by its name; whether its difference really is noise is a question +about its bytes, and nothing was asking it. + +`share/layer-state-paths` lists the paths that can hold state that matters: +accounts, credentials, host keys, database directories. For those, `capture` +keeps the bytes and `attribute` compares them in both pairs and classifies +each one: + +| Verdict | Meaning | +|---------|---------| +| `noise` | every difference between the control and the unit is one the control also shows against itself, at the same line or the same byte offset | +| `real` | it differs at a line or a byte offset the control pair does not, so the change is what differs, whatever the path is called | +| `unsampled` | the bytes were not kept on some side, so neither can be said | + +`real` and `unsampled` both fail the run. `unsampled` fails on purpose: a +measurement that could not read a file must not pass as if the file were +unchanged. Either raise `--state-max-bytes` (the default is 16 MiB per file) +or say in the pull request why that path does not matter. + +The comparison is per line for a text file and per byte offset for a binary +one, and the noise floor is subtracted at that level rather than per path. A +clock that moves in every build touches the same line or the same offset in +both pairs, so it cancels. A deleted account row leaves a line or an offset +the control pair never produces, so it is reported, with a three-way diff of +the control, the unit and the second control printed underneath. + +## Layout + +Logic in `bin/layer-measure-lib`, a thin main in `bt-layer-measure`, tests in +`tests/measure`, measured by `tests/coverage.sh` (Keel decisions 0003 and +0004). Nothing is built in the tests: the builder is a stub that copies a +fixture rootfs into place, so every capture the suite makes is a real capture +of a real tree, including a path with a space in it, a symlink whose target +has a space, a text state file, a binary state file and a state file above the +sample cap. diff --git a/share/layer-state-paths b/share/layer-state-paths new file mode 100644 index 0000000..bd8bb04 --- /dev/null +++ b/share/layer-state-paths @@ -0,0 +1,46 @@ +# Paths that can hold state that matters: accounts, credentials, keys and +# database content. bt-layer-measure never subtracts a difference at one of +# these because it fell inside the noise floor. It keeps their bytes at +# capture time and compares them in both pairs, so a clock that moves in +# every build is reported as noise while an account that is gone in one +# build is reported as real. +# +# The rule this list exists for: for the mariadb component the noise floor +# was 179 files of which 173 were ./var/lib/mysql/**, and that directory +# holds mysql/global_priv and mysql/user.*, so the measurement of a +# component whose build-time job includes deleting accounts was blind +# exactly where the accounts live. +# +# One bash glob per line; * crosses a slash, so ** and * are the same thing +# here and ** is written where a whole subtree is meant. Adding a path is +# cheap: it costs a copy of the file in each capture and a byte comparison. + +# accounts and credentials +./etc/shadow +./etc/gshadow +./etc/passwd +./etc/group +./etc/subuid +./etc/subgid +./etc/sudoers +./etc/sudoers.d/** +./etc/security/opasswd + +# host and user keys +./etc/ssh/ssh_host_* +./root/.ssh/** +./home/**/.ssh/** +./etc/ssl/private/** + +# database state +./var/lib/mysql/** +./var/lib/postgresql/** +./var/lib/redis/** +./var/lib/couchdb/** +./var/lib/mongodb/** + +# application and console accounts +./etc/webmin/miniserv.users +./etc/webmin/miniserv.conf +./var/lib/nodebb/** +./etc/keel/instance.d/** diff --git a/tests/coverage.sh b/tests/coverage.sh index ac85515..0bb49d0 100755 --- a/tests/coverage.sh +++ b/tests/coverage.sh @@ -3,6 +3,7 @@ # bin/layer-lib and bt-layer under tests/layer # bin/aplinfo-lib and bt-aplinfo under tests/aplinfo # bin/signature-lib and bin/generate-signature under tests/signature +# bin/layer-measure-lib and bt-layer-measure under tests/measure # Exits 1 when a measured file is below the threshold (default 95), 2 when a # tool is missing. # @@ -30,9 +31,9 @@ for tool in kcov zstd git gpg; do done report="${COVERAGE_DIR:-$(mktemp -d)}" -measured=/bin/layer-lib,/bt-layer,/bin/aplinfo-lib,/bt-aplinfo,/bin/signature-lib,/bin/generate-signature +measured=/bin/layer-lib,/bt-layer,/bin/aplinfo-lib,/bt-aplinfo,/bin/signature-lib,/bin/generate-signature,/bin/layer-measure-lib,/bt-layer-measure skipped=/tests/,/scratch-bt/bin/generate-signature -for suite in layer aplinfo signature; do +for suite in layer aplinfo signature measure; do kcov --include-pattern="$measured" --exclude-pattern="$skipped" \ "$report" "$here/$suite" done diff --git a/tests/measure b/tests/measure new file mode 100755 index 0000000..af840da --- /dev/null +++ b/tests/measure @@ -0,0 +1,507 @@ +#!/bin/bash -e +# Test bt-layer-measure and bin/layer-measure-lib against scratch directories. +# Nothing is built: the builder is a stub that copies a fixture rootfs into +# place, so every capture below is a real capture of a real tree. +# +# Code paths covered: +# layer-measure-lib: measure_path_safe (plain, space, tab, newline); +# measure_tree_record (sorted, tab keyed, size and mode, a path with a +# space, a rejected tab, a missing root); measure_links_record (target +# with a space, none); measure_pkgs_record (installed only, dpkg-query +# missing, admindir missing); measure_state_patterns (globs, comments, +# blanks, missing file, empty file); measure_state_match (exact, star, +# double star, no match, no patterns); measure_state_sample (sampled, +# above the cap, nested path, a path with a space, nothing matching); +# measure_capture_check (complete, incomplete, missing); measure_meta_get +# (present, absent); measure_only_in (some, none); measure_changed (some, +# none); measure_value (present, absent); measure_is_text (text, NUL, +# missing); measure_signature (text same, text changed line, text added +# line, binary same, binary offsets, binary length, missing side); +# measure_set_minus (empty, residue); measure_sample_path; +# measure_state_verdict (text noise, text real by content, text real by +# an extra changed line, binary noise, binary real offset, binary real +# length, not sampled on one side, absent on one side); +# measure_state_evidence (text, binary, unsampled); measure_count. +# bt-layer-measure: --help; no subcommand; unknown subcommand; capture +# without --dir, without --layer, without --tag, with a bad tag, with a +# failing build, with no rootfs, ok, and refusing to overwrite a tag; +# compare with a missing capture, ok and identical, ok and differing; +# attribute with a missing argument, with a missing capture, a clean unit +# (PASS), a dirty unit (FAIL, attributable file, attributable addition, +# package and symlink differences, a real state difference inside the +# noise floor), and a run below the sample cap (FAIL, not sampled). + +# pipefail as bt-layer-measure sets it, so a library function that inherits +# a pipeline's status is caught here and not in a report nobody reads twice. +set -o pipefail + +fatal() { echo "FATAL [$(basename "$0")]: $*" 1>&2; exit 1; } +info() { echo "INFO [$(basename "$0")]: $*"; } + +usage() { +cat</dev/null 2>&1 || fatal "$label: should have passed" +} + +assert_fail() { + local label=$1 + shift + if "$@" >/dev/null 2>&1; then + fatal "$label: should have failed" + fi +} + +assert_status() { + local label=$1 want=$2 + shift 2 + local got=0 + "$@" >/dev/null 2>&1 || got=$? + [[ "$got" == "$want" ]] || fatal "$label: exit $got, want $want" +} + +assert_grep() { + local label=$1 pattern=$2 haystack=$3 + grep -q -- "$pattern" <<<"$haystack" || fatal "$label: no '$pattern' in: $haystack" +} + +assert_no_grep() { + local label=$1 pattern=$2 haystack=$3 + grep -q -- "$pattern" <<<"$haystack" && fatal "$label: unwanted '$pattern'" + return 0 +} + +# --------------------------------------------------------------------------- +# fixtures +# --------------------------------------------------------------------------- + +# A rootfs small enough to read and wide enough to exercise every rule: a +# path with a space in it (the one the traps note names), a symlink whose +# target has a space, a text state file, a binary state file, a large state +# file, non-state noise and a dpkg database. +# +# make_rootfs DIR SHADOW_STAMP SHADOW_ACCOUNTS PRIV_STAMP ACCOUNT LOG EXTRA +make_rootfs() { + local dir=$1 stamp=$2 accounts=$3 priv=$4 account=$5 log=$6 extra=$7 + rm -rf "$dir" + mkdir -p "$dir/bin" "$dir/etc/keel" "$dir/var/log" "$dir/var/lib/dpkg" \ + "$dir/var/lib/mysql/mysql" "$dir/usr/lib/python3/setuptools/text" + printf '#!/bin/sh\n' > "$dir/bin/bash" + chmod 755 "$dir/bin/bash" + echo "keel" > "$dir/etc/hostname" + echo "component=1" > "$dir/etc/keel/component.conf" + # the path the traps note names: one internal space, nothing else + echo "lorem" > "$dir/usr/lib/python3/setuptools/text/Lorem ipsum.txt" + ln -s "text/Lorem ipsum.txt" "$dir/usr/lib/python3/setuptools/lorem" + echo "$log" > "$dir/var/log/alternatives.log" + # a text state file: line 1 is a clock, the rest are accounts + { echo "# written $stamp" + echo "root:!:20000:0:99999:7:::" + local row + for row in $accounts; do echo "$row:!!:20000:0:99999:7:::"; done + } > "$dir/etc/shadow" + chmod 640 "$dir/etc/shadow" + # a binary state file: a fixed header, a clock at byte 9, an account + # record of a fixed width from byte 12, which is the shape that makes a + # deleted or edited account look exactly like a moved clock to anything + # that compares whole-file hashes + { printf 'PRIVHDR\x00' + printf '%s' "$priv" + printf '\x00\x00' + printf '%s' "$account" + } > "$dir/var/lib/mysql/mysql/global_priv.MAD" + # a state file large enough to run past a small sample cap + head -c 512 /dev/zero | tr '\0' "$priv" > "$dir/var/lib/mysql/ibdata1" + cat > "$dir/var/lib/dpkg/status" < "$dir/$extra" + fi +} + +fix=$tmp/fix +mkdir -p "$fix" +# control, and control again: the clocks move, nothing else does +make_rootfs "$fix/c1" 1790475089 "mysql daemon" 8 "root@localhost" "first" "" +make_rootfs "$fix/c2" 1790475323 "mysql daemon" 9 "root@localhost" "second" "" +# a clean unit: the same clocks move and nothing else does +make_rootfs "$fix/u1" 1790475511 "mysql daemon" 7 "root@localhost" "third" "" +# a dirty unit: the mysql account is gone from the text state file, the +# account record in the binary state file is edited at a byte the control +# pair never touches, a non-state file differs, a symlink points elsewhere, +# a package version moves and a path exists that neither control has. Every +# one of those is inside a path a hash-only comparison would have called +# noise. +make_rootfs "$fix/u2" 1790475777 "daemon" 6 "root@localhosX" "fourth" "tmp/new-file" +echo "component=2" > "$fix/u2/etc/keel/component.conf" +rm "$fix/u2/usr/lib/python3/setuptools/lorem" +ln -s "text/other file.txt" "$fix/u2/usr/lib/python3/setuptools/lorem" +sed -i 's/^Version: 9.4-3/Version: 9.5-1/' "$fix/u2/var/lib/dpkg/status" + +# the builder stub: it copies the fixture named in $layers/NEXT into place +layers=$tmp/layers +mkdir -p "$layers" +builder=$tmp/bin/stub-builder +mkdir -p "$tmp/bin" +cat > "$builder" < "$layers/\$layer.manifest" ;; +esac +echo "stub built \$layer from \$next" +EOF +chmod 755 "$builder" + +patterns=$tmp/state-paths +cat > "$patterns" <<'EOF' +# accounts, credentials and database state +./etc/shadow +./var/lib/mysql/** +EOF + +M=$BT/bt-layer-measure +caps=$tmp/captures +export MEASURE_BT_LAYER=$builder +export MEASURE_LAYERS_DIR=$layers +export MEASURE_LOCK=$tmp/build.lock +export MEASURE_STATE_PATHS=$patterns + +capture() { # capture FIXTURE TAG [extra args...] + local fixture=$1 tag=$2 + shift 2 + echo "$fixture" > "$layers/NEXT" + "$M" capture --dir "$caps" --layer mariadb --parent core --tag "$tag" "$@" +} + +# --------------------------------------------------------------------------- +# the library +# --------------------------------------------------------------------------- + +info "1: measure_path_safe" +assert_ok safe-plain measure_path_safe "./etc/shadow" +assert_ok safe-space measure_path_safe "./a/Lorem ipsum.txt" +assert_fail safe-tab measure_path_safe "$(printf './a\tb')" +assert_fail safe-newline measure_path_safe "$(printf './a\nb')" +assert_fail safe-backslash measure_path_safe './a\\b' + +info "2: measure_tree_record" +tree=$(measure_tree_record "$fix/c1") +assert_eq tree-sorted "$tree" "$(LC_ALL=C sort <<<"$tree")" +assert_grep tree-space "^\./usr/lib/python3/setuptools/text/Lorem ipsum\.txt " "$tree" +assert_eq tree-fields "$(awk -F'\t' 'NF != 4' <<<"$tree" | wc -l)" 0 +assert_eq tree-mode "$(awk -F'\t' '$1 == "./etc/shadow" { print $4 }' <<<"$tree")" 640 +assert_eq tree-size "$(awk -F'\t' '$1 == "./etc/hostname" { print $3 }' <<<"$tree")" 5 +assert_eq tree-hash \ + "$(awk -F'\t' '$1 == "./etc/hostname" { print $2 }' <<<"$tree")" \ + "$(sha256sum < "$fix/c1/etc/hostname" | cut -d' ' -f1)" +assert_eq tree-nolinks "$(grep -c 'setuptools/lorem$' <<<"$tree" || true)" 0 +assert_fail tree-missing measure_tree_record "$tmp/absent" +mkdir -p "$tmp/tabroot" +: > "$tmp/tabroot/$(printf 'a\tb')" +assert_fail tree-tab measure_tree_record "$tmp/tabroot" +mkdir -p "$tmp/slashroot" +: > "$tmp/slashroot/a\\b" +assert_fail tree-backslash measure_tree_record "$tmp/slashroot" +mkdir -p "$tmp/nlroot" +: > "$tmp/nlroot/$(printf 'a\nb')" +assert_fail tree-newline measure_tree_record "$tmp/nlroot" + +info "3: measure_links_record" +links=$(measure_links_record "$fix/c1") +assert_eq links-one "$(wc -l <<<"$links")" 1 +assert_eq links-target "$(cut -f2 <<<"$links")" "text/Lorem ipsum.txt" +assert_eq links-path "$(cut -f1 <<<"$links")" "./usr/lib/python3/setuptools/lorem" +mkdir -p "$tmp/nolinks" +assert_eq links-none "$(measure_links_record "$tmp/nolinks")" "" +assert_fail links-missing measure_links_record "$tmp/absent" +mkdir -p "$tmp/tablink" +ln -s "$(printf 'a\tb')" "$tmp/tablink/l" +assert_fail links-tab-target measure_links_record "$tmp/tablink" + +info "4: measure_pkgs_record" +pkgs=$(measure_pkgs_record "$fix/c1") +assert_eq pkgs-count "$(wc -l <<<"$pkgs")" 2 +assert_eq pkgs-line "$(head -1 <<<"$pkgs")" "$(printf 'coreutils\t9.4-3')" +assert_no_grep pkgs-configfiles "removed-leftover" "$pkgs" +assert_fail pkgs-noadmindir measure_pkgs_record "$tmp/absent" +assert_fail pkgs-notool env MEASURE_DPKG_QUERY=no-such-dpkg-query bash -c \ + "source '$BT/bin/layer-measure-lib'; measure_pkgs_record '$fix/c1'" + +info "5: measure_state_patterns" +assert_eq patterns-two "$(measure_state_patterns "$patterns" | wc -l)" 2 +assert_eq patterns-first "$(measure_state_patterns "$patterns" | head -1)" "./etc/shadow" +assert_fail patterns-missing measure_state_patterns "$tmp/absent" +: > "$tmp/empty-patterns" +assert_fail patterns-empty measure_state_patterns "$tmp/empty-patterns" + +info "6: measure_state_match" +globs=$(measure_state_patterns "$patterns") +assert_ok match-exact measure_state_match "./etc/shadow" "$globs" +assert_ok match-deep measure_state_match "./var/lib/mysql/mysql/user.MAD" "$globs" +assert_ok match-shallow measure_state_match "./var/lib/mysql/ibdata1" "$globs" +assert_fail match-none measure_state_match "./etc/hostname" "$globs" +assert_fail match-noglobs measure_state_match "./etc/shadow" "" +assert_ok match-star measure_state_match "./etc/ssh/ssh_host_rsa_key" \ + "./etc/ssh/ssh_host_*" + +info "7: measure_state_sample" +s=$tmp/sample +mkdir -p "$s" +measure_tree_record "$fix/c1" > "$s/tree.tsv" +measure_state_sample "$fix/c1" "$s/tree.tsv" "$s" 100000 "$globs" +assert_eq sample-count "$(wc -l < "$s/state.tsv")" 3 +assert_eq sample-shadow "$(cat "$s/state/etc/shadow")" "$(cat "$fix/c1/etc/shadow")" +assert_grep sample-flag " sampled " "$(cat "$s/state.tsv")" +s2=$tmp/sample-cap +mkdir -p "$s2" +measure_state_sample "$fix/c1" "$s/tree.tsv" "$s2" 64 "$globs" +assert_grep sample-toolarge "^\./var/lib/mysql/ibdata1 too-large 512$" "$(cat "$s2/state.tsv")" +[[ ! -e "$s2/state/var/lib/mysql/ibdata1" ]] || fatal "sample-cap: copied anyway" +s3=$tmp/sample-none +mkdir -p "$s3" +measure_state_sample "$fix/c1" "$s/tree.tsv" "$s3" 100000 "./nothing/**" +assert_eq sample-nomatch "$(wc -l < "$s3/state.tsv")" 0 + +info "8: measure_only_in, measure_changed and measure_value" +a=$tmp/a.tsv; b=$tmp/b.tsv +printf 'p1\tx\np2\ty\np3\tz\n' > "$a" +printf 'p2\ty\np3\tQ\np4\tw\n' > "$b" +assert_eq only-a "$(measure_only_in "$a" "$b")" "p1" +assert_eq only-b "$(measure_only_in "$b" "$a")" "p4" +assert_eq only-none "$(measure_only_in "$a" "$a")" "" +assert_eq changed "$(measure_changed "$a" "$b")" "p3" +assert_eq changed-none "$(measure_changed "$a" "$a")" "" +assert_eq value "$(measure_value "$a" p2)" "y" +assert_eq value-absent "$(measure_value "$a" p9)" "" +assert_eq count-lines "$(measure_count "$a")" 3 +assert_eq count-missing "$(measure_count "$tmp/absent")" 0 + +info "9: measure_is_text" +assert_ok text-yes measure_is_text "$fix/c1/etc/shadow" +assert_fail text-no measure_is_text "$fix/c1/var/lib/mysql/mysql/global_priv.MAD" +assert_fail text-missing measure_is_text "$tmp/absent" + +info "10: measure_signature and measure_set_minus" +printf 'one\ntwo\nthree\n' > "$tmp/t1" +printf 'one\nTWO\nthree\n' > "$tmp/t2" +printf 'one\ntwo\nthree\nfour\n' > "$tmp/t3" +assert_eq sig-text-same "$(measure_signature "$tmp/t1" "$tmp/t1")" "" +assert_eq sig-text-changed "$(measure_signature "$tmp/t1" "$tmp/t2")" "$(printf 'n2\no2')" +assert_eq sig-text-added "$(measure_signature "$tmp/t1" "$tmp/t3")" "n4" +printf 'AAAA' > "$tmp/b1" +printf 'AABA' > "$tmp/b2" +printf 'AAAAA' > "$tmp/b3" +printf '\x00\x00\x00\x00' > "$tmp/b0" +assert_eq sig-bin-same "$(measure_signature "$tmp/b0" "$tmp/b0")" "" +assert_eq sig-bin-offset "$(measure_signature "$tmp/b0" "$(printf %s "$tmp/b0"; :)")" "" +cp "$tmp/b0" "$tmp/b0x"; printf '\x01' | dd of="$tmp/b0x" bs=1 seek=2 conv=notrunc status=none +assert_eq sig-bin-at "$(measure_signature "$tmp/b0" "$tmp/b0x")" "@3" +head -c 5 /dev/zero > "$tmp/b0y" +assert_eq sig-bin-len "$(measure_signature "$tmp/b0" "$tmp/b0y")" "len" +assert_fail sig-missing measure_signature "$tmp/b0" "$tmp/absent" +assert_eq minus-empty "$(measure_set_minus "$(printf 'a\nb')" "$(printf 'a\nb\nc')")" "" +assert_eq minus-residue "$(measure_set_minus "$(printf 'a\nb')" "$(printf 'a')")" "b" +assert_eq minus-nothing "$(measure_set_minus "" "$(printf 'a')")" "" + +info "10b: measure_nature_list" +assert_eq nature-empty "$(measure_nature_list "")" "" +assert_eq nature-all "$(measure_nature_list "$(printf 'len\no3\nn4\n@7')")" \ + "the length, other line 4, control line 3, byte 7" +assert_eq nature-unknown "$(measure_nature_list "wat")" "wat" +assert_grep nature-overflow "and 4 more" \ + "$(measure_nature_list "$(seq 1 12 | sed 's/^/@/')")" + +info "10b2: measure_print_set caps a long list" +saved=$MEASURE_PRINT_MAX +MEASURE_PRINT_MAX=2 +assert_grep print-set-capped "and 2 more" "$(measure_print_set paths "$(printf 'a\nb\nc\nd')")" +assert_eq print-set-empty "$(measure_print_set paths "")" "" +MEASURE_PRINT_MAX=$saved + +info "10c: measure_lines never splits a path on a space" +assert_eq lines-space "$(measure_lines "> " "./a b/c d")" "> ./a b/c d" +assert_eq lines-empty "$(measure_lines "> " "")" "" + +info "11: captures" +capture c1 c1 > "$tmp/cap-c1.log" +capture u1 u1 > /dev/null +capture c2 c2 > /dev/null +capture u2 u2 > /dev/null +assert_grep capture-summary "2 packages" "$(cat "$tmp/cap-c1.log")" +assert_ok capture-check measure_capture_check "$caps/c1" +assert_fail capture-check-missing measure_capture_check "$caps/absent" +mkdir -p "$tmp/half"; : > "$tmp/half/tree.tsv" +assert_fail capture-check-half measure_capture_check "$tmp/half" +assert_eq meta-layer "$(measure_meta_get "$caps/c1" layer)" mariadb +assert_eq meta-parent "$(measure_meta_get "$caps/c1" parent)" core +assert_eq meta-absent "$(measure_meta_get "$caps/c1" nothing)" "" +assert_eq sample-path "$(measure_sample_path "$caps/c1" "./etc/shadow")" \ + "$caps/c1/state/etc/shadow" + +info "12: measure_state_verdict, text" +v=$(measure_state_verdict "$caps/c1" "$caps/u1" "$caps/c2" "./etc/shadow") +assert_eq verdict-text-noise "$(cut -f1 <<<"$v")" noise +assert_grep verdict-text-noise-why "control pair" "$v" +v=$(measure_state_verdict "$caps/c1" "$caps/u2" "$caps/c2" "./etc/shadow") +assert_eq verdict-text-real "$(cut -f1 <<<"$v")" real +assert_grep verdict-text-real-why "line" "$v" + +info "13: measure_state_verdict, binary" +v=$(measure_state_verdict "$caps/c1" "$caps/u1" "$caps/c2" \ + "./var/lib/mysql/mysql/global_priv.MAD") +assert_eq verdict-bin-noise "$(cut -f1 <<<"$v")" noise +v=$(measure_state_verdict "$caps/c1" "$caps/u2" "$caps/c2" \ + "./var/lib/mysql/mysql/global_priv.MAD") +assert_eq verdict-bin-real "$(cut -f1 <<<"$v")" real +assert_grep verdict-bin-real-why "byte" "$v" + +info "14: measure_state_verdict, not sampled and absent" +capture c1 s1 --state-max-bytes 8 > /dev/null +capture u1 s2 --state-max-bytes 8 > /dev/null +capture c2 s3 --state-max-bytes 8 > /dev/null +v=$(measure_state_verdict "$caps/s1" "$caps/s2" "$caps/s3" "./var/lib/mysql/ibdata1") +assert_eq verdict-unsampled "$(cut -f1 <<<"$v")" unsampled +assert_grep verdict-unsampled-why "not sampled" "$v" +v=$(measure_state_verdict "$caps/c1" "$caps/u2" "$caps/c2" "./var/lib/mysql/absent") +assert_eq verdict-absent "$(cut -f1 <<<"$v")" unsampled + +info "15: measure_state_evidence" +e=$(measure_state_evidence "$caps/c1" "$caps/u2" "$caps/c2" "./etc/shadow") +assert_grep evidence-text-unit "control vs unit" "$e" +assert_grep evidence-text-noise "control vs control" "$e" +assert_grep evidence-text-row "mysql" "$e" +e=$(measure_state_evidence "$caps/c1" "$caps/u2" "$caps/c2" \ + "./var/lib/mysql/mysql/global_priv.MAD") +assert_grep evidence-bin "offset" "$e" +e=$(measure_state_evidence "$caps/s1" "$caps/s2" "$caps/s3" "./var/lib/mysql/ibdata1") +assert_grep evidence-unsampled "not sampled" "$e" + +# --------------------------------------------------------------------------- +# the executable +# --------------------------------------------------------------------------- + +info "16: usage" +assert_status help 0 "$M" --help +assert_status help-short 0 "$M" -h +assert_status no-subcommand 2 "$M" +assert_status unknown-subcommand 2 "$M" frobnicate +assert_status unknown-option 2 "$M" capture --nope +assert_status value-missing 2 "$M" capture --dir +assert_status two-subcommands 2 "$M" capture compare +assert_grep help-text "attribute" "$("$M" --help)" + +info "17: capture errors" +assert_status capture-nodir 2 "$M" capture --layer mariadb --tag t +assert_status capture-nolayer 2 "$M" capture --dir "$caps" --tag t +assert_status capture-notag 2 "$M" capture --dir "$caps" --layer mariadb +assert_status capture-unknown-opt 2 "$M" capture --dir "$caps" --layer m --tag t --wat +assert_status capture-badtag 2 "$M" capture --dir "$caps" --layer m --tag "../escape" +assert_status capture-badcap 2 "$M" capture --dir "$caps" --layer m --tag t \ + --state-max-bytes plenty +assert_status capture-nopatterns 3 env MEASURE_STATE_PATHS=$tmp/absent \ + "$M" capture --dir "$caps" --layer m --tag t +assert_status attribute-nopatterns 3 env MEASURE_STATE_PATHS=$tmp/absent \ + "$M" attribute --dir "$caps" --control c1 --unit u1 --control-again c2 +assert_status capture-exists 2 "$M" capture --dir "$caps" --layer m --tag c1 +echo FAIL > "$layers/NEXT" +assert_status capture-buildfail 3 "$M" capture --dir "$caps" --layer m --tag bad +echo NOROOTFS > "$layers/NEXT" +assert_status capture-norootfs 3 "$M" capture --dir "$caps" --layer m --tag bad2 +echo NOMANIFEST > "$layers/NEXT" +r=$("$M" capture --dir "$caps" --layer m --tag nomanifest) +assert_grep capture-nomanifest "no layer manifest" "$r" +[[ ! -e "$caps/nomanifest/manifest.txt" ]] || fatal "capture-nomanifest: copied one anyway" +[[ -f "$caps/c1/manifest.txt" ]] || fatal "capture: the layer manifest was not kept" + +info "18: compare" +assert_status compare-missing 3 "$M" compare --dir "$caps" --a c1 --b absent +assert_status compare-args 2 "$M" compare --dir "$caps" --a c1 +r=$("$M" compare --dir "$caps" --a c1 --b c1) || fatal "compare-same: exit $?" +assert_grep compare-identical "identical" "$r" +r=$("$M" compare --dir "$caps" --a c1 --b u2 || true) +assert_status compare-differs 1 "$M" compare --dir "$caps" --a c1 --b u2 +assert_grep compare-file "./etc/keel/component.conf" "$r" +assert_grep compare-pkg "coreutils" "$r" +assert_grep compare-link "lorem" "$r" +assert_grep compare-added "./tmp/new-file" "$r" + +info "19: attribute, a clean unit" +assert_status attr-args 2 "$M" attribute --dir "$caps" --control c1 --unit u1 +assert_status attr-missing 3 "$M" attribute --dir "$caps" --control c1 --unit u1 \ + --control-again absent +r=$("$M" attribute --dir "$caps" --control c1 --unit u1 --control-again c2) +assert_grep attr-pass "verdict: PASS" "$r" +assert_grep attr-zero "attributable differences" "$r" +assert_eq attr-zero-count \ + "$(awk -F: '/attributable differences/ { gsub(/ /, "", $2); print $2 }' <<<"$r")" 0 +assert_grep attr-command "bt-layer-measure attribute" "$r" +assert_grep attr-floor "noise floor" "$r" +# the state paths inside the floor are named and classified, not subtracted +assert_grep attr-state-named "./etc/shadow" "$r" +assert_grep attr-state-noise "noise" "$r" + +info "20: attribute, a dirty unit" +assert_status attr-dirty 1 "$M" attribute --dir "$caps" --control c1 --unit u2 \ + --control-again c2 +r=$("$M" attribute --dir "$caps" --control c1 --unit u2 --control-again c2 || true) +assert_grep attr-fail "verdict: FAIL" "$r" +assert_grep attr-attributable "./etc/keel/component.conf" "$r" +assert_grep attr-added "./tmp/new-file" "$r" +assert_grep attr-pkgdiff "coreutils" "$r" +assert_grep attr-linkdiff "lorem" "$r" +# the point of the whole exercise: a real difference inside the noise floor +assert_grep attr-real-state "real" "$r" +assert_grep attr-real-shadow "./etc/shadow" "$r" +assert_grep attr-real-row "mysql" "$r" + +info "21: attribute, blind because nothing was sampled" +assert_status attr-unsampled 1 "$M" attribute --dir "$caps" --control s1 --unit s2 \ + --control-again s3 +r=$("$M" attribute --dir "$caps" --control s1 --unit s2 --control-again s3 || true) +assert_grep attr-unsampled-why "not sampled" "$r" +assert_grep attr-unsampled-fail "verdict: FAIL" "$r" + +echo +info "all tests passed" From 6b1baec125fcbde8f57d3018935bd0fa4144d00f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Mon, 28 Sep 2026 05:49:54 +0000 Subject: [PATCH 2/9] fix: compare content, not positions, and stop the blindness being switchable Review of #11 found two ways this failed in the same direction as the defect it exists to correct. Both are closed here. The signature recorded where two files differ and threw the content away, so two unrelated changes at one line number or one byte offset annihilated, and the evidence block was suppressed for anything called noise. Four cases came back as noise, exit 0, with nothing printed: an account added to /etc/shadow at the line number the control pair also appends at; a root password set on the line the control pair rewrites for a clock reason; a binary differing at the same eight offsets with different byte values; a file 3584 bytes shorter against a control pair that varies by one. The mariadb captures on the build host show global_priv.MAD, .MAI, .frm and user.frm differing in both pairs today, so the next extraction would have got four noise lines and a count. The controls are now the model and the model carries content. At each position the controls vary at, the longest common prefix and suffix of their variants is the shape of that variation, and the unit is admitted only if its line has that shape. A mask with neither a prefix nor a suffix is what two unrelated lines have in common, which is no evidence about a third, so it never yields noise. Bytes carry no shape at all: the most that can be said of a binary is that it coincides in position, in size delta and in how much differs, which is the new verdict `overlapping`, and it fails until somebody reads the bytes and says so in --cleared. The evidence is printed for every verdict, noise included, because the one mistake this scheme can still make is the one nobody would otherwise see. The state path list was read from the environment when attribute ran, so MEASURE_STATE_PATHS pointing at a one line file turned FAIL (167 not sampled) into PASS on the real captures while the block still printed a digest. capture now copies the list into the capture, attribute reads that copy and never the variable, and it refuses unless every capture carries the same list and the digest each one recorded matches it. Also from the review: - --control-again is repeatable and the floor is the union over every control pair. One pair is not a floor: mariadb differs by 179 files in one pair and 182 in another, and the residue is 3 against the first and 0 against the second. The block prints what each single pair would have said. - attribute refuses unless the captures share layer, parent and SOURCE_DATE_EPOCH, were built with an epoch at all, and are distinct. Recording a precondition and never checking it is not recording it. - share/layer-state-paths gains ./etc/mysql/**, ./etc/postgresql/**, ./etc/redis/**, ./root/.my.cnf, ./root/.pgpass and ./etc/machine-id. The postgresql residue lived in the second of those and was caught only because it fell outside the floor. - The list is treated as a failure mode, not a list. Every subtracted path is named rather than counted, and any whose name looks like state (share/layer-state-suspect) fails the run, with two committed ways out: add it to the list, or clear it with a reason. - Text is no longer "contains no NUL": a file with no newline, or with a line over 8 KiB, is a blob and is compared as bytes. - The verdict totals must add up to the paths examined, and an empty verdict is fatal rather than uncounted. - A capture directory is created 0700, and the retention rule is in the doc. - tests/layer, tests/aplinfo and tests/signature set pipefail, which the scripts they test already did. - docs/layer-measure.md exports BT_BUILDS, and the layers directory is derived from it through config/common.cfg the way bt-layer derives it. Logic split under decision 0004: bin/layer-measure-lib keeps capture and the formats, bin/layer-compare-lib decides whether a difference is noise, bin/layer-report-lib prints the blocks and enforces the preconditions. tests/compare is the four false negatives above. Coverage: 204 of 204, 189 of 189, 230 of 230 and 70 of 70, all four at 100 percent, gate unchanged at 99. --- COVERAGE.md | 72 +++++-- bin/layer-compare-lib | 394 ++++++++++++++++++++++++++++++++++++ bin/layer-measure-lib | 333 ++---------------------------- bin/layer-report-lib | 413 ++++++++++++++++++++++++++++++++++++++ bt-layer-measure | 108 +++++++--- docs/layer-measure.md | 149 +++++++++++--- share/layer-state-paths | 16 ++ share/layer-state-suspect | 36 ++++ tests/aplinfo | 4 + tests/compare | 296 +++++++++++++++++++++++++++ tests/coverage.sh | 8 +- tests/layer | 4 + tests/measure | 298 +++++++++++++++------------ tests/signature | 7 +- 14 files changed, 1617 insertions(+), 521 deletions(-) create mode 100644 bin/layer-compare-lib create mode 100644 bin/layer-report-lib create mode 100644 share/layer-state-suspect create mode 100755 tests/compare diff --git a/COVERAGE.md b/COVERAGE.md index bbd6e8e..8e5a97e 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -112,12 +112,16 @@ The gate stays at 99. ## 2026-09-28: the three-build comparison -`bt-layer-measure` and `bin/layer-measure-lib` make handbook decision 0010 -step 3 executable: a capture subcommand that builds a layer and records its -package list, file tree, symlinks and state paths, a compare subcommand over -two captures and an attribute subcommand over the three the decision asks -for. `tests/coverage.sh` now runs `tests/measure` as a fourth suite and -measures eight files: +`bt-layer-measure` makes handbook decision 0010 step 3 executable: a capture +subcommand that builds a layer and records its package list, file tree, +symlinks and state paths, a compare subcommand over two captures and an +attribute subcommand over the three or more the decision asks for. The logic +is split three ways under decision 0004: `bin/layer-measure-lib` (capture and +formats), `bin/layer-compare-lib` (whether a difference is noise) and +`bin/layer-report-lib` (the blocks and the preconditions), with +`bt-layer-measure` a thin main over them. `tests/coverage.sh` runs +`tests/measure` and `tests/compare` as its fourth and fifth suites and +measures ten files: | File | Lines covered | Cover | |------|---------------|-------| @@ -127,23 +131,49 @@ measures eight files: | bt-aplinfo | 47 of 47 | 100 percent | | bin/signature-lib | 76 of 76 | 100 percent | | bin/generate-signature | 87 of 87 | 100 percent | -| bin/layer-measure-lib | 379 of 379 | 100 percent | -| bt-layer-measure | 64 of 64 | 100 percent | +| bin/layer-measure-lib | 204 of 204 | 100 percent | +| bin/layer-compare-lib | 189 of 189 | 100 percent | +| bin/layer-report-lib | 230 of 230 | 100 percent | +| bt-layer-measure | 70 of 70 | 100 percent | -The gate stays at 99, the lowest file rounded down. Both new files are at +The gate stays at 99, the lowest file rounded down. All four new files are at 100 percent line and branch: every subcommand, every exit code (0, 1, 2 and -3) and every error path is exercised, including the comparison itself, where -`tests/measure` covers a difference inside the noise floor that is real and -has to be reported (an account row gone from a text state file, and an edited -account record in a binary one, both at paths whose clocks move in every -build) as well as a run where nothing was sampled and the measurement fails -rather than reporting a clean number it could not see. - -`tests/measure` sets `pipefail` the way `bt-layer-measure` does. Without it -the suite passed while the executable did not: `measure_signature` inherited -`cmp`'s "the files differ" status from a pipeline and reported a failure -instead of a signature, which silently dropped one state path's verdict from -every report. +3), every verdict and every error path, including the two guards that must +never fire, which `tests/measure` reaches by replacing `measure_verdict` with +a stub. + +## 2026-09-28: what the first version of the comparison got wrong + +Worth recording, because the defect was in the same direction as the one the +tool exists to correct and the suite did not catch it. + +The first implementation compared *positions*: the line numbers or byte +offsets at which two files differ, subtracting the control pair's set from +the control-unit set. Two unrelated changes at one position annihilate. Four +cases came back as `noise`, exit 0, with the evidence suppressed because they +were noise: an account added to `/etc/shadow` at the line number the control +pair also appends at; a root password set on the line the control pair +rewrites for a clock reason; a binary differing at the same eight offsets +with different byte values; a file 3584 bytes shorter against a control pair +that varies by one. + +`tests/compare` is that list. The controls are now the model and the model +carries content: at each position the controls vary at, the longest common +prefix and suffix of their variants is the shape the unit has to fit. A mask +with nothing in it proves nothing, and bytes carry no shape at all, so a +binary coincidence is `overlapping` and fails until it is cleared in writing. +The evidence is printed for every verdict, `noise` included, because the one +mistake the scheme can still make is the one nobody would otherwise see. + +## 2026-09-28: pipefail in the three older suites + +`tests/layer`, `tests/aplinfo` and `tests/signature` did not set `pipefail` +while `bt-layer`, `bt-aplinfo` and `bin/generate-signature` all do, so a +library function that inherits a pipeline's status fails in production and +passes in the suite. That is exactly how `measure_signature` shipped broken +once. All five suites now set it. One assertion had to change with it: +`tests/signature` piped `generate-signature --help` into `grep`, and usage +exits non-zero by convention, so the output is captured first. ## Not measured diff --git a/bin/layer-compare-lib b/bin/layer-compare-lib new file mode 100644 index 0000000..bb37b5d --- /dev/null +++ b/bin/layer-compare-lib @@ -0,0 +1,394 @@ +#!/bin/bash +# Copyright (c) 2026 KeelLinux +# +# This file is part of buildtasks. +# +# Buildtasks is free software; you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published by the +# Free Software Foundation; either version 3 of the License, or (at your +# option) any later version. +# +# Deciding whether a difference at a state path is noise or the change being +# measured. Sourced by bt-layer-measure after bin/layer-measure-lib. +# +# Why this is not a set of differing positions minus another set of differing +# positions. That was the first implementation and it cancelled a real +# difference against a coincidental one at the same place: an account added +# to /etc/shadow at the line number the control pair also appends at, a root +# password set on the line the control pair rewrites for a clock reason, a +# binary differing at the same offsets with different byte values, a file +# 3584 bytes shorter where the control pair varies by one. All four came back +# as noise, with the evidence suppressed because they were noise. +# +# So the control captures are the model, and the model has content in it. At +# each position the controls vary at, the longest common prefix and suffix of +# their variants is the shape of that variation; the unit is admitted only if +# its line has that shape. A mask with neither a prefix nor a suffix explains +# nothing, so it never yields noise. Bytes carry no shape at all, so a binary +# coincidence is never noise either: it is `overlapping`, which fails until +# somebody clears it in writing. +# +# Three or more control captures make the model better, because the floor is +# the union over every control pair and the magnitudes are a range rather +# than a single value. + +# A line longer than this, or a file with no newline at all, is not text: a +# newline-free blob is one line, and one line masks against anything. +MEASURE_TEXT_MAX_LINE=8192 + +# Above this many differing positions the masking is abandoned and the path +# is reported as overlapping rather than guessed at. +MEASURE_POSITION_MAX=200 + +# The field separator inside a mask, which cannot occur in a text line. +MEASURE_MASK_SEP=$'\001' + +# How many lines of a diff the evidence block prints per pair, and how many +# paths get a full three-way diff before the rest are abbreviated to their +# summary line. Every `real` path is always printed in full. Both are read +# by bin/layer-report-lib. +# shellcheck disable=SC2034 +MEASURE_EVIDENCE_LINES=20 +# shellcheck disable=SC2034 +MEASURE_EVIDENCE_PATHS=20 + +# --------------------------------------------------------------------------- +# primitives +# --------------------------------------------------------------------------- + +# measure_is_text FILE +# No NUL, at least one newline, and no line long enough to be a blob. +measure_is_text() { + local f=$1 + if [[ ! -f "$f" ]]; then + measure_error "no such file: $f" + return 1 + fi + local all stripped + all=$(wc -c < "$f") + stripped=$(tr -d '\0' < "$f" | wc -c) + [[ "$all" == "$stripped" ]] || return 1 + [[ "$(tr -dc '\n' < "$f" | wc -c)" -gt 0 ]] || return 1 + [[ "$(wc -L < "$f")" -le "$MEASURE_TEXT_MAX_LINE" ]] || return 1 + return 0 +} + +# measure_line N FILE +measure_line() { + sed -n "$1{p;q}" "$2" +} + +# measure_common_prefix STRING... +measure_common_prefix() { + local out=$1 s i + shift + for s in "$@"; do + i=0 + while [[ $i -lt ${#out} ]] && [[ "${s:$i:1}" == "${out:$i:1}" ]]; do + i=$((i + 1)) + done + out=${out:0:$i} + [[ -n "$out" ]] || break + done + printf '%s\n' "$out" +} + +# measure_common_suffix STRING... +measure_common_suffix() { + local out=$1 s i + shift + for s in "$@"; do + i=0 + while [[ $i -lt ${#out} ]] && [[ "${s: -$((i + 1)):1}" == "${out: -$((i + 1)):1}" ]]; do + i=$((i + 1)) + done + out=${out: ${#out}-$i} + [[ -n "$out" ]] || break + done + printf '%s\n' "$out" +} + +# measure_text_positions A B -> o for a line of A that moved, n for a +# line of B that is not in A where it was +measure_text_positions() { + diff --old-line-format=$'o%dn\n' --new-line-format=$'n%dn\n' \ + --unchanged-line-format='' -- "$1" "$2" | LC_ALL=C sort -u + return 0 +} + +# measure_byte_offsets A B -> the 1-based offsets differing in the common +# prefix of the two files, one per line +measure_byte_offsets() { + if [[ ! -f "$1" ]] || [[ ! -f "$2" ]]; then + measure_error "cannot compare $1 with $2: one of them is not a file" + return 1 + fi + cmp -l -- "$1" "$2" 2>/dev/null | awk '{ print $1 }' || true + return 0 +} + +# measure_pair_positions KIND A B +measure_pair_positions() { + if [[ "$1" == text ]]; then + measure_text_positions "$2" "$3" + return 0 + fi + measure_byte_offsets "$2" "$3" +} + +# measure_floor_positions KIND CONTROL... -> the union over every unordered +# pair of control captures. Three controls give three pairs, which is the +# point of allowing more than two. +measure_floor_positions() { + local kind=$1 + shift + local -a c=("$@") + local i j all="" + for i in $(seq 0 $(( ${#c[@]} - 1 ))); do + for j in $(seq $(( i + 1 )) $(( ${#c[@]} - 1 ))); do + all+=$(measure_pair_positions "$kind" "${c[$i]}" "${c[$j]}")$'\n' + done + done + [[ -n "${all//[$'\n']/}" ]] || return 0 + LC_ALL=C sort -u <<<"$all" | grep . + return 0 +} + +# measure_range size|bytes CONTROL... -> "MIN MAX" over every ordered pair of +# distinct controls, of the size delta or of the number of differing bytes. +# Ordered, so the range is symmetric: which control is written first is an +# accident of the command line and must not decide whether the unit fits. +measure_range() { + local what=$1 + shift + local -a c=("$@") + local i j v min="" max="" + for i in $(seq 0 $(( ${#c[@]} - 1 ))); do + for j in $(seq 0 $(( ${#c[@]} - 1 ))); do + [[ $i -ne $j ]] || continue + if [[ "$what" == size ]]; then + v=$(( $(wc -c < "${c[$j]}") - $(wc -c < "${c[$i]}") )) + else + v=$(measure_byte_offsets "${c[$i]}" "${c[$j]}" | grep -c . || true) + fi + [[ -n "$min" ]] && [[ "$v" -ge "$min" ]] || min=$v + [[ -n "$max" ]] && [[ "$v" -le "$max" ]] || max=$v + done + done + printf '%s %s\n' "${min:-0}" "${max:-0}" +} + +# measure_mask_at N CONTROL... -> PREFIXSUFFIX, the shape the controls +# show at line N +measure_mask_at() { + local n=$1 + shift + local -a lines=() + local f + for f in "$@"; do + lines+=("$(measure_line "$n" "$f")") + done + printf '%s%s%s\n' \ + "$(measure_common_prefix "${lines[@]}")" \ + "$MEASURE_MASK_SEP" \ + "$(measure_common_suffix "${lines[@]}")" +} + +# measure_mask_admits LINE PREFIX SUFFIX +# A mask with neither a prefix nor a suffix admits nothing: it is what two +# unrelated lines have in common, which is no evidence about a third. +measure_mask_admits() { + local line=$1 prefix=$2 suffix=$3 + [[ -n "$prefix" ]] || [[ -n "$suffix" ]] || return 1 + [[ ${#line} -ge $(( ${#prefix} + ${#suffix} )) ]] || return 1 + [[ "$line" == "$prefix"* ]] || return 1 + [[ -z "$suffix" ]] || [[ "$line" == *"$suffix" ]] || return 1 + return 0 +} + +# measure_cleared_reason FILE PATH -> the reason a waiver gives, or nothing +# +# A waiver line is PATHREASON, and PATH may be a glob, because a real +# MariaDB data directory is 173 files that one sentence explains. A glob +# waiver is only honest because the report names every path it covered, so +# it is a record of what was waived and not a way of not looking. +measure_cleared_reason() { + local file=$1 path=$2 rule reason line + [[ -f "$file" ]] || return 0 + local -a rules=() + mapfile -t rules < "$file" + for line in "${rules[@]}"; do + IFS=$'\t' read -r rule reason <<<"$line" + [[ -n "$rule" ]] || continue + [[ "$rule" != \#* ]] || continue + # shellcheck disable=SC2053 + if [[ "$path" == "$rule" ]] || [[ "$path" == $rule ]]; then + printf '%s\n' "$reason" + return 0 + fi + done + return 0 +} + +# --------------------------------------------------------------------------- +# the verdict +# --------------------------------------------------------------------------- + +# measure_verdict UNIT CONTROL... -> verdictwhy +# +# none the unit is byte identical to the control +# noise every difference is at a position the controls vary at and +# has the shape the controls show there +# overlapping every difference coincides with the controls' variation but +# its content cannot be shown to be that variation; bytes +# never can, and neither can a mask with nothing in it +# real it differs where the controls do not vary, or with content +# or a magnitude they do not show +# unsampled some side has no bytes, so nothing can be said +measure_verdict() { + local unit=$1 + shift + if [[ $# -lt 1 ]]; then + measure_error "measure_verdict needs a unit sample and at least one control" + return 1 + fi + local f + for f in "$unit" "$@"; do + if [[ ! -f "$f" ]]; then + printf 'unsampled\t%s\n' \ + "no bytes kept for $f, so neither noise nor a real difference can be shown" + return 0 + fi + done + local base=$1 + if cmp -s -- "$base" "$unit"; then + printf 'none\t%s\n' "identical to the control" + return 0 + fi + local kind=binary + if measure_is_text "$base" && measure_is_text "$unit"; then + kind=text + fi + local floor unit_pos outside n + floor=$(measure_floor_positions "$kind" "$@") + unit_pos=$(measure_pair_positions "$kind" "$base" "$unit") + outside=$(measure_set_minus "$unit_pos" "$floor") + if [[ -n "$outside" ]]; then + printf 'real\t%s\n' \ + "differs where the controls do not vary: $(measure_nature_list "$outside")" + return 0 + fi + if [[ "$kind" == binary ]]; then + measure_verdict_binary "$unit" "$@" + return 0 + fi + n=$(grep -c . <<<"$unit_pos" || true) + if [[ "$n" -gt "$MEASURE_POSITION_MAX" ]]; then + printf 'overlapping\t%s\n' \ + "$n differing positions, too many to mask (the cap is $MEASURE_POSITION_MAX); read the evidence" + return 0 + fi + measure_verdict_text "$unit_pos" "$unit" "$@" +} + +# measure_verdict_text POSITIONS UNIT CONTROL... +measure_verdict_text() { + local positions=$1 unit=$2 + shift 2 + local token n mask prefix suffix line unexplained=0 masked=0 + while IFS= read -r token; do + [[ -n "$token" ]] || continue + n=${token#[on]} + mask=$(measure_mask_at "$n" "$@") + prefix=${mask%%"$MEASURE_MASK_SEP"*} + suffix=${mask#*"$MEASURE_MASK_SEP"} + line=$(measure_line "$n" "$unit") + if measure_mask_admits "$line" "$prefix" "$suffix"; then + masked=$((masked + 1)) + continue + fi + if [[ -z "$prefix" ]] && [[ -z "$suffix" ]]; then + printf 'overlapping\t%s\n' \ + "line $n varies in the controls with nothing in common, so nothing about it can be shown; read the evidence" + return 0 + fi + unexplained=$((unexplained + 1)) + printf 'real\t%s\n' \ + "line $n does not have the shape the controls vary with there (prefix '$prefix', suffix '$suffix')" + return 0 + done <<<"$positions" + printf 'noise\t%s\n' \ + "$masked differing lines, every one of them matching the mask the controls establish at that line" + return 0 +} + +# measure_verdict_binary UNIT CONTROL... +# Bytes carry no shape, so the best this can say is that the unit coincides +# with the controls in position, in size delta and in how much differs. +measure_verdict_binary() { + local unit=$1 base=$2 + shift + local delta count dmin dmax cmin cmax + read -r dmin dmax < <(measure_range size "$@") + read -r cmin cmax < <(measure_range bytes "$@") + delta=$(( $(wc -c < "$unit") - $(wc -c < "$base") )) + count=$(measure_byte_offsets "$base" "$unit" | grep -c . || true) + if [[ "$delta" -lt "$dmin" ]] || [[ "$delta" -gt "$dmax" ]]; then + printf 'real\t%s\n' \ + "size moves by $delta bytes where the controls move by $dmin to $dmax" + return 0 + fi + if [[ "$count" -lt "$cmin" ]] || [[ "$count" -gt "$cmax" ]]; then + printf 'real\t%s\n' \ + "$count bytes differ where the controls differ by $cmin to $cmax" + return 0 + fi + printf 'overlapping\t%s\n' \ + "$count bytes at offsets the controls also vary at, size delta $delta within $dmin to $dmax; bytes carry no shape, so this is a coincidence nobody has ruled out" + return 0 +} + +# --------------------------------------------------------------------------- +# evidence, printed for every verdict including noise +# --------------------------------------------------------------------------- + +# measure_evidence PATH UNIT CONTROL... +measure_evidence() { + local path=$1 unit=$2 + shift 2 + local base=$1 f + echo " $path" + for f in "$unit" "$@"; do + if [[ ! -f "$f" ]]; then + echo " not sampled: $f is absent, raise --state-max-bytes and capture again" + return 0 + fi + done + echo " control vs unit:" + measure_pair_evidence "$base" "$unit" + local i=2 + for f in "$@"; do + [[ "$f" != "$base" ]] || continue + echo " control vs control $i (the floor at this path):" + measure_pair_evidence "$base" "$f" + i=$((i + 1)) + done + return 0 +} + +# measure_pair_evidence A B +measure_pair_evidence() { + local a=$1 b=$2 + if measure_is_text "$a" && measure_is_text "$b"; then + diff -u --label control --label other -- "$a" "$b" \ + | tail -n +3 | head -n "$MEASURE_EVIDENCE_LINES" | sed 's/^/ /' || true + return 0 + fi + local all n offsets + all=$(measure_byte_offsets "$a" "$b") + n=$(grep -c . <<<"$all" || true) + offsets=$(head -n "$MEASURE_LIST_MAX" <<<"$all" | paste -sd, - || true) + printf ' sizes %s and %s; differing bytes: %s; first offsets: %s\n' \ + "$(wc -c < "$a")" "$(wc -c < "$b")" "$n" "${offsets:-none}" + return 0 +} diff --git a/bin/layer-measure-lib b/bin/layer-measure-lib index 03d5157..f694f2e 100644 --- a/bin/layer-measure-lib +++ b/bin/layer-measure-lib @@ -37,9 +37,6 @@ MEASURE_TSV_FIELDS_LINKS=2 # How many items a one-line summary names before it says "and N more". MEASURE_LIST_MAX=8 -# How many lines of a diff the evidence block prints per pair. -MEASURE_EVIDENCE_LINES=20 - # Default ceiling on a single state file's sampled copy. Above it the bytes # are not kept, the comparison has nothing to compare and says so; it never # silently treats an unread file as unchanged. 16 MiB covers a MariaDB @@ -47,7 +44,7 @@ MEASURE_EVIDENCE_LINES=20 MEASURE_STATE_MAX_BYTES_DEFAULT=16777216 # Files every capture directory carries. -MEASURE_CAPTURE_FILES="meta.tsv pkgs.tsv tree.tsv links.tsv state.tsv" +MEASURE_CAPTURE_FILES="meta.tsv pkgs.tsv tree.tsv links.tsv state.tsv state-paths.txt" measure_error() { echo "ERROR [layer-measure]: $*" 1>&2; } @@ -156,6 +153,8 @@ measure_pkgs_record() { measure_error "no dpkg database under $root" return 1 fi + # the braces are dpkg-query's own field syntax, not shell expansion + # shellcheck disable=SC2016 "$dpkg_query" --admindir="$root/var/lib/dpkg" -W \ -f '${Package}\t${Version}\t${Status}\n' 2>/dev/null \ | awk -F'\t' -v OFS='\t' '{ split($3, s, " "); if (s[3] == "installed") print $1, $2 }' \ @@ -280,55 +279,9 @@ measure_changed() { } # --------------------------------------------------------------------------- -# the nature of one difference +# set helpers over the difference tokens # --------------------------------------------------------------------------- -# measure_is_text FILE -measure_is_text() { - local f=$1 - if [[ ! -f "$f" ]]; then - measure_error "no such file: $f" - return 1 - fi - local all stripped - all=$(wc -c < "$f") - stripped=$(tr -d '\0' < "$f" | wc -c) - [[ "$all" == "$stripped" ]] -} - -# measure_signature A B -> the tokens by which A and B differ, C sorted: -# o line n of A is not in B where it was (text) -# n line n of B is not in A where it was (text) -# @ byte n differs (binary) -# len the two files are not the same length (binary) -# -# The tokens, and not the bytes themselves, are what the noise floor is -# subtracted on: a clock that moves in both pairs touches the same line or -# the same offset in both, so its tokens cancel, while a deleted account -# leaves a token the control pair never produces. -measure_signature() { - local a=$1 b=$2 - if [[ ! -f "$a" ]] || [[ ! -f "$b" ]]; then - measure_error "cannot compare $a with $b: one of them is not a file" - return 1 - fi - if measure_is_text "$a" && measure_is_text "$b"; then - diff --old-line-format=$'o%dn\n' --new-line-format=$'n%dn\n' --unchanged-line-format='' -- "$a" "$b" | LC_ALL=C sort -u - return 0 - fi - local sa sb tokens - sa=$(wc -c < "$a") - sb=$(wc -c < "$b") - # cmp exits 1 when the files differ, which is the normal case here, so - # its status is collected rather than inherited: under pipefail an - # inherited 1 made this function report a failure instead of a signature. - tokens=$(cmp -l -- "$a" "$b" 2>/dev/null | awk '{ print "@" $1 }' || true) - [[ "$sa" == "$sb" ]] || tokens="len"$'\n'"$tokens" - [[ -n "$tokens" ]] || return 0 - LC_ALL=C sort -u <<<"$tokens" | grep . - return 0 -} - # measure_set_minus A B -> the lines of A that are not lines of B measure_set_minus() { [[ -n "$1" ]] || return 0 @@ -360,80 +313,6 @@ measure_nature_list() { printf '%s\n' "${out[*]}" | sed 's/,/, /g' } -# measure_state_verdict CONTROL UNIT CONTROL_AGAIN PATH -> verdictwhy -# -# noise the unit build differs from the control only where the -# control differs from itself -# real it differs somewhere the control pair does not, so the change -# being measured is what differs, whatever the path is called -# unsampled no bytes were kept on some side, so neither can be said -measure_state_verdict() { - local c=$1 u=$2 c2=$3 path=$4 - local sc su sc2 side - sc=$(measure_sample_path "$c" "$path") - su=$(measure_sample_path "$u" "$path") - sc2=$(measure_sample_path "$c2" "$path") - for side in "$sc" "$su" "$sc2"; do - if [[ ! -f "$side" ]]; then - printf 'unsampled\t%s\n' \ - "not sampled in every build (no bytes kept at $side), so nothing can be said about it" - return 0 - fi - done - local ra rn residue - ra=$(measure_signature "$sc" "$su") || return 1 - rn=$(measure_signature "$sc" "$sc2") || return 1 - residue=$(measure_set_minus "$ra" "$rn") - if [[ -z "$residue" ]]; then - printf 'noise\t%s\n' \ - "every difference here is one the control pair shows against itself ($(grep -c . <<<"$ra" || true) in total)" - return 0 - fi - printf 'real\t%s\n' \ - "differences the control pair does not show against itself ($(grep -c . <<<"$residue" || true) of them): $(measure_nature_list "$residue")" -} - -# measure_state_evidence CONTROL UNIT CONTROL_AGAIN PATH -# What actually differs, both ways round, so the verdict above can be -# checked rather than believed. -measure_state_evidence() { - local c=$1 u=$2 c2=$3 path=$4 - local sc su sc2 - sc=$(measure_sample_path "$c" "$path") - su=$(measure_sample_path "$u" "$path") - sc2=$(measure_sample_path "$c2" "$path") - echo " $path" - local side - for side in "$sc" "$su" "$sc2"; do - if [[ ! -f "$side" ]]; then - echo " not sampled: $side is absent, raise --state-max-bytes and capture again" - return 0 - fi - done - echo " control vs unit:" - measure_pair_evidence "$sc" "$su" - echo " control vs control again (the noise floor at this path):" - measure_pair_evidence "$sc" "$sc2" - return 0 -} - -# measure_pair_evidence A B -measure_pair_evidence() { - local a=$1 b=$2 - if measure_is_text "$a" && measure_is_text "$b"; then - diff -u --label control --label other -- "$a" "$b" \ - | tail -n +3 | head -n "$MEASURE_EVIDENCE_LINES" | sed 's/^/ /' - return 0 - fi - local all offsets n - all=$(cmp -l -- "$a" "$b" 2>/dev/null | awk '{ print $1 }' || true) - n=$(grep -c . <<<"$all" || true) - offsets=$(head -n "$MEASURE_LIST_MAX" <<<"$all" | paste -sd, - || true) - printf ' sizes %s and %s; differing bytes: %s; first offsets: %s\n' \ - "$(wc -c < "$a")" "$(wc -c < "$b")" "$n" "${offsets:-none}" - return 0 -} - # --------------------------------------------------------------------------- # arguments # --------------------------------------------------------------------------- @@ -455,7 +334,8 @@ measure_parse_args() { MEASURE_B= MEASURE_CONTROL= MEASURE_UNIT= - MEASURE_CONTROL2= + MEASURE_CONTROL2=() + MEASURE_CLEARED= MEASURE_HELP= MEASURE_CAP=$MEASURE_STATE_MAX_BYTES_DEFAULT local opt val @@ -467,7 +347,7 @@ measure_parse_args() { shift continue ;; - --dir|--layer|--parent|--tag|--a|--b|--control|--unit|--control-again|--state-max-bytes) + --dir|--layer|--parent|--tag|--a|--b|--control|--unit|--control-again|--state-max-bytes|--cleared) if [[ $# -lt 2 ]]; then measure_error "$opt needs a value" return 1 @@ -498,7 +378,8 @@ measure_parse_args() { --b) MEASURE_B=$val ;; --control) MEASURE_CONTROL=$val ;; --unit) MEASURE_UNIT=$val ;; - --control-again) MEASURE_CONTROL2=$val ;; + --control-again) MEASURE_CONTROL2+=("$val") ;; + --cleared) MEASURE_CLEARED=$val ;; --state-max-bytes) MEASURE_CAP=$val ;; esac done @@ -539,7 +420,10 @@ measure_check_args() { measure_require dir "$MEASURE_DIR" || return 1 measure_require control "$MEASURE_CONTROL" || return 1 measure_require unit "$MEASURE_UNIT" || return 1 - measure_require control-again "$MEASURE_CONTROL2" || return 1 + if [[ ${#MEASURE_CONTROL2[@]} -eq 0 ]]; then + measure_error "attribute needs --control-again at least once; it is repeatable, and every extra control build narrows what counts as noise" + return 1 + fi ;; *) measure_error "unknown subcommand '$MEASURE_SUB' (one of: $MEASURE_SUBCOMMANDS)" @@ -548,194 +432,3 @@ measure_check_args() { esac return 0 } - -# --------------------------------------------------------------------------- -# reporting -# --------------------------------------------------------------------------- - -# How many paths a report names before it says how many it left out. -MEASURE_PRINT_MAX=40 - -# measure_print_set LABEL SET -measure_print_set() { - local label=$1 set=$2 n - n=$(grep -c . <<<"$set" || true) - [[ "$n" -gt 0 ]] || return 0 - printf ' %s (%s):\n' "$label" "$n" - measure_lines " " "$(head -n "$MEASURE_PRINT_MAX" <<<"$set")" - if [[ "$n" -gt "$MEASURE_PRINT_MAX" ]]; then - printf ' and %s more\n' "$((n - MEASURE_PRINT_MAX))" - fi - return 0 -} - -# measure_describe DIR TAG -> the one line a report prints per capture -measure_describe() { - printf '%s %s on %s, epoch %s, captured %s, %s packages, %s files, %s symlinks\n' \ - "$2" \ - "$(measure_meta_get "$1/$2" layer)" \ - "$(measure_meta_get "$1/$2" parent)" \ - "$(measure_meta_get "$1/$2" epoch)" \ - "$(measure_meta_get "$1/$2" captured)" \ - "$(measure_count "$1/$2/pkgs.tsv")" \ - "$(measure_count "$1/$2/tree.tsv")" \ - "$(measure_count "$1/$2/links.tsv")" -} - -# measure_compare_one A B KIND -> the report for one kind, 1 if it differs -measure_compare_one() { - local a=$1 b=$2 kind=$3 - local onlya onlyb changed na nb nc - onlya=$(measure_only_in "$a/$kind.tsv" "$b/$kind.tsv") - onlyb=$(measure_only_in "$b/$kind.tsv" "$a/$kind.tsv") - changed=$(measure_changed "$a/$kind.tsv" "$b/$kind.tsv") - na=$(grep -c . <<<"$onlya" || true) - nb=$(grep -c . <<<"$onlyb" || true) - nc=$(grep -c . <<<"$changed" || true) - printf '%s: %s differing, %s only in the first, %s only in the second\n' \ - "$kind" "$nc" "$na" "$nb" - measure_print_set "differing" "$changed" - measure_print_set "only in the first" "$onlya" - measure_print_set "only in the second" "$onlyb" - [[ $((na + nb + nc)) -eq 0 ]] -} - -# measure_compare_report DIR A B -> 0 when the two captures are identical -measure_compare_report() { - local dir=$1 a=$2 b=$3 kind differs=0 - echo "=== bt-layer-measure compare ===============================================" - printf 'command bt-layer-measure compare --dir %s --a %s --b %s\n' "$dir" "$a" "$b" - printf 'first %s\n' "$(measure_describe "$dir" "$a")" - printf 'second %s\n' "$(measure_describe "$dir" "$b")" - for kind in pkgs links tree; do - measure_compare_one "$dir/$a" "$dir/$b" "$kind" || differs=1 - done - if [[ $differs -eq 0 ]]; then - echo "the two captures are identical: same packages, same symlinks, same files" - fi - echo "============================================================================" - return $differs -} - -# measure_attributable A B C2_BASE C2_OTHER KIND WHICH -# The differences between A and B that the control pair does not produce. -# changed keys both have whose record differs -# gone keys A has and B does not -# added keys B has and A does not -measure_attributable() { - local a=$1 b=$2 c=$3 c2=$4 kind=$5 which=$6 - local pair floor - case "$which" in - changed) - pair=$(measure_changed "$a/$kind.tsv" "$b/$kind.tsv") - floor=$(measure_changed "$c/$kind.tsv" "$c2/$kind.tsv") - ;; - gone) - pair=$(measure_only_in "$a/$kind.tsv" "$b/$kind.tsv") - floor=$(measure_only_in "$c/$kind.tsv" "$c2/$kind.tsv") - ;; - added) - pair=$(measure_only_in "$b/$kind.tsv" "$a/$kind.tsv") - floor=$(measure_only_in "$c2/$kind.tsv" "$c/$kind.tsv") - ;; - esac - measure_set_minus "$pair" "$floor" -} - -# measure_state_paths C U GLOBS -> every state path the unit pair differs at -measure_state_paths() { - local c=$1 u=$2 globs=$3 path all=() sorted=() - mapfile -t all < <(measure_changed "$c/tree.tsv" "$u/tree.tsv"; measure_only_in "$c/tree.tsv" "$u/tree.tsv"; measure_only_in "$u/tree.tsv" "$c/tree.tsv") - [[ ${#all[@]} -gt 0 ]] || return 0 - mapfile -t sorted < <(printf '%s\n' "${all[@]}" | LC_ALL=C sort -u) - for path in "${sorted[@]}"; do - [[ -n "$path" ]] || continue - measure_state_match "$path" "$globs" && printf '%s\n' "$path" - done - return 0 -} - -# measure_attribute_report DIR CONTROL UNIT CONTROL_AGAIN GLOBS GLOBS_FILE -# The three-build comparison handbook decision 0010 step 3 asks for. Returns -# 0 only when nothing is attributable to the change and nothing inside the -# noise floor was either real or left unread. -measure_attribute_report() { - local dir=$1 ctl=$2 unit=$3 ctl2=$4 globs=$5 globs_file=$6 - local C=$dir/$ctl U=$dir/$unit C2=$dir/$ctl2 - local kind which set n attributable=0 - echo "=== bt-layer-measure attribution ===========================================" - printf 'command bt-layer-measure attribute --dir %s --control %s --unit %s --control-again %s\n' \ - "$dir" "$ctl" "$unit" "$ctl2" - printf 'state paths %s (sha256 %s)\n' "$globs_file" \ - "$(sha256sum < "$globs_file" | cut -c1-16)" - printf 'control %s\n' "$(measure_describe "$dir" "$ctl")" - printf 'unit %s\n' "$(measure_describe "$dir" "$unit")" - printf 'control again %s\n' "$(measure_describe "$dir" "$ctl2")" - echo - echo "noise floor, the control against itself:" - for kind in pkgs links tree; do - printf ' %-6s %s differing, %s only in the first, %s only in the second\n' "$kind" \ - "$(grep -c . <<<"$(measure_changed "$C/$kind.tsv" "$C2/$kind.tsv")" || true)" \ - "$(grep -c . <<<"$(measure_only_in "$C/$kind.tsv" "$C2/$kind.tsv")" || true)" \ - "$(grep -c . <<<"$(measure_only_in "$C2/$kind.tsv" "$C/$kind.tsv")" || true)" - done - echo "the control against the unit:" - for kind in pkgs links tree; do - printf ' %-6s %s differing, %s only in the first, %s only in the second\n' "$kind" \ - "$(grep -c . <<<"$(measure_changed "$C/$kind.tsv" "$U/$kind.tsv")" || true)" \ - "$(grep -c . <<<"$(measure_only_in "$C/$kind.tsv" "$U/$kind.tsv")" || true)" \ - "$(grep -c . <<<"$(measure_only_in "$U/$kind.tsv" "$C/$kind.tsv")" || true)" - done - echo - local detail="" - for kind in pkgs links tree; do - for which in changed gone added; do - set=$(measure_attributable "$C" "$U" "$C" "$C2" "$kind" "$which") - n=$(grep -c . <<<"$set" || true) - attributable=$((attributable + n)) - [[ "$n" -gt 0 ]] || continue - detail="$detail$(measure_print_set "$kind $which" "$set")"$'\n' - done - done - printf 'attributable differences: %s\n' "$attributable" - echo " (the control-unit pair minus the control-control pair, over packages, symlinks and files)" - [[ -z "$detail" ]] || printf '%s' "$detail" - echo - - local paths verdict why path noise=0 real=0 unsampled=0 evidence="" - paths=$(measure_state_paths "$C" "$U" "$globs") - printf 'state paths the unit pair differs at: %s\n' "$(grep -c . <<<"$paths" || true)" - echo " (a path that can hold accounts, credentials, keys or database content is" - echo " never subtracted because it fell inside the noise floor; its bytes are" - echo " compared in both pairs and the difference is named)" - while IFS= read -r path; do - [[ -n "$path" ]] || continue - IFS=$'\t' read -r verdict why < <(measure_state_verdict "$C" "$U" "$C2" "$path") - case "$verdict" in - noise) noise=$((noise + 1)) ;; - real) real=$((real + 1)) ;; - unsampled) unsampled=$((unsampled + 1)) ;; - esac - printf ' %-10s %s\n' "$verdict" "$path" - printf ' %s\n' "$why" - if [[ "$verdict" != noise ]]; then - evidence="$evidence$(measure_state_evidence "$C" "$U" "$C2" "$path")"$'\n' - fi - done <<<"$paths" - printf ' totals: noise %s, real %s, not sampled %s\n' "$noise" "$real" "$unsampled" - if [[ -n "$evidence" ]]; then - echo - echo "evidence for every state path that is not plain noise:" - printf '%s' "$evidence" - fi - echo - if [[ $((attributable + real + unsampled)) -eq 0 ]]; then - echo "verdict: PASS" - echo "============================================================================" - return 0 - fi - printf 'verdict: FAIL (%s attributable, %s real inside the noise floor, %s not sampled)\n' \ - "$attributable" "$real" "$unsampled" - echo "============================================================================" - return 1 -} diff --git a/bin/layer-report-lib b/bin/layer-report-lib new file mode 100644 index 0000000..ea3c551 --- /dev/null +++ b/bin/layer-report-lib @@ -0,0 +1,413 @@ +#!/bin/bash +# Copyright (c) 2026 KeelLinux +# +# This file is part of buildtasks. +# +# Buildtasks is free software; you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published by the +# Free Software Foundation; either version 3 of the License, or (at your +# option) any later version. +# +# The blocks bt-layer-measure prints, and the preconditions it refuses to +# measure without. Sourced after bin/layer-measure-lib and +# bin/layer-compare-lib. +# +# Two rules this file exists to hold, both of them learned the hard way. +# +# Nothing the report subtracts is invisible. A count of 176 subtracted paths +# told a reviewer nothing; the paths are named, and any of them whose name +# looks like state (share/layer-state-suspect) fails the run rather than +# passing unexamined, because state at a path nobody listed is the original +# defect relocated. +# +# The floor is the union over every control pair. With two control captures +# the mariadb residue is three files and with the other pair it is zero, so +# a single pair decides the verdict by accident. --control-again is +# repeatable and every pair contributes. + +# How many paths a report names before it says how many it left out. +MEASURE_PRINT_MAX=40 + +# Set by measure_attribute_report and read by the sections it calls, which +# is how a verdict line reaches the totals without threading a dozen +# arguments through three functions. +MEASURE_ATTRIBUTABLE=0 +MEASURE_SUSPECT_COUNT=0 + +# --------------------------------------------------------------------------- +# printing helpers +# --------------------------------------------------------------------------- + +# measure_print_set LABEL SET +measure_print_set() { + local label=$1 set=$2 n + n=$(grep -c . <<<"$set" || true) + [[ "$n" -gt 0 ]] || return 0 + printf ' %s (%s):\n' "$label" "$n" + measure_lines " " "$(head -n "$MEASURE_PRINT_MAX" <<<"$set")" + if [[ "$n" -gt "$MEASURE_PRINT_MAX" ]]; then + printf ' and %s more\n' "$((n - MEASURE_PRINT_MAX))" + fi + return 0 +} + +# measure_describe DIR TAG +measure_describe() { + printf '%s %s on %s, epoch %s, captured %s, %s packages, %s files, %s symlinks, %s state paths\n' \ + "$2" \ + "$(measure_meta_get "$1/$2" layer)" \ + "$(measure_meta_get "$1/$2" parent)" \ + "$(measure_meta_get "$1/$2" epoch)" \ + "$(measure_meta_get "$1/$2" captured)" \ + "$(measure_count "$1/$2/pkgs.tsv")" \ + "$(measure_count "$1/$2/tree.tsv")" \ + "$(measure_count "$1/$2/links.tsv")" \ + "$(measure_count "$1/$2/state.tsv")" +} + +# --------------------------------------------------------------------------- +# preconditions +# --------------------------------------------------------------------------- + +# measure_preconditions DIR UNIT CONTROL... +# The note this implements says the captures must share SOURCE_DATE_EPOCH, +# one recipe and one parent "or nothing the comparison prints means +# anything". Recording that and never checking it is the same as not +# recording it. +measure_preconditions() { + local dir=$1 unit=$2 + shift 2 + local -a tags=("$unit" "$@") + local tag key want got base=$1 rc=0 + for tag in "${tags[@]}"; do + measure_capture_check "$dir/$tag" || return 1 + done + if [[ "$(printf '%s\n' "${tags[@]}" | LC_ALL=C sort -u | wc -l)" -ne "${#tags[@]}" ]]; then + measure_error "the same capture is named more than once: ${tags[*]}" + return 1 + fi + for key in layer parent epoch; do + want=$(measure_meta_get "$dir/$base" "$key") + for tag in "${tags[@]}"; do + got=$(measure_meta_get "$dir/$tag" "$key") + if [[ "$got" != "$want" ]]; then + measure_error "$key differs between captures: $base says '$want', $tag says '$got'" + rc=1 + fi + done + done + if [[ "$(measure_meta_get "$dir/$base" epoch)" == none ]]; then + measure_error "these captures were built without SOURCE_DATE_EPOCH, so every timestamp in them is free to move and no comparison of them means anything" + rc=1 + fi + for tag in "${tags[@]}"; do + if ! cmp -s -- "$dir/$base/state-paths.txt" "$dir/$tag/state-paths.txt"; then + measure_error "$tag was captured against a different state path list than $base; the list is what decides which bytes were kept, so the two cannot be compared" + rc=1 + fi + want=$(measure_meta_get "$dir/$tag" state_paths_sha256) + got=$(sha256sum < "$dir/$tag/state-paths.txt" | cut -d' ' -f1) + if [[ "$want" != "$got" ]]; then + measure_error "$tag records state_paths_sha256 $want and carries a list that hashes to $got" + rc=1 + fi + done + return $rc +} + +# --------------------------------------------------------------------------- +# compare: two captures +# --------------------------------------------------------------------------- + +# measure_compare_one A B KIND +measure_compare_one() { + local a=$1 b=$2 kind=$3 + local onlya onlyb changed na nb nc + onlya=$(measure_only_in "$a/$kind.tsv" "$b/$kind.tsv") + onlyb=$(measure_only_in "$b/$kind.tsv" "$a/$kind.tsv") + changed=$(measure_changed "$a/$kind.tsv" "$b/$kind.tsv") + na=$(grep -c . <<<"$onlya" || true) + nb=$(grep -c . <<<"$onlyb" || true) + nc=$(grep -c . <<<"$changed" || true) + printf '%s: %s differing, %s only in the first, %s only in the second\n' \ + "$kind" "$nc" "$na" "$nb" + measure_print_set "differing" "$changed" + measure_print_set "only in the first" "$onlya" + measure_print_set "only in the second" "$onlyb" + [[ $((na + nb + nc)) -eq 0 ]] +} + +# measure_compare_report DIR A B +measure_compare_report() { + local dir=$1 a=$2 b=$3 kind differs=0 + echo "=== bt-layer-measure compare ===============================================" + printf 'command bt-layer-measure compare --dir %s --a %s --b %s\n' "$dir" "$a" "$b" + printf 'first %s\n' "$(measure_describe "$dir" "$a")" + printf 'second %s\n' "$(measure_describe "$dir" "$b")" + for kind in pkgs links tree; do + measure_compare_one "$dir/$a" "$dir/$b" "$kind" || differs=1 + done + if [[ $differs -eq 0 ]]; then + echo "the two captures are identical: same packages, same symlinks, same files" + fi + echo "============================================================================" + return $differs +} + +# --------------------------------------------------------------------------- +# attribute: one unit against every control pair +# --------------------------------------------------------------------------- + +# measure_floor_set DIR KIND WHICH CONTROL... +# The union over every control pair, which is what makes a residue mean +# something rather than reflect which pair was named first. +measure_floor_set() { + local dir=$1 kind=$2 which=$3 + shift 3 + local -a c=("$@") + local i j all="" + for i in $(seq 0 $(( ${#c[@]} - 1 ))); do + for j in $(seq 0 $(( ${#c[@]} - 1 ))); do + [[ $i -ne $j ]] || continue + if [[ "$which" == changed ]]; then + [[ $i -lt $j ]] || continue + all+=$(measure_changed "$dir/${c[$i]}/$kind.tsv" "$dir/${c[$j]}/$kind.tsv")$'\n' + else + all+=$(measure_only_in "$dir/${c[$i]}/$kind.tsv" "$dir/${c[$j]}/$kind.tsv")$'\n' + fi + done + done + [[ -n "${all//[$'\n']/}" ]] || return 0 + LC_ALL=C sort -u <<<"$all" | grep . + return 0 +} + +# measure_pair_set DIR KIND WHICH BASE UNIT +measure_pair_set() { + local dir=$1 kind=$2 which=$3 base=$4 unit=$5 + case "$which" in + changed) measure_changed "$dir/$base/$kind.tsv" "$dir/$unit/$kind.tsv" ;; + gone) measure_only_in "$dir/$base/$kind.tsv" "$dir/$unit/$kind.tsv" ;; + added) measure_only_in "$dir/$unit/$kind.tsv" "$dir/$base/$kind.tsv" ;; + esac +} + +# measure_attributable DIR KIND WHICH UNIT CONTROL... +measure_attributable() { + local dir=$1 kind=$2 which=$3 unit=$4 + shift 4 + measure_set_minus "$(measure_pair_set "$dir" "$kind" "$which" "$1" "$unit")" \ + "$(measure_floor_set "$dir" "$kind" "$which" "$@")" +} + +# measure_state_differs DIR GLOBS UNIT BASE -> every state path the unit pair +# differs at, changed, gone or added +measure_state_differs() { + local dir=$1 globs=$2 unit=$3 base=$4 + local path all=() sorted=() + mapfile -t all < <(measure_pair_set "$dir" tree changed "$base" "$unit" + measure_pair_set "$dir" tree gone "$base" "$unit" + measure_pair_set "$dir" tree added "$base" "$unit") + [[ ${#all[@]} -gt 0 ]] || return 0 + mapfile -t sorted < <(printf '%s\n' "${all[@]}" | LC_ALL=C sort -u | grep . || true) + for path in "${sorted[@]}"; do + measure_state_match "$path" "$globs" && printf '%s\n' "$path" + done + return 0 +} + +# measure_suspect_check SET SUSPECTS -> the subtracted paths whose name looks +# like state although no state path glob matches them +measure_suspect_check() { + local set=$1 suspects=$2 path + [[ -n "$set" ]] || return 0 + [[ -f "$suspects" ]] || return 0 + local globs + globs=$(measure_state_patterns "$suspects") || return 0 + while IFS= read -r path; do + [[ -n "$path" ]] || continue + measure_state_match "$path" "$globs" && printf '%s\n' "$path" + done <<<"$set" + return 0 +} + +# measure_attribute_report DIR UNIT CLEARED SUSPECTS CONTROL... +measure_attribute_report() { + local dir=$1 unit=$2 cleared=$3 suspects=$4 + shift 4 + local -a c=("$@") + local base=${c[0]} + local globs_file=$dir/$base/state-paths.txt + local globs kind which set n tag i j + globs=$(measure_state_patterns "$globs_file") || return 1 + + echo "=== bt-layer-measure attribution ===========================================" + printf 'command bt-layer-measure attribute --dir %s --unit %s --control %s' \ + "$dir" "$unit" "$base" + for i in $(seq 1 $(( ${#c[@]} - 1 ))); do + printf ' --control-again %s' "${c[$i]}" + done + [[ ! -f "$cleared" ]] || printf ' --cleared %s' "$cleared" + printf '\n' + printf 'state paths %s captures agree on the list (sha256 %s)\n' \ + "$(( ${#c[@]} + 1 ))" "$(measure_meta_get "$dir/$base" state_paths_sha256 | cut -c1-16)" + printf 'unit %s\n' "$(measure_describe "$dir" "$unit")" + for tag in "${c[@]}"; do + printf 'control %s\n' "$(measure_describe "$dir" "$tag")" + done + printf 'control pairs %s, and the floor below is their union\n' \ + "$(( ${#c[@]} * (${#c[@]} - 1) / 2 ))" + echo + + # the per-pair counts, so the swing that one pair can cause is visible + if [[ ${#c[@]} -gt 1 ]]; then + echo "attributable file differences if a single control pair were the floor:" + for i in $(seq 0 $(( ${#c[@]} - 1 ))); do + for j in $(seq $(( i + 1 )) $(( ${#c[@]} - 1 ))); do + printf ' %s vs %s: %s\n' "${c[$i]}" "${c[$j]}" \ + "$(grep -c . <<<"$(measure_set_minus \ + "$(measure_pair_set "$dir" tree changed "$base" "$unit")" \ + "$(measure_changed "$dir/${c[$i]}/tree.tsv" "$dir/${c[$j]}/tree.tsv")")" || true)" + done + done + echo " (this command uses the union of all of them, below)" + echo + fi + + local attributable=0 detail="" + for kind in pkgs links tree; do + for which in changed gone added; do + set=$(measure_attributable "$dir" "$kind" "$which" "$unit" "${c[@]}") + n=$(grep -c . <<<"$set" || true) + attributable=$((attributable + n)) + [[ "$n" -gt 0 ]] || continue + detail="$detail$(measure_print_set "$kind $which" "$set")"$'\n' + done + done + MEASURE_ATTRIBUTABLE=$attributable + printf 'attributable differences: %s\n' "$attributable" + echo " (the control-unit pair minus the union of every control pair, over" + echo " packages, symlinks and files)" + [[ -z "$detail" ]] || printf '%s' "$detail" + echo + + measure_state_section "$dir" "$unit" "$cleared" "$suspects" "$globs" "${c[@]}" +} + +# measure_state_section DIR UNIT CLEARED SUSPECTS GLOBS CONTROL... +measure_state_section() { + local dir=$1 unit=$2 cleared=$3 suspects=$4 globs=$5 + shift 5 + local -a c=("$@") + local base=${c[0]} + local paths path verdict why reason tag + local -a samples=() + local noise=0 real=0 overlapping=0 unsampled=0 none=0 waived=0 examined=0 + local evidence="" shown=0 abbreviated=0 waivers="" + + paths=$(measure_state_differs "$dir" "$globs" "$unit" "$base") + printf 'state paths the unit pair differs at: %s\n' "$(grep -c . <<<"$paths" || true)" + echo " (a path that can hold accounts, credentials, keys or database content is" + echo " never subtracted because it fell inside the floor: its bytes are compared" + echo " against every control, and the evidence below is printed whatever the" + echo " verdict, because the one mistake this scheme can make is to call a real" + echo " difference noise)" + while IFS= read -r path; do + [[ -n "$path" ]] || continue + examined=$((examined + 1)) + samples=() + for tag in "${c[@]}"; do + samples+=("$(measure_sample_path "$dir/$tag" "$path")") + done + IFS=$'\t' read -r verdict why < <(measure_verdict \ + "$(measure_sample_path "$dir/$unit" "$path")" "${samples[@]}") + if [[ -z "$verdict" ]]; then + measure_error "no verdict for $path, which must not happen" + return 1 + fi + reason=$(measure_cleared_reason "$cleared" "$path") + if [[ -n "$reason" ]] && [[ "$verdict" == overlapping ]]; then + waived=$((waived + 1)) + waivers="$waivers$(printf '%s\t%s' "$path" "$reason")"$'\n' + continue + fi + case "$verdict" in + noise) noise=$((noise + 1)) ;; + real) real=$((real + 1)) ;; + overlapping) overlapping=$((overlapping + 1)) ;; + unsampled) unsampled=$((unsampled + 1)) ;; + none) none=$((none + 1)) ;; + esac + printf ' %-11s %s\n' "$verdict" "$path" + printf ' %s\n' "$why" + if [[ "$verdict" == real ]] || [[ $shown -lt $MEASURE_EVIDENCE_PATHS ]]; then + shown=$((shown + 1)) + evidence="$evidence$(measure_evidence "$path" "$(measure_sample_path "$dir/$unit" "$path")" "${samples[@]}")"$'\n' + else + abbreviated=$((abbreviated + 1)) + fi + done <<<"$paths" + if [[ $((noise + real + overlapping + unsampled + none + waived)) -ne $examined ]]; then + measure_error "the verdict totals do not add up to the $examined paths examined" + return 1 + fi + printf ' totals: noise %s, overlapping %s, real %s, not sampled %s, unchanged %s, cleared %s\n' \ + "$noise" "$overlapping" "$real" "$unsampled" "$none" "$waived" + if [[ -n "$waivers" ]]; then + echo + printf 'waived in writing (%s paths, every one of them named):\n' "$waived" + measure_lines " " "$(head -n "$MEASURE_PRINT_MAX" <<<"$waivers" \ + | sed "s/\t/ -> /")" + [[ "$waived" -le "$MEASURE_PRINT_MAX" ]] \ + || printf ' and %s more, all covered by the same waiver file\n' \ + "$((waived - MEASURE_PRINT_MAX))" + fi + if [[ -n "$evidence" ]]; then + echo + echo "evidence, for every state path above:" + printf '%s' "$evidence" + [[ $abbreviated -eq 0 ]] || printf ' (%s more abbreviated to their summary line above)\n' "$abbreviated" + fi + echo + + measure_subtracted_section "$dir" "$unit" "$suspects" "$globs" "${c[@]}" + local suspect=$MEASURE_SUSPECT_COUNT + echo + if [[ $((MEASURE_ATTRIBUTABLE + real + overlapping + unsampled + suspect)) -eq 0 ]]; then + echo "verdict: PASS" + echo "============================================================================" + return 0 + fi + printf 'verdict: FAIL (%s attributable, %s real, %s overlapping and uncleared, %s not sampled, %s subtracted but state shaped)\n' \ + "$MEASURE_ATTRIBUTABLE" "$real" "$overlapping" "$unsampled" "$suspect" + echo "============================================================================" + return 1 +} + +# measure_subtracted_section DIR UNIT SUSPECTS GLOBS CONTROL... +# What the floor removed, by name, because a count of 176 tells a reviewer +# nothing. Sets MEASURE_SUSPECT_COUNT to the number of subtracted paths whose +# name looks like state although the list does not cover them. +measure_subtracted_section() { + local dir=$1 unit=$2 suspects=$3 globs=$4 + shift 4 + local -a c=("$@") + local base=${c[0]} + local inside path notstate="" suspect + inside=$(LC_ALL=C comm -12 <(measure_pair_set "$dir" tree changed "$base" "$unit" | LC_ALL=C sort -u) <(measure_floor_set "$dir" tree changed "${c[@]}" | LC_ALL=C sort -u)) + while IFS= read -r path; do + [[ -n "$path" ]] || continue + measure_state_match "$path" "$globs" || notstate="$notstate$path"$'\n' + done <<<"$inside" + printf 'subtracted as floor noise, and not state paths: %s\n' \ + "$(grep -c . <<<"$notstate" || true)" + measure_print_set "subtracted" "$notstate" + suspect=$(measure_suspect_check "$notstate" "$suspects") + if [[ -n "$suspect" ]]; then + echo " these were subtracted and their names look like state; add them to" + echo " share/layer-state-paths so their bytes are compared, or clear them:" + measure_lines " " "$suspect" + fi + MEASURE_SUSPECT_COUNT=$(grep -c . <<<"$suspect" || true) + return 0 +} diff --git a/bt-layer-measure b/bt-layer-measure index 20be1ef..8cf40e3 100755 --- a/bt-layer-measure +++ b/bt-layer-measure @@ -15,6 +15,16 @@ APP=$(basename "$0") BT=${BT:-$(dirname "$(readlink -f "$0")")} # shellcheck source=bin/layer-measure-lib source "$BT/bin/layer-measure-lib" +# shellcheck source=bin/layer-compare-lib +source "$BT/bin/layer-compare-lib" +# shellcheck source=bin/layer-report-lib +source "$BT/bin/layer-report-lib" + +# bt-layer learns BT_BUILDS from this file, so this reads it the same way +# rather than disagreeing with the builder about where the rootfs landed. +BT_CONFIG=${BT_CONFIG:-$BT/config} +# shellcheck disable=SC1091 +[[ ! -f "$BT_CONFIG/common.cfg" ]] || source "$BT_CONFIG/common.cfg" EXIT_DIFFERENCES=1 EXIT_USAGE=2 @@ -29,14 +39,22 @@ usage() { Syntax: $APP capture --dir DIR --layer NAME [--parent NAME] --tag TAG [--state-max-bytes N] $APP compare --dir DIR --a TAG --b TAG - $APP attribute --dir DIR --control TAG --unit TAG --control-again TAG - -Builds a layer and records what it contains, then says what two or three of -those records differ by. This is the three-build comparison handbook decision -0010 step 3 requires of every component extracted from the shared tree: -a control build, the same recipe with the component supplied as a fab unit, -and the control again, because without the third build a difference cannot be -attributed to anything. + $APP attribute --dir DIR --unit TAG --control TAG + --control-again TAG [--control-again TAG]... + [--cleared FILE] + +Builds a layer and records what it contains, then says what those records +differ by. This is the three-build comparison handbook decision 0010 step 3 +requires of every component extracted from the shared tree: a control build, +the same recipe with the component supplied as a fab unit, and the control +again, because without a second control a difference cannot be attributed to +anything. + +Give more than one --control-again. Two builds of one recipe do not differ by +a fixed set: on the mariadb captures of 2026-09-27 one control pair differs by +179 files and another by 182, and the attributable count is 3 against the +first and 0 against the second. The floor is the union over every control +pair, so each extra control build narrows what this command will call noise. Subcommands:: @@ -66,21 +84,49 @@ Options:: --a, --b TAG the two captures compare reads --control TAG the control build --unit TAG the build with the component supplied as a unit - --control-again TAG the second control build, which is what makes a - difference attributable + --control-again TAG another control build. Repeatable, and the floor is + the union over every pair of controls + --cleared FILE waivers: lines of PATHREASON. A state path whose + difference coincides with the floor but cannot be + shown to be that variation is 'overlapping' and fails + the run; a waiver clears it and prints the reason in + the block, so the judgement is on the record. A + waiver cannot clear a 'real' difference Environment:: - MEASURE_STATE_PATHS the list of state path globs - (default \$BT/share/layer-state-paths) + MEASURE_STATE_PATHS the list of state path globs a *capture* is taken + against (default \$BT/share/layer-state-paths). The + list is copied into each capture and 'attribute' + reads the copy, never this variable, so what a + measurement was blind to cannot be changed after the + fact + MEASURE_SUSPECT_PATHS + names that look like state (default + \$BT/share/layer-state-suspect). A path the floor + subtracted whose name matches one of these, and which + no state path glob covers, fails the run MEASURE_BT_LAYER the builder to run (default \$BT/bt-layer) - MEASURE_LAYERS_DIR where the builder leaves \$layer.rootfs - (default /mnt/builds/layers) + BT_BUILDS bt-layer's output root; \$BT_BUILDS/layers is where + the built rootfs is read from, and bt-layer itself + refuses to run without it + MEASURE_LAYERS_DIR overrides \$BT_BUILDS/layers MEASURE_LOCK the build lock (default /run/lock/keel-build.lock) MEASURE_LOCK_WAIT seconds to wait for it (default 7200) SOURCE_DATE_EPOCH passed through to the builder and recorded; the three builds must share it or nothing below means anything +Verdicts on a state path:: + + noise every difference is at a position the controls vary + at and has the shape they vary with there + overlapping it coincides with the controls' variation but cannot + be shown to be that variation; bytes never can be, so + a binary is never noise. Fails unless cleared + real it differs where the controls do not vary, or with + content or a magnitude they never show + not sampled no bytes were kept, so nothing can be said. Fails + Exit status:: 0 no differences, or none attributable to the change @@ -92,10 +138,18 @@ Documentation: docs/layer-measure.md EOF } +# The list of state path globs a capture is taken against. Only capture +# consults it: attribute reads the copy each capture carries, so the +# blindness of a measurement cannot be switched off after the fact by an +# environment variable while the printed block still looks provenanced. state_paths_file() { printf '%s\n' "${MEASURE_STATE_PATHS:-$BT/share/layer-state-paths}" } +suspect_paths_file() { + printf '%s\n' "${MEASURE_SUSPECT_PATHS:-$BT/share/layer-state-suspect}" +} + do_capture() { local dest=$MEASURE_DIR/$MEASURE_TAG if [[ -e "$dest" ]]; then @@ -106,14 +160,20 @@ do_capture() { globs=$(measure_state_patterns "$globs_file") || die "$EXIT_INPUT" "no usable state path list" local builder=${MEASURE_BT_LAYER:-$BT/bt-layer} - local layers=${MEASURE_LAYERS_DIR:-/mnt/builds/layers} + if [[ -z "${MEASURE_LAYERS_DIR:-}" ]] && [[ -z "${BT_BUILDS:-}" ]]; then + die "$EXIT_INPUT" "BT_BUILDS is not set and $BT_CONFIG/common.cfg did not set it; bt-layer refuses to run without it, so export it or set MEASURE_LAYERS_DIR" + fi + local layers=${MEASURE_LAYERS_DIR:-$BT_BUILDS/layers} local lock=${MEASURE_LOCK:-/run/lock/keel-build.lock} local lock_wait=${MEASURE_LOCK_WAIT:-7200} local argv=("$builder") [[ -z "$MEASURE_PARENT" ]] || argv+=(--parent "$MEASURE_PARENT") argv+=("$MEASURE_LAYER") + mkdir -p "$MEASURE_DIR" + chmod 700 "$MEASURE_DIR" mkdir -p "$dest" + chmod 700 "$dest" info "$MEASURE_TAG: ${argv[*]}, waiting up to ${lock_wait}s for $lock" if ! flock -w "$lock_wait" "$lock" "${argv[@]}" > "$dest/build.log" 2>&1; then tail -n 20 "$dest/build.log" 1>&2 @@ -131,6 +191,9 @@ do_capture() { measure_pkgs_record "$rootfs" > "$dest/pkgs.tsv" \ || die "$EXIT_INPUT" "could not record the package list of $rootfs" measure_state_sample "$rootfs" "$dest/tree.tsv" "$dest" "$MEASURE_CAP" "$globs" + # the list travels with the capture, so attribute never has to be told + # which one was used and cannot be told a different one + cp -- "$globs_file" "$dest/state-paths.txt" if [[ -f "$layers/$MEASURE_LAYER.manifest" ]]; then cp -- "$layers/$MEASURE_LAYER.manifest" "$dest/manifest.txt" @@ -166,15 +229,12 @@ do_compare() { } do_attribute() { - local tag - for tag in "$MEASURE_CONTROL" "$MEASURE_UNIT" "$MEASURE_CONTROL2"; do - measure_capture_check "$MEASURE_DIR/$tag" || die "$EXIT_INPUT" "no complete capture $tag in $MEASURE_DIR" - done - local globs_file globs - globs_file=$(state_paths_file) - globs=$(measure_state_patterns "$globs_file") || die "$EXIT_INPUT" "no usable state path list" - measure_attribute_report "$MEASURE_DIR" "$MEASURE_CONTROL" "$MEASURE_UNIT" \ - "$MEASURE_CONTROL2" "$globs" "$globs_file" || exit "$EXIT_DIFFERENCES" + measure_preconditions "$MEASURE_DIR" "$MEASURE_UNIT" \ + "$MEASURE_CONTROL" "${MEASURE_CONTROL2[@]}" \ + || die "$EXIT_INPUT" "the captures cannot be compared" + measure_attribute_report "$MEASURE_DIR" "$MEASURE_UNIT" \ + "${MEASURE_CLEARED:-/nonexistent}" "$(suspect_paths_file)" \ + "$MEASURE_CONTROL" "${MEASURE_CONTROL2[@]}" || exit "$EXIT_DIFFERENCES" } measure_parse_args "$@" || { usage 1>&2; exit "$EXIT_USAGE"; } diff --git a/docs/layer-measure.md b/docs/layer-measure.md index 6dcb019..9819482 100644 --- a/docs/layer-measure.md +++ b/docs/layer-measure.md @@ -9,7 +9,7 @@ and the comparison half was never written down, so two component extractions (keel-mariadb#11, keel-postgresql#7) were approved on a number only their author could produce. -## Why three builds +## Why three builds, and why four is better Two builds of the *same* recipe are not identical. They differ in install-time state: clocks, log lines, Perl hash order, a database data directory created @@ -19,6 +19,14 @@ control at all. Without a second control build there is nothing to subtract, so no difference can be attributed to the change, which is what the third build is for. +The floor is not a fixed set either. On the mariadb captures of 2026-09-27 one +control pair differs by 179 files and another by 182, and the count +attributable to the unit form is **3** against the first pair and **0** +against the second. A single pair therefore decides the verdict by accident. +`--control-again` is repeatable, the floor is the union over every pair of +controls, and the block prints what each single pair would have said so the +swing is visible rather than lucky. + ## Usage bt-layer-measure capture --dir DIR --layer NAME [--parent NAME] --tag TAG @@ -42,7 +50,14 @@ one host, and one lock"). export FAB_PATH=/turnkey/fab-keel RELEASE=debian/trixie FAB_ARCH=amd64 export BT_PATH=/turnkey/buildtasks-keel BT_PRODUCTS=$FAB_PATH/products export TKLBAM_PATH=/turnkey/tklbam-profiles-keel BT_GPGKEY= + export BT_BUILDS=/mnt/builds export SOURCE_DATE_EPOCH=1700000000 + +`BT_BUILDS` is the one `bt-layer` refuses to run without, and it is not in +root's environment on the build host. `bt-layer-measure` reads +`$BT_PATH/config/common.cfg` the way `bt-layer` does, so the default of +`/mnt/builds` applies either way, but exporting it keeps the command honest +about what it depends on. M=/mnt/builds/measure/mariadb-$(date +%Y%m%d) # 1. the control: the recipe as it is on the default branch @@ -55,9 +70,13 @@ one host, and one lock"). # 3. the control again, from the same tree as step 1 $BT_PATH/bt-layer-measure capture --dir $M --layer mariadb --parent core --tag control-again + # 3b. a third control, because one control pair is not a floor + $BT_PATH/bt-layer-measure capture --dir $M --layer mariadb --parent core --tag control-third + # 4. the result the pull request quotes - $BT_PATH/bt-layer-measure attribute --dir $M \ - --control control --unit unit --control-again control-again + $BT_PATH/bt-layer-measure attribute --dir $M --unit unit \ + --control control --control-again control-again \ + --control-again control-third [--cleared waivers.tsv] Step 4 prints one block. Paste it into the pull request whole: it names the command, the three captures with their epoch and capture time, the noise @@ -80,6 +99,7 @@ error, 3 when a capture, a build or a tool is missing. | `links.tsv` | `pathtarget` for every symlink | | `state.tsv` | `pathsampled\|too-largesize` for every state path | | `state/` | the bytes of every sampled state path | +| `state-paths.txt` | the state path list this capture was taken against | | `manifest.txt` | the layer manifest `bt-layer` wrote | | `build.log` | the builder's output | @@ -87,6 +107,20 @@ A tag is never overwritten. Re-measuring means a new tag or a new directory, so the numbers a pull request quoted stay on disk next to the ones that replaced them. +**Retention.** A capture holds byte-for-byte copies of `/etc/shadow`, of host +keys and of database files. `capture` creates the directory `0700` and `cp` +preserves each file's own mode, but nothing expires them: a capture directory +is as sensitive as the image it came from. Keep it on the build host, off any +shared filesystem, and delete it once the pull request it supports is merged. +A `mariadb` capture with the cap raised past `ib_logfile0` is about 150 MiB, +so four of them are roughly 600 MiB. + +`attribute` reads the state path list from the captures, never from the +environment. The list is what decides which bytes were kept, so a run cannot +be made to look clean afterwards by pointing `MEASURE_STATE_PATHS` at a +narrower file: all the captures must carry the same list, and their recorded +digests must match it. + ### Why a tab, and why the path comes first `sha256sum` prints `` then two spaces then the path, and a Debian @@ -109,33 +143,94 @@ noise floor by its name; whether its difference really is noise is a question about its bytes, and nothing was asking it. `share/layer-state-paths` lists the paths that can hold state that matters: -accounts, credentials, host keys, database directories. For those, `capture` -keeps the bytes and `attribute` compares them in both pairs and classifies -each one: - -| Verdict | Meaning | -|---------|---------| -| `noise` | every difference between the control and the unit is one the control also shows against itself, at the same line or the same byte offset | -| `real` | it differs at a line or a byte offset the control pair does not, so the change is what differs, whatever the path is called | -| `unsampled` | the bytes were not kept on some side, so neither can be said | - -`real` and `unsampled` both fail the run. `unsampled` fails on purpose: a -measurement that could not read a file must not pass as if the file were -unchanged. Either raise `--state-max-bytes` (the default is 16 MiB per file) -or say in the pull request why that path does not matter. - -The comparison is per line for a text file and per byte offset for a binary -one, and the noise floor is subtracted at that level rather than per path. A -clock that moves in every build touches the same line or the same offset in -both pairs, so it cancels. A deleted account row leaves a line or an offset -the control pair never produces, so it is reported, with a three-way diff of -the control, the unit and the second control printed underneath. +accounts, credentials, host keys, database directories, `/etc/machine-id`. +For those, `capture` keeps the bytes and `attribute` classifies each one. + +### A position is not enough + +The first version of this tool compared *positions*: the line numbers or byte +offsets at which two files differ, subtracting one set from another. That +cancels a real difference against a coincidental one at the same place. All +four of these were reported as noise, with the evidence suppressed because +they were noise: + +- an account added to `/etc/shadow` at the line number the control pair also + appends at, +- a root password set on the line the control pair rewrites for a clock + reason, +- a binary differing at the same eight offsets as the control pair, with + different byte values, +- a file 3584 bytes shorter, against a control pair that varies by one byte. + +So the controls are the model, and the model has content in it. At each +position the controls vary at, the longest common prefix and suffix of their +variants is the *shape* of that variation, and the unit is admitted only if +its line has that shape. `# written 1790475089` against `# written 1790475323` +gives the shape `# written ...`, which `# written 1790475511` fits and +`root:$y$hash:...` does not. + +| Verdict | Meaning | Run | +|---------|---------|-----| +| `noise` | every differing line is at a position the controls vary at and has the shape they vary with there | passes | +| `overlapping` | it coincides with the controls' variation but cannot be shown to be that variation | **fails** unless cleared | +| `real` | it differs where the controls do not vary, or with content or a magnitude they never show | **fails** | +| `not sampled` | no bytes were kept on some side | **fails** | +| `unchanged` | the path differs in mode only; the bytes are identical | passes | + +A binary is never `noise`. Bytes carry no shape: a timestamp and an account +record look the same at the byte level, so the most that can be said is that +the unit coincides with the controls in position, in size delta and in how +much differs. That is `overlapping`, and it fails until somebody reads the +bytes and says so in `--cleared`. For mariadb that is the honest answer for +`/var/lib/mysql/**`, which is 173 of the 179 files in the floor and holds +`mysql/global_priv` and `mysql/user.*`. + +A mask with neither a prefix nor a suffix is what two unrelated lines have in +common, which is no evidence about a third, so it never yields `noise` either. + +`not sampled` fails on purpose: a measurement that could not read a file must +not pass as if the file were unchanged. Raise `--state-max-bytes` (default +16 MiB per file) or clear the path in writing. + +### Waivers + +`--cleared FILE` takes lines of `PATHREASON`, and `PATH` may be a glob, +because a real data directory is 173 files that one sentence explains: + + ./var/lib/mysql/** InnoDB and Aria creation stamps, read byte by byte on 2026-09-28 + +A waiver clears `overlapping` and nothing else: it cannot clear a `real` +difference. Every path a waiver covered is named in the block, so a reviewer +sees what was waived and on what grounds, and the file itself is committed +next to the component. + +### What the floor removed + +Every path the floor subtracted that is *not* a state path is named in the +block, not just counted, because 176 subtracted paths as a single number told +the reviewer of this tool's first version nothing. Any of them whose name +looks like state (`share/layer-state-suspect`: `*shadow*`, `*passwd*`, +`*.pem`, `*.key`, `*machine-id*` and so on) fails the run, with two ways out +and both of them committed: add the path to `share/layer-state-paths` so its +bytes are compared, or clear it with a reason. State at a path nobody listed +is the original defect relocated, so the list is treated as a failure mode +rather than as a list. + +### Preconditions + +`attribute` refuses to run unless the captures share their `layer`, `parent` +and `SOURCE_DATE_EPOCH`, were built with an epoch at all, carry the same state +path list, and are distinct from one another. The note this implements says +those must hold "or nothing the comparison prints means anything"; recording +them and never checking them is the same as not recording them. ## Layout -Logic in `bin/layer-measure-lib`, a thin main in `bt-layer-measure`, tests in -`tests/measure`, measured by `tests/coverage.sh` (Keel decisions 0003 and -0004). Nothing is built in the tests: the builder is a stub that copies a +Capture and formats in `bin/layer-measure-lib`, the verdicts in +`bin/layer-compare-lib`, the blocks and the preconditions in +`bin/layer-report-lib`, a thin main in `bt-layer-measure`. Tests in +`tests/measure` and `tests/compare`, measured by `tests/coverage.sh` (Keel +decisions 0003 and 0004). Nothing is built in the tests: the builder is a stub that copies a fixture rootfs into place, so every capture the suite makes is a real capture of a real tree, including a path with a space in it, a symlink whose target has a space, a text state file, a binary state file and a state file above the diff --git a/share/layer-state-paths b/share/layer-state-paths index bd8bb04..dfc9bb7 100644 --- a/share/layer-state-paths +++ b/share/layer-state-paths @@ -32,6 +32,22 @@ ./home/**/.ssh/** ./etc/ssl/private/** +# service configuration that carries a credential +./etc/mysql/** +./etc/postgresql/** +./etc/redis/** +./etc/mongodb.conf +./root/.my.cnf +./root/.pgpass +./root/.netrc +./home/**/.my.cnf +./home/**/.pgpass + +# identity that must not be shared between images +# (docs/traps.md, "Every appliance built from core has the same machine-id") +./etc/machine-id +./var/lib/dbus/machine-id + # database state ./var/lib/mysql/** ./var/lib/postgresql/** diff --git a/share/layer-state-suspect b/share/layer-state-suspect new file mode 100644 index 0000000..acf3e64 --- /dev/null +++ b/share/layer-state-suspect @@ -0,0 +1,36 @@ +# Names that look like accounts, credentials or keys. A path matching one of +# these that the noise floor subtracted, and that share/layer-state-paths +# does not cover, fails the run: state at a path nobody listed is the +# original defect relocated, and a count of subtracted paths hides it. +# +# The answer to a hit is one of two committed things: add the path to +# share/layer-state-paths so its bytes are compared, or clear it in the +# --cleared file with a reason. Both are visible in the block a pull +# request quotes. + +*shadow* +*passwd* +*gshadow* +*.htpasswd* +*secret* +*secrets* +*credential* +*token* +*apikey* +*api_key* +*.pem +*.key +*_key +*_key.pub +*keyring* +*.gpg +*.asc +*password* +*passwd.d* +*.my.cnf +*.pgpass +*.netrc +*.npmrc +*authorized_keys* +*machine-id* +*miniserv.users* diff --git a/tests/aplinfo b/tests/aplinfo index c9cd6fe..d01d213 100755 --- a/tests/aplinfo +++ b/tests/aplinfo @@ -26,6 +26,10 @@ # output, changelog mismatch warning, no changelog warning; signed run # with -u and -o, verified with sqv or gpg; parse check of the index. +# pipefail as the scripts under test set it: a library function that +# inherits a pipeline's status must fail here and not in production. +set -o pipefail + fatal() { echo "FATAL [$(basename "$0")]: $*" 1>&2; exit 1; } info() { echo "INFO [$(basename "$0")]: $*"; } diff --git a/tests/compare b/tests/compare new file mode 100755 index 0000000..f643235 --- /dev/null +++ b/tests/compare @@ -0,0 +1,296 @@ +#!/bin/bash -e +# Test bin/layer-compare-lib: the part that decides whether a difference at a +# state path is noise or the change being measured. +# +# The rule this suite exists to hold. A token that records only *where* two +# files differ cancels a real difference against a coincidental one at the +# same position. Every case below was a false negative of the first +# implementation, which reported them as noise, printed no evidence and +# exited 0: +# +# an account added to /etc/shadow at the line number the control pair also +# appends at; a root password set on the line the control pair rewrites for +# a clock reason; a binary differing at the same offsets as the control pair +# but with different byte values; a length change of -3584 bytes against a +# control pair that varies by +1. +# +# So a position is not enough. The control captures are the model of what +# varies, and the unit is admitted only if its *content* at that position is +# the kind of variation the controls show there: the longest common prefix +# and suffix of the control variants of that line. A mask with neither +# prefix nor suffix explains nothing and never yields noise, and bytes are +# never provable at all, so a binary coincidence is `overlapping`, which +# fails until somebody clears it in writing. +# +# Code paths covered: +# measure_is_text (text, NUL, no newline, over-long line, missing); +# measure_line; measure_common_prefix and measure_common_suffix (none, +# partial, whole, one string, empty); measure_text_positions (same, +# changed, added, removed); measure_byte_offsets (same, differing, shorter); +# measure_pair_positions (text, binary); measure_floor_positions (one pair, +# three controls, identical controls); measure_range (single pair, several, +# negative); measure_mask_at (common prefix and suffix, prefix only, suffix +# only, neither, absent line); measure_mask_admits (matches, too short, +# wrong prefix, wrong suffix, empty mask refused); +# measure_verdict (text noise; text real outside the floor; text real by +# content at a floor position; text real by an added line the mask refuses; +# text overlapping on an empty mask; text overlapping over the position +# cap; binary overlapping; binary real outside the floor; binary real by +# size delta; binary real by differing-byte count; identical controls; +# unsampled; a unit identical to the control); +# measure_cleared_reason (listed, absent, missing file, comment); +# measure_evidence (text, binary, unsampled, always printed for noise). + +set -o pipefail + +fatal() { echo "FATAL [$(basename "$0")]: $*" 1>&2; exit 1; } +info() { echo "INFO [$(basename "$0")]: $*"; } + +usage() { +cat</dev/null 2>&1 || fatal "$label: should have passed" +} + +assert_fail() { + local label=$1 + shift + if "$@" >/dev/null 2>&1; then + fatal "$label: should have failed" + fi +} + +assert_grep() { + local label=$1 pattern=$2 haystack=$3 + grep -q -- "$pattern" <<<"$haystack" || fatal "$label: no '$pattern' in: $haystack" +} + +# verdict LABEL WANT UNIT CONTROL... +verdict() { + local label=$1 want=$2 + shift 2 + local got + got=$(measure_verdict "$@") || fatal "$label: measure_verdict failed" + assert_eq "$label" "$(cut -f1 <<<"$got")" "$want" + printf '%s\n' "$got" +} + +shadow() { # shadow FILE STAMP ACCOUNT... + local f=$1 stamp=$2 + shift 2 + { echo "# written $stamp" + echo "root:!:20000:0:99999:7:::" + local a + for a in "$@"; do echo "$a"; done + } > "$f" +} + +info "1: measure_is_text" +printf 'one\ntwo\n' > "$tmp/t" +printf 'a\0b\n' > "$tmp/nul" +printf 'no trailing newline and no newline at all' > "$tmp/noeol" +{ head -c 9000 /dev/zero | tr '\0' 'x'; echo; } > "$tmp/longline" +assert_ok text-plain measure_is_text "$tmp/t" +assert_fail text-nul measure_is_text "$tmp/nul" +assert_fail text-nonewline measure_is_text "$tmp/noeol" +assert_fail text-longline measure_is_text "$tmp/longline" +assert_fail text-missing measure_is_text "$tmp/absent" + +info "2: measure_line, measure_common_prefix and measure_common_suffix" +assert_eq line-1 "$(measure_line 1 "$tmp/t")" one +assert_eq line-2 "$(measure_line 2 "$tmp/t")" two +assert_eq line-absent "$(measure_line 9 "$tmp/t")" "" +assert_eq prefix-some "$(measure_common_prefix "# written 1" "# written 2")" "# written " +assert_eq prefix-none "$(measure_common_prefix abc xyz)" "" +assert_eq prefix-whole "$(measure_common_prefix abc abc)" abc +assert_eq prefix-one "$(measure_common_prefix abc)" abc +assert_eq prefix-empty "$(measure_common_prefix "" abc)" "" +assert_eq suffix-some "$(measure_common_suffix "a:!!:20000" "bb:!!:20000")" ":!!:20000" +assert_eq suffix-none "$(measure_common_suffix abc xyz)" "" +assert_eq suffix-whole "$(measure_common_suffix abc abc)" abc +assert_eq suffix-one "$(measure_common_suffix abc)" abc + +info "3: measure_text_positions and measure_byte_offsets" +printf 'one\ntwo\nthree\n' > "$tmp/a" +printf 'one\nTWO\nthree\n' > "$tmp/b" +printf 'one\ntwo\nthree\nfour\n' > "$tmp/c" +printf 'one\nthree\n' > "$tmp/d" +assert_eq pos-same "$(measure_text_positions "$tmp/a" "$tmp/a")" "" +assert_eq pos-changed "$(measure_text_positions "$tmp/a" "$tmp/b")" "$(printf 'n2\no2')" +assert_eq pos-added "$(measure_text_positions "$tmp/a" "$tmp/c")" "n4" +assert_eq pos-removed "$(measure_text_positions "$tmp/a" "$tmp/d")" "o2" +printf 'AAAA' > "$tmp/ba" +printf 'AABA' > "$tmp/bb" +printf 'AAA' > "$tmp/bc" +assert_eq off-same "$(measure_byte_offsets "$tmp/ba" "$tmp/ba")" "" +assert_eq off-one "$(measure_byte_offsets "$tmp/ba" "$tmp/bb")" 3 +assert_eq off-shorter "$(measure_byte_offsets "$tmp/ba" "$tmp/bc")" "" +assert_fail off-missing measure_byte_offsets "$tmp/ba" "$tmp/absent" + +info "4: measure_floor_positions over every control pair" +shadow "$tmp/c1" 1000 "mysql:!!:1:::" +shadow "$tmp/c2" 2000 "mysql:!!:1:::" +shadow "$tmp/c3" 3000 "mysql:!!:1:::" +assert_eq floor-pair "$(measure_floor_positions text "$tmp/c1" "$tmp/c2")" "$(printf 'n1\no1')" +assert_eq floor-three "$(measure_floor_positions text "$tmp/c1" "$tmp/c2" "$tmp/c3")" "$(printf 'n1\no1')" +assert_eq floor-identical "$(measure_floor_positions text "$tmp/c1" "$tmp/c1")" "" +assert_eq floor-binary "$(measure_floor_positions binary "$tmp/ba" "$tmp/bb")" 3 + +info "5: measure_range" +assert_eq range-size "$(measure_range size "$tmp/c1" "$tmp/c2" "$tmp/c3")" "0 0" +printf '12345' > "$tmp/s5" +printf '123456' > "$tmp/s6" +printf '1' > "$tmp/s1" +assert_eq range-grow "$(measure_range size "$tmp/s5" "$tmp/s6")" "-1 1" +assert_eq range-both "$(measure_range size "$tmp/s5" "$tmp/s6" "$tmp/s1")" "-5 5" +assert_eq range-bytes "$(measure_range bytes "$tmp/ba" "$tmp/bb")" "1 1" + +info "6: measure_mask_at and measure_mask_admits" +m=$(measure_mask_at 1 "$tmp/c1" "$tmp/c2" "$tmp/c3") +assert_eq mask-prefix "$(cut -d"$(printf '\001')" -f1 <<<"$m")" "# written " +assert_eq mask-suffix "$(cut -d"$(printf '\001')" -f2 <<<"$m")" "000" +printf 'aaaZZZ\n' > "$tmp/p1"; printf 'aaaYYY\n' > "$tmp/p2" +assert_eq mask-prefixonly "$(measure_mask_at 1 "$tmp/p1" "$tmp/p2")" "$(printf 'aaa\001')" +printf 'ZZZaaa\n' > "$tmp/q1"; printf 'YYYaaa\n' > "$tmp/q2" +assert_eq mask-suffixonly "$(measure_mask_at 1 "$tmp/q1" "$tmp/q2")" "$(printf '\001aaa')" +printf 'abc\n' > "$tmp/r1"; printf 'xyz\n' > "$tmp/r2" +assert_eq mask-neither "$(measure_mask_at 1 "$tmp/r1" "$tmp/r2")" "$(printf '\001')" +assert_ok admits-yes measure_mask_admits "# written 4000" "# written " "000" +assert_fail admits-prefix measure_mask_admits "root:x:0" "# written " "" +assert_fail admits-suffix measure_mask_admits "# written 4242" "# written " "000" +assert_fail admits-short measure_mask_admits "# wr" "# written " "" +assert_fail admits-emptymask measure_mask_admits "anything at all" "" "" + +# --------------------------------------------------------------------------- +# the verdicts, and the four false negatives that made this rewrite necessary +# --------------------------------------------------------------------------- + +info "7: a clock that moves in every build is noise" +shadow "$tmp/u-clock" 4000 "mysql:!!:1:::" +v=$(verdict clock noise "$tmp/u-clock" "$tmp/c1" "$tmp/c2" "$tmp/c3") +assert_grep clock-why "mask" "$v" + +info "8: an account added where the control pair also appends is NOT noise" +# the control pair appends a different account at the same line number, so +# the position cancels; only the content tells them apart +shadow "$tmp/ac1" 1000 "mysql:!!:1:::" "alice:!!:20000:0:99999:7:::" +shadow "$tmp/ac2" 2000 "mysql:!!:1:::" "bob:!!:20000:0:99999:7:::" +# shellcheck disable=SC2016 # a literal crypt hash, not an expansion +shadow "$tmp/au" 3000 "mysql:!!:1:::" 'mallory:$y$REALPASSWORDHASH:20000:::' +v=$(verdict added-account real "$tmp/au" "$tmp/ac1" "$tmp/ac2") +assert_grep added-account-why "line 4" "$v" + +info "9: a root password set on the line the floor rewrites is NOT noise" +printf '# written 1000\nroot:!:20000:0:99999:7:::\n' > "$tmp/rc1" +printf '# written 2000\nroot:!:20000:0:99999:7:::\n' > "$tmp/rc2" +# shellcheck disable=SC2016 # a literal crypt hash, not an expansion +printf 'root:$y$REALHASH:20000:0:99999:7:::\nroot:!:20000:0:99999:7:::\n' > "$tmp/ru" +verdict root-password real "$tmp/ru" "$tmp/rc1" "$tmp/rc2" >/dev/null + +info "10: a binary coinciding with the floor is overlapping, never noise" +bin() { printf 'HDR\0%s\0%s' "$2" "$3" > "$1"; } +bin "$tmp/kc1" AAAAAAAA root@localhost +bin "$tmp/kc2" BBBBBBBB root@localhost +bin "$tmp/ku" CCCCCCCC root@localhost +v=$(verdict binary-coincide overlapping "$tmp/ku" "$tmp/kc1" "$tmp/kc2") +assert_grep binary-why "byte" "$v" +bin "$tmp/ku2" AAAAAAAA root@localhosX +verdict binary-outside real "$tmp/ku2" "$tmp/kc1" "$tmp/kc2" >/dev/null +# every control pair differs at the same eight bytes, so a unit differing at +# only four of them is inside the floor by position and outside it by +# magnitude, which is the only handle a binary offers +bin "$tmp/kc3" CCCCCCCC root@localhost +bin "$tmp/ku3" AAAADDDD root@localhost +v=$(verdict binary-count real "$tmp/ku3" "$tmp/kc1" "$tmp/kc2" "$tmp/kc3") +assert_grep binary-count-why "bytes differ where the controls" "$v" + +info "11: a length change the floor does not show is NOT noise" +head -c 4096 /dev/zero | tr '\0' 'x' > "$tmp/lc1" +{ head -c 4096 /dev/zero | tr '\0' 'x'; printf 'y'; } > "$tmp/lc2" +head -c 512 /dev/zero | tr '\0' 'x' > "$tmp/lu" +v=$(verdict length-outside real "$tmp/lu" "$tmp/lc1" "$tmp/lc2") +assert_grep length-why "size" "$v" + +info "12: a mask that explains nothing never yields noise" +printf 'abc\n' > "$tmp/ec1"; printf 'xyz\n' > "$tmp/ec2"; printf 'pqr\n' > "$tmp/eu" +verdict empty-mask overlapping "$tmp/eu" "$tmp/ec1" "$tmp/ec2" >/dev/null +# the NUL-free, newline-free blob that used to collapse to one line +head -c 1500 /dev/urandom | tr -d '\0\n' > "$tmp/bl1" +head -c 1500 /dev/urandom | tr -d '\0\n' > "$tmp/bl2" +head -c 1500 /dev/urandom | tr -d '\0\n' > "$tmp/blu" +v=$(measure_verdict "$tmp/blu" "$tmp/bl1" "$tmp/bl2") +[[ "$(cut -f1 <<<"$v")" != noise ]] || fatal "blob: a newline-free blob came back as noise" + +info "13: identical controls leave no floor, so any difference is real" +verdict no-floor real "$tmp/u-clock" "$tmp/c1" "$tmp/c1" >/dev/null + +info "14: a unit identical to the control has nothing to report" +v=$(verdict identical none "$tmp/c1" "$tmp/c1" "$tmp/c2") +assert_grep identical-why "identical" "$v" + +info "15: unsampled" +verdict unsampled-unit unsampled "$tmp/absent" "$tmp/c1" "$tmp/c2" >/dev/null +verdict unsampled-control unsampled "$tmp/c1" "$tmp/absent" "$tmp/c2" >/dev/null +assert_fail verdict-nocontrol measure_verdict "$tmp/c1" + +info "16: too many differing positions to mask" +saved=$MEASURE_POSITION_MAX +MEASURE_POSITION_MAX=2 +seq 1 20 | sed 's/$/ a/' > "$tmp/mc1" +seq 1 20 | sed 's/$/ b/' > "$tmp/mc2" +seq 1 20 | sed 's/$/ c/' > "$tmp/mu" +v=$(verdict position-cap overlapping "$tmp/mu" "$tmp/mc1" "$tmp/mc2") +assert_grep position-cap-why "too many" "$v" +MEASURE_POSITION_MAX=$saved + +info "17: measure_cleared_reason" +cat > "$tmp/cleared" <<'EOF' +# a waiver is committed, and the report prints it +./var/lib/mysql/ibdata1 InnoDB creation stamps, checked by hand on 2026-09-28 +EOF +assert_eq cleared-hit "$(measure_cleared_reason "$tmp/cleared" ./var/lib/mysql/ibdata1)" \ + "InnoDB creation stamps, checked by hand on 2026-09-28" +assert_eq cleared-miss "$(measure_cleared_reason "$tmp/cleared" ./etc/shadow)" "" +assert_eq cleared-nofile "$(measure_cleared_reason "$tmp/absent" ./etc/shadow)" "" +# a glob waiver, because a real data directory is 173 files and one sentence +printf './var/lib/postgresql/**\tinitdb output, read on 2026-09-28\n' >> "$tmp/cleared" +assert_eq cleared-glob "$(measure_cleared_reason "$tmp/cleared" ./var/lib/postgresql/17/main/PG_VERSION)" \ + "initdb output, read on 2026-09-28" +assert_eq cleared-glob-miss "$(measure_cleared_reason "$tmp/cleared" ./var/lib/other/x)" "" + +info "18: measure_evidence is printed for every verdict, noise included" +e=$(measure_evidence ./etc/shadow "$tmp/u-clock" "$tmp/c1" "$tmp/c2" "$tmp/c3") +assert_grep evidence-noise-unit "control vs unit" "$e" +assert_grep evidence-noise-floor "control vs control" "$e" +assert_grep evidence-noise-content "written" "$e" +e=$(measure_evidence ./k "$tmp/ku" "$tmp/kc1" "$tmp/kc2") +assert_grep evidence-binary "offset" "$e" +assert_grep evidence-binary-size "size" "$e" +e=$(measure_evidence ./x "$tmp/absent" "$tmp/c1" "$tmp/c2") +assert_grep evidence-unsampled "not sampled" "$e" + +echo +info "all tests passed" diff --git a/tests/coverage.sh b/tests/coverage.sh index 0bb49d0..173aa76 100755 --- a/tests/coverage.sh +++ b/tests/coverage.sh @@ -3,7 +3,9 @@ # bin/layer-lib and bt-layer under tests/layer # bin/aplinfo-lib and bt-aplinfo under tests/aplinfo # bin/signature-lib and bin/generate-signature under tests/signature -# bin/layer-measure-lib and bt-layer-measure under tests/measure +# bin/layer-measure-lib, bin/layer-report-lib and bt-layer-measure under +# tests/measure +# bin/layer-compare-lib under tests/compare # Exits 1 when a measured file is below the threshold (default 95), 2 when a # tool is missing. # @@ -31,9 +33,9 @@ for tool in kcov zstd git gpg; do done report="${COVERAGE_DIR:-$(mktemp -d)}" -measured=/bin/layer-lib,/bt-layer,/bin/aplinfo-lib,/bt-aplinfo,/bin/signature-lib,/bin/generate-signature,/bin/layer-measure-lib,/bt-layer-measure +measured=/bin/layer-lib,/bt-layer,/bin/aplinfo-lib,/bt-aplinfo,/bin/signature-lib,/bin/generate-signature,/bin/layer-measure-lib,/bin/layer-compare-lib,/bin/layer-report-lib,/bt-layer-measure skipped=/tests/,/scratch-bt/bin/generate-signature -for suite in layer aplinfo signature measure; do +for suite in layer aplinfo signature measure compare; do kcov --include-pattern="$measured" --exclude-pattern="$skipped" \ "$report" "$here/$suite" done diff --git a/tests/layer b/tests/layer index 0b4d2fd..d9f0d5d 100755 --- a/tests/layer +++ b/tests/layer @@ -41,6 +41,10 @@ # version than the parent applied; a unit whose conf fab would skip; a # parent manifest written before the units field existed. +# pipefail as the scripts under test set it: a library function that +# inherits a pipeline's status must fail here and not in production. +set -o pipefail + fatal() { echo "FATAL [$(basename "$0")]: $*" 1>&2; exit 1; } info() { echo "INFO [$(basename "$0")]: $*"; } diff --git a/tests/measure b/tests/measure index af840da..69cd246 100755 --- a/tests/measure +++ b/tests/measure @@ -46,11 +46,13 @@ EOF exit 1 } -[[ "${1:-}" != "-h" ]] && [[ "${1:-}" != "--help" ]] || usage +if [[ "${1:-}" == "-h" ]] || [[ "${1:-}" == "--help" ]]; then usage; fi BT=$(dirname "$(dirname "$(readlink -f "$0")")") export BT source "$BT/bin/layer-measure-lib" +source "$BT/bin/layer-compare-lib" +source "$BT/bin/layer-report-lib" tmp=$(mktemp -d) trap 'rm -rf "$tmp"' EXIT @@ -132,6 +134,9 @@ make_rootfs() { printf '\x00\x00' printf '%s' "$account" } > "$dir/var/lib/mysql/mysql/global_priv.MAD" + # a state file whose bytes never change, so a mode-only difference can + # be told apart from a content one + echo "fixed" > "$dir/var/lib/mysql/fixed.cnf" # a state file large enough to run past a small sample cap head -c 512 /dev/zero | tr '\0' "$priv" > "$dir/var/lib/mysql/ibdata1" cat > "$dir/var/lib/dpkg/status" < "$fix/u2/etc/keel/component.conf" rm "$fix/u2/usr/lib/python3/setuptools/lorem" ln -s "text/other file.txt" "$fix/u2/usr/lib/python3/setuptools/lorem" @@ -291,7 +300,7 @@ s=$tmp/sample mkdir -p "$s" measure_tree_record "$fix/c1" > "$s/tree.tsv" measure_state_sample "$fix/c1" "$s/tree.tsv" "$s" 100000 "$globs" -assert_eq sample-count "$(wc -l < "$s/state.tsv")" 3 +assert_eq sample-count "$(wc -l < "$s/state.tsv")" 4 assert_eq sample-shadow "$(cat "$s/state/etc/shadow")" "$(cat "$fix/c1/etc/shadow")" assert_grep sample-flag " sampled " "$(cat "$s/state.tsv")" s2=$tmp/sample-cap @@ -318,56 +327,12 @@ assert_eq value-absent "$(measure_value "$a" p9)" "" assert_eq count-lines "$(measure_count "$a")" 3 assert_eq count-missing "$(measure_count "$tmp/absent")" 0 -info "9: measure_is_text" -assert_ok text-yes measure_is_text "$fix/c1/etc/shadow" -assert_fail text-no measure_is_text "$fix/c1/var/lib/mysql/mysql/global_priv.MAD" -assert_fail text-missing measure_is_text "$tmp/absent" - -info "10: measure_signature and measure_set_minus" -printf 'one\ntwo\nthree\n' > "$tmp/t1" -printf 'one\nTWO\nthree\n' > "$tmp/t2" -printf 'one\ntwo\nthree\nfour\n' > "$tmp/t3" -assert_eq sig-text-same "$(measure_signature "$tmp/t1" "$tmp/t1")" "" -assert_eq sig-text-changed "$(measure_signature "$tmp/t1" "$tmp/t2")" "$(printf 'n2\no2')" -assert_eq sig-text-added "$(measure_signature "$tmp/t1" "$tmp/t3")" "n4" -printf 'AAAA' > "$tmp/b1" -printf 'AABA' > "$tmp/b2" -printf 'AAAAA' > "$tmp/b3" -printf '\x00\x00\x00\x00' > "$tmp/b0" -assert_eq sig-bin-same "$(measure_signature "$tmp/b0" "$tmp/b0")" "" -assert_eq sig-bin-offset "$(measure_signature "$tmp/b0" "$(printf %s "$tmp/b0"; :)")" "" -cp "$tmp/b0" "$tmp/b0x"; printf '\x01' | dd of="$tmp/b0x" bs=1 seek=2 conv=notrunc status=none -assert_eq sig-bin-at "$(measure_signature "$tmp/b0" "$tmp/b0x")" "@3" -head -c 5 /dev/zero > "$tmp/b0y" -assert_eq sig-bin-len "$(measure_signature "$tmp/b0" "$tmp/b0y")" "len" -assert_fail sig-missing measure_signature "$tmp/b0" "$tmp/absent" -assert_eq minus-empty "$(measure_set_minus "$(printf 'a\nb')" "$(printf 'a\nb\nc')")" "" -assert_eq minus-residue "$(measure_set_minus "$(printf 'a\nb')" "$(printf 'a')")" "b" -assert_eq minus-nothing "$(measure_set_minus "" "$(printf 'a')")" "" - -info "10b: measure_nature_list" -assert_eq nature-empty "$(measure_nature_list "")" "" -assert_eq nature-all "$(measure_nature_list "$(printf 'len\no3\nn4\n@7')")" \ - "the length, other line 4, control line 3, byte 7" -assert_eq nature-unknown "$(measure_nature_list "wat")" "wat" -assert_grep nature-overflow "and 4 more" \ - "$(measure_nature_list "$(seq 1 12 | sed 's/^/@/')")" - -info "10b2: measure_print_set caps a long list" -saved=$MEASURE_PRINT_MAX -MEASURE_PRINT_MAX=2 -assert_grep print-set-capped "and 2 more" "$(measure_print_set paths "$(printf 'a\nb\nc\nd')")" -assert_eq print-set-empty "$(measure_print_set paths "")" "" -MEASURE_PRINT_MAX=$saved - -info "10c: measure_lines never splits a path on a space" -assert_eq lines-space "$(measure_lines "> " "./a b/c d")" "> ./a b/c d" -assert_eq lines-empty "$(measure_lines "> " "")" "" - -info "11: captures" +info "9: captures, with three controls so the floor is a union" +export SOURCE_DATE_EPOCH=1700000000 capture c1 c1 > "$tmp/cap-c1.log" -capture u1 u1 > /dev/null capture c2 c2 > /dev/null +capture c3 c3 > /dev/null +capture u1 u1 > /dev/null capture u2 u2 > /dev/null assert_grep capture-summary "2 packages" "$(cat "$tmp/cap-c1.log")" assert_ok capture-check measure_capture_check "$caps/c1" @@ -376,53 +341,40 @@ mkdir -p "$tmp/half"; : > "$tmp/half/tree.tsv" assert_fail capture-check-half measure_capture_check "$tmp/half" assert_eq meta-layer "$(measure_meta_get "$caps/c1" layer)" mariadb assert_eq meta-parent "$(measure_meta_get "$caps/c1" parent)" core +assert_eq meta-epoch "$(measure_meta_get "$caps/c1" epoch)" 1700000000 assert_eq meta-absent "$(measure_meta_get "$caps/c1" nothing)" "" assert_eq sample-path "$(measure_sample_path "$caps/c1" "./etc/shadow")" \ "$caps/c1/state/etc/shadow" - -info "12: measure_state_verdict, text" -v=$(measure_state_verdict "$caps/c1" "$caps/u1" "$caps/c2" "./etc/shadow") -assert_eq verdict-text-noise "$(cut -f1 <<<"$v")" noise -assert_grep verdict-text-noise-why "control pair" "$v" -v=$(measure_state_verdict "$caps/c1" "$caps/u2" "$caps/c2" "./etc/shadow") -assert_eq verdict-text-real "$(cut -f1 <<<"$v")" real -assert_grep verdict-text-real-why "line" "$v" - -info "13: measure_state_verdict, binary" -v=$(measure_state_verdict "$caps/c1" "$caps/u1" "$caps/c2" \ - "./var/lib/mysql/mysql/global_priv.MAD") -assert_eq verdict-bin-noise "$(cut -f1 <<<"$v")" noise -v=$(measure_state_verdict "$caps/c1" "$caps/u2" "$caps/c2" \ - "./var/lib/mysql/mysql/global_priv.MAD") -assert_eq verdict-bin-real "$(cut -f1 <<<"$v")" real -assert_grep verdict-bin-real-why "byte" "$v" - -info "14: measure_state_verdict, not sampled and absent" -capture c1 s1 --state-max-bytes 8 > /dev/null -capture u1 s2 --state-max-bytes 8 > /dev/null -capture c2 s3 --state-max-bytes 8 > /dev/null -v=$(measure_state_verdict "$caps/s1" "$caps/s2" "$caps/s3" "./var/lib/mysql/ibdata1") -assert_eq verdict-unsampled "$(cut -f1 <<<"$v")" unsampled -assert_grep verdict-unsampled-why "not sampled" "$v" -v=$(measure_state_verdict "$caps/c1" "$caps/u2" "$caps/c2" "./var/lib/mysql/absent") -assert_eq verdict-absent "$(cut -f1 <<<"$v")" unsampled - -info "15: measure_state_evidence" -e=$(measure_state_evidence "$caps/c1" "$caps/u2" "$caps/c2" "./etc/shadow") -assert_grep evidence-text-unit "control vs unit" "$e" -assert_grep evidence-text-noise "control vs control" "$e" -assert_grep evidence-text-row "mysql" "$e" -e=$(measure_state_evidence "$caps/c1" "$caps/u2" "$caps/c2" \ - "./var/lib/mysql/mysql/global_priv.MAD") -assert_grep evidence-bin "offset" "$e" -e=$(measure_state_evidence "$caps/s1" "$caps/s2" "$caps/s3" "./var/lib/mysql/ibdata1") -assert_grep evidence-unsampled "not sampled" "$e" - -# --------------------------------------------------------------------------- -# the executable -# --------------------------------------------------------------------------- - -info "16: usage" +# the list travels with the capture, which is what makes finding 2 unrepeatable +[[ -f "$caps/c1/state-paths.txt" ]] || fatal "the capture kept no copy of the state path list" +assert_eq state-paths-copy "$(cat "$caps/c1/state-paths.txt")" "$(cat "$patterns")" +assert_eq state-paths-digest "$(measure_meta_get "$caps/c1" state_paths_sha256)" \ + "$(sha256sum < "$patterns" | cut -d' ' -f1)" +assert_eq capture-mode "$(stat -c %a "$caps/c1")" 700 + +info "10: measure_preconditions" +measure_preconditions "$caps" u1 c1 c2 c3 || fatal "pre-ok: should have passed" +assert_fail pre-missing measure_preconditions "$caps" u1 c1 absent +assert_fail pre-duplicate measure_preconditions "$caps" u1 c1 c1 +assert_fail pre-same-as-unit measure_preconditions "$caps" c1 c1 c2 +SOURCE_DATE_EPOCH=1800000000 capture c1 other-epoch > /dev/null +assert_fail pre-epoch measure_preconditions "$caps" u1 c1 other-epoch +capture c1 noepoch-a > /dev/null +capture c2 noepoch-b > /dev/null +capture c3 noepoch-c > /dev/null +for t in noepoch-a noepoch-b noepoch-c; do + sed -i 's/^epoch\t.*/epoch\tnone/' "$caps/$t/meta.tsv" +done +assert_fail pre-epoch-none measure_preconditions "$caps" noepoch-a noepoch-b noepoch-c +"$M" capture --dir "$caps" --layer mariadb --parent core --tag other-parent >/dev/null 2>&1 || true +echo c1 > "$layers/NEXT" +"$M" capture --dir "$caps" --layer mariadb --tag no-parent > /dev/null +assert_fail pre-parent measure_preconditions "$caps" u1 c1 no-parent +capture c2 tampered > /dev/null +echo "./extra/**" >> "$caps/tampered/state-paths.txt" +assert_fail pre-list-differs measure_preconditions "$caps" u1 c1 tampered + +info "11: usage" assert_status help 0 "$M" --help assert_status help-short 0 "$M" -h assert_status no-subcommand 2 "$M" @@ -430,9 +382,11 @@ assert_status unknown-subcommand 2 "$M" frobnicate assert_status unknown-option 2 "$M" capture --nope assert_status value-missing 2 "$M" capture --dir assert_status two-subcommands 2 "$M" capture compare +assert_status attr-no-again 2 "$M" attribute --dir "$caps" --control c1 --unit u1 assert_grep help-text "attribute" "$("$M" --help)" +assert_grep help-cleared "cleared" "$("$M" --help)" -info "17: capture errors" +info "12: capture errors" assert_status capture-nodir 2 "$M" capture --layer mariadb --tag t assert_status capture-nolayer 2 "$M" capture --dir "$caps" --tag t assert_status capture-notag 2 "$M" capture --dir "$caps" --layer mariadb @@ -440,11 +394,9 @@ assert_status capture-unknown-opt 2 "$M" capture --dir "$caps" --layer m --tag t assert_status capture-badtag 2 "$M" capture --dir "$caps" --layer m --tag "../escape" assert_status capture-badcap 2 "$M" capture --dir "$caps" --layer m --tag t \ --state-max-bytes plenty -assert_status capture-nopatterns 3 env MEASURE_STATE_PATHS=$tmp/absent \ - "$M" capture --dir "$caps" --layer m --tag t -assert_status attribute-nopatterns 3 env MEASURE_STATE_PATHS=$tmp/absent \ - "$M" attribute --dir "$caps" --control c1 --unit u1 --control-again c2 assert_status capture-exists 2 "$M" capture --dir "$caps" --layer m --tag c1 +assert_status capture-nopatterns 3 env "MEASURE_STATE_PATHS=$tmp/absent" \ + "$M" capture --dir "$caps" --layer m --tag t echo FAIL > "$layers/NEXT" assert_status capture-buildfail 3 "$M" capture --dir "$caps" --layer m --tag bad echo NOROOTFS > "$layers/NEXT" @@ -454,54 +406,150 @@ r=$("$M" capture --dir "$caps" --layer m --tag nomanifest) assert_grep capture-nomanifest "no layer manifest" "$r" [[ ! -e "$caps/nomanifest/manifest.txt" ]] || fatal "capture-nomanifest: copied one anyway" [[ -f "$caps/c1/manifest.txt" ]] || fatal "capture: the layer manifest was not kept" +assert_status capture-nobtbuilds 3 env -u MEASURE_LAYERS_DIR -u BT_BUILDS \ + "$M" capture --dir "$caps" --layer m --tag nobtbuilds -info "18: compare" +info "13: compare" assert_status compare-missing 3 "$M" compare --dir "$caps" --a c1 --b absent assert_status compare-args 2 "$M" compare --dir "$caps" --a c1 r=$("$M" compare --dir "$caps" --a c1 --b c1) || fatal "compare-same: exit $?" assert_grep compare-identical "identical" "$r" -r=$("$M" compare --dir "$caps" --a c1 --b u2 || true) assert_status compare-differs 1 "$M" compare --dir "$caps" --a c1 --b u2 +r=$("$M" compare --dir "$caps" --a c1 --b u2 || true) assert_grep compare-file "./etc/keel/component.conf" "$r" assert_grep compare-pkg "coreutils" "$r" assert_grep compare-link "lorem" "$r" assert_grep compare-added "./tmp/new-file" "$r" -info "19: attribute, a clean unit" -assert_status attr-args 2 "$M" attribute --dir "$caps" --control c1 --unit u1 +# --------------------------------------------------------------------------- +# attribute +# --------------------------------------------------------------------------- + +attr() { "$M" attribute --dir "$caps" --unit "$1" --control c1 \ + --control-again c2 --control-again c3 "${@:2}"; } +suspects() { printf '%s\n' "$BT/share/layer-state-suspect"; } + +info "14: a clean unit still fails while two binaries are unexplained" assert_status attr-missing 3 "$M" attribute --dir "$caps" --control c1 --unit u1 \ --control-again absent -r=$("$M" attribute --dir "$caps" --control c1 --unit u1 --control-again c2) -assert_grep attr-pass "verdict: PASS" "$r" -assert_grep attr-zero "attributable differences" "$r" +assert_status attr-clean-uncleared 1 attr u1 +r=$(attr u1 || true) +assert_grep attr-pairs "control pairs 3" "$r" +assert_grep attr-perpair "attributable file differences if a single control pair" "$r" assert_eq attr-zero-count \ - "$(awk -F: '/attributable differences/ { gsub(/ /, "", $2); print $2 }' <<<"$r")" 0 + "$(awk -F: '/^attributable differences/ { gsub(/ /, "", $2); print $2 }' <<<"$r")" 0 assert_grep attr-command "bt-layer-measure attribute" "$r" -assert_grep attr-floor "noise floor" "$r" -# the state paths inside the floor are named and classified, not subtracted -assert_grep attr-state-named "./etc/shadow" "$r" -assert_grep attr-state-noise "noise" "$r" - -info "20: attribute, a dirty unit" -assert_status attr-dirty 1 "$M" attribute --dir "$caps" --control c1 --unit u2 \ - --control-again c2 -r=$("$M" attribute --dir "$caps" --control c1 --unit u2 --control-again c2 || true) +assert_grep attr-command-again "control-again c3" "$r" +# the text state file is explained by the mask, the binaries are not +assert_grep attr-shadow-noise "noise .*/etc/shadow" "$r" +assert_grep attr-binary-overlapping "overlapping .*global_priv" "$r" assert_grep attr-fail "verdict: FAIL" "$r" +# and the evidence is printed even for the path called noise, which is the +# whole reason the first implementation could not be trusted +assert_grep attr-evidence-for-noise "control vs unit" "$r" +assert_grep attr-subtracted-named "./var/log/alternatives.log" "$r" + +info "15: a waiver in writing clears an overlapping path and is printed" +cat > "$tmp/cleared" <<'EOF' +# checked by hand: both are install-time database state +./var/lib/mysql/mysql/global_priv.MAD creation stamps, read byte by byte on 2026-09-28 +./var/lib/mysql/ibdata1 InnoDB page header, read byte by byte on 2026-09-28 +EOF +r=$(attr u1 --cleared "$tmp/cleared") +assert_grep attr-pass "verdict: PASS" "$r" +assert_grep attr-waiver-printed "waived in writing" "$r" +assert_grep attr-waiver-reason "read byte by byte" "$r" +assert_status attr-pass-status 0 attr u1 --cleared "$tmp/cleared" + +info "16: a dirty unit, and a real difference inside the floor" +assert_status attr-dirty 1 attr u2 --cleared "$tmp/cleared" +r=$(attr u2 --cleared "$tmp/cleared" || true) +assert_grep attr-dirty-fail "verdict: FAIL" "$r" assert_grep attr-attributable "./etc/keel/component.conf" "$r" assert_grep attr-added "./tmp/new-file" "$r" assert_grep attr-pkgdiff "coreutils" "$r" assert_grep attr-linkdiff "lorem" "$r" -# the point of the whole exercise: a real difference inside the noise floor -assert_grep attr-real-state "real" "$r" -assert_grep attr-real-shadow "./etc/shadow" "$r" +# the account row is gone from a text state file whose line 1 moves in every +# build: a positional comparison called this noise +assert_grep attr-real-shadow "real .*/etc/shadow" "$r" assert_grep attr-real-row "mysql" "$r" +# a waiver cannot wave away a real difference, only an overlapping one +assert_grep attr-real-binary "real .*global_priv" "$r" -info "21: attribute, blind because nothing was sampled" -assert_status attr-unsampled 1 "$M" attribute --dir "$caps" --control s1 --unit s2 \ - --control-again s3 -r=$("$M" attribute --dir "$caps" --control s1 --unit s2 --control-again s3 || true) +info "17: blind because nothing was sampled" +capture c1 s1 --state-max-bytes 8 > /dev/null +capture c2 s2 --state-max-bytes 8 > /dev/null +capture c3 s3 --state-max-bytes 8 > /dev/null +capture u1 su --state-max-bytes 8 > /dev/null +r=$("$M" attribute --dir "$caps" --unit su --control s1 --control-again s2 \ + --control-again s3 || true) assert_grep attr-unsampled-why "not sampled" "$r" assert_grep attr-unsampled-fail "verdict: FAIL" "$r" +assert_status attr-unsampled-status 1 "$M" attribute --dir "$caps" --unit su \ + --control s1 --control-again s2 --control-again s3 + +info "18: the blindness cannot be switched off at attribute time" +# a one line list that matches nothing would have turned the run above into a +# PASS with a provenanced looking digest; attribute reads the captures' own +# copies and never the environment +echo './nothing/**' > "$tmp/narrow" +r=$(MEASURE_STATE_PATHS="$tmp/narrow" "$M" attribute --dir "$caps" --unit su \ + --control s1 --control-again s2 --control-again s3 || true) +assert_grep attr-nosubst-fail "verdict: FAIL" "$r" +assert_grep attr-nosubst-still "not sampled" "$r" +assert_status attr-nosubst-status 1 env "MEASURE_STATE_PATHS=$tmp/narrow" \ + "$M" attribute --dir "$caps" --unit su --control s1 --control-again s2 \ + --control-again s3 + +info "19: a subtracted path whose name looks like state fails the run" +echo '*alternatives*' > "$tmp/suspect" +r=$(MEASURE_SUSPECT_PATHS="$tmp/suspect" attr u1 --cleared "$tmp/cleared" || true) +assert_grep suspect-named "./var/log/alternatives.log" "$r" +assert_grep suspect-told "add them to" "$r" +assert_grep suspect-fail "verdict: FAIL" "$r" +assert_status suspect-status 1 env "MEASURE_SUSPECT_PATHS=$tmp/suspect" \ + "$M" attribute --dir "$caps" --unit u1 --control c1 --control-again c2 \ + --control-again c3 --cleared "$tmp/cleared" + +info "20: a state path whose bytes are identical and whose mode moved" +capture u3 u3 > /dev/null +r=$("$M" attribute --dir "$caps" --unit u3 --control c1 --control-again c2 \ + --control-again c3 --cleared "$tmp/cleared" || true) +assert_grep mode-only "unchanged 1" "$r" +assert_grep mode-only-path "fixed.cnf" "$r" + +info "21: the evidence is abbreviated past the cap, never dropped" +saved_ev=$MEASURE_EVIDENCE_PATHS +MEASURE_EVIDENCE_PATHS=1 +r=$(measure_attribute_report "$caps" u1 /nonexistent "$(suspects)" c1 c2 c3 || true) +assert_grep evidence-abbrev "more abbreviated to their summary line" "$r" +MEASURE_EVIDENCE_PATHS=$saved_ev + +info "22: the guards that must never fire" +globs=$(measure_state_patterns "$patterns") +measure_verdict() { printf '\t\n'; } +r=$(measure_state_section "$caps" u1 /nonexistent "$(suspects)" "$globs" c1 c2 c3 2>&1 || true) +assert_grep guard-noverdict "no verdict for" "$r" +measure_verdict() { printf 'invented\tnot one of the five\n'; } +r=$(measure_state_section "$caps" u1 /nonexistent "$(suspects)" "$globs" c1 c2 c3 2>&1 || true) +assert_grep guard-totals "do not add up" "$r" +unset -f measure_verdict +source "$BT/bin/layer-compare-lib" + +info "23: measure_nature_list" +assert_eq nature-empty "$(measure_nature_list "")" "" +assert_eq nature-all "$(measure_nature_list "$(printf 'len\no3\nn4\n@7')")" \ + "the length, other line 4, control line 3, byte 7" +assert_eq nature-unknown "$(measure_nature_list "wat")" "wat" +assert_grep nature-overflow "and 4 more" \ + "$(measure_nature_list "$(seq 1 12 | sed 's/^/@/')")" + +info "24: measure_print_set caps a long list" +saved=$MEASURE_PRINT_MAX +MEASURE_PRINT_MAX=2 +assert_grep print-set-capped "and 2 more" "$(measure_print_set paths "$(printf 'a\nb\nc\nd')")" +assert_eq print-set-empty "$(measure_print_set paths "")" "" +MEASURE_PRINT_MAX=$saved echo info "all tests passed" diff --git a/tests/signature b/tests/signature index ead9af3..355b355 100755 --- a/tests/signature +++ b/tests/signature @@ -26,6 +26,10 @@ # file behind; and the upstream key on an upstream artifact, which must # produce the text of tests/signature-reference byte for byte. +# pipefail as the scripts under test set it: a library function that +# inherits a pipeline's status must fail here and not in production. +set -o pipefail + fatal() { echo "FATAL [$(basename "$0")]: $*" 1>&2; exit 1; } info() { echo "INFO [$(basename "$0")]: $*"; } @@ -240,7 +244,8 @@ export RELEASE=debian/trixie unset BT_GPGKEY BT_SIGN_KEY_FILE BT_SIGN_KEY_EMAIL BT_SIGN_KEY_URL BT_SIGN_KEY_FPR info "6: usage and early failures" -"$GS" --help | grep -q '^Syntax:' || fatal "help text missing" +help_out=$("$GS" --help || true) # usage exits non-zero, and pipefail sees it +grep -q '^Syntax:' <<<"$help_out" || fatal "help text missing" expect_fail no-args "$GS" expect_fail two-files "$GS" a b expect_fail unknown-option "$GS" --sign-everything file From 662ec1a4092ba25d5a036fb956c1787a90ade400 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Mon, 28 Sep 2026 06:11:01 +0000 Subject: [PATCH 3/9] fix: print the bytes in capture order, and run the measurement for real The first real use of this, four captures of mariadb on the build host, found one thing missing from the evidence and confirmed the rest. For a binary the block listed the offsets that differ and not the bytes at them, which is not enough to settle anything by hand. It now prints a window from every capture at the first differing offset, in capture order, and that turns an unreadable list into a finding: for mysql/global_priv.MAI the value reads b9fe1a, b9fec6, b9ff64, ba0004 down the four captures, so a reader sees an Aria header clock that carried into the byte above while three control builds moved only its low bytes. measure_is_text also refuses a file carrying control bytes other than tab, newline and carriage return. Being wrong about text is safe in one direction only: a binary masked line by line can come out as noise, while a text file compared as bytes comes out overlapping and has to be cleared by hand. It stays strict rather than paranoid on purpose, because mysql/user.frm is a TYPE=VIEW definition that really is text, and its only difference between builds is one timestamp line that the mask admits. The run itself, recorded in docs/layer-measure.md: 0 attributable differences, against each single control pair and against the union of all three; 167 state paths that differ, of which 102 noise, 33 overlapping, 32 real, 0 not sampled; 6 subtracted paths, all named, none state shaped. The four the review asked about were checked against the bytes one at a time and all four say what the bytes say: global_priv.MAD and .frm overlapping at the offsets every control also moves, .MAI real because its counter carried where no control's did, and user.frm noise on a timestamp line. Coverage: 204, 216, 230 and 70 lines, all four files at 100 percent, gate unchanged at 99. --- COVERAGE.md | 2 +- bin/layer-compare-lib | 92 ++++++++++++++++++++++++++++++++++++++----- docs/layer-measure.md | 47 ++++++++++++++++++++++ tests/compare | 29 ++++++++++++++ 4 files changed, 160 insertions(+), 10 deletions(-) diff --git a/COVERAGE.md b/COVERAGE.md index 8e5a97e..8a597f6 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -132,7 +132,7 @@ measures ten files: | bin/signature-lib | 76 of 76 | 100 percent | | bin/generate-signature | 87 of 87 | 100 percent | | bin/layer-measure-lib | 204 of 204 | 100 percent | -| bin/layer-compare-lib | 189 of 189 | 100 percent | +| bin/layer-compare-lib | 216 of 216 | 100 percent | | bin/layer-report-lib | 230 of 230 | 100 percent | | bt-layer-measure | 70 of 70 | 100 percent | diff --git a/bin/layer-compare-lib b/bin/layer-compare-lib index bb37b5d..8de4cd0 100644 --- a/bin/layer-compare-lib +++ b/bin/layer-compare-lib @@ -40,9 +40,20 @@ MEASURE_TEXT_MAX_LINE=8192 # is reported as overlapping rather than guessed at. MEASURE_POSITION_MAX=200 +# Above this many differing bytes a binary is reported as overlapping without +# listing the offsets. A MariaDB ib_logfile0 is 100 MiB and two builds of it +# differ almost everywhere; materialising that list would put tens of +# millions of offsets into a shell variable. +MEASURE_OFFSET_MAX=20000 + # The field separator inside a mask, which cannot occur in a text line. MEASURE_MASK_SEP=$'\001' +# How many bytes of a window the binary evidence prints from each capture at +# the first differing offset. Eight covers a 32 or 64 bit counter, which is +# what most of these differences turn out to be. +MEASURE_WINDOW_BYTES=8 + # How many lines of a diff the evidence block prints per pair, and how many # paths get a full three-way diff before the rest are abbreviated to their # summary line. Every `real` path is always printed in full. Both are read @@ -57,7 +68,15 @@ MEASURE_EVIDENCE_PATHS=20 # --------------------------------------------------------------------------- # measure_is_text FILE -# No NUL, at least one newline, and no line long enough to be a blob. +# No NUL, no control byte other than tab, newline and carriage return, at +# least one newline, and no line long enough to be a blob. Being wrong here +# is only safe in one direction: a binary treated as text gets masked line by +# line and can come out `noise`, while a text file treated as binary comes out +# `overlapping` and has to be cleared by hand. So the test is strict. +# +# It is strict rather than paranoid on purpose: mysql/user.frm is a TYPE=VIEW +# definition, which really is a text file whose only difference between builds +# is a `timestamp=` line, and calling it binary would bury that in a waiver. measure_is_text() { local f=$1 if [[ ! -f "$f" ]]; then @@ -68,6 +87,7 @@ measure_is_text() { all=$(wc -c < "$f") stripped=$(tr -d '\0' < "$f" | wc -c) [[ "$all" == "$stripped" ]] || return 1 + [[ "$(tr -dc '\1-\10\13\14\16-\37\177' < "$f" | wc -c)" -eq 0 ]] || return 1 [[ "$(tr -dc '\n' < "$f" | wc -c)" -gt 0 ]] || return 1 [[ "$(wc -L < "$f")" -le "$MEASURE_TEXT_MAX_LINE" ]] || return 1 return 0 @@ -123,10 +143,18 @@ measure_byte_offsets() { measure_error "cannot compare $1 with $2: one of them is not a file" return 1 fi - cmp -l -- "$1" "$2" 2>/dev/null | awk '{ print $1 }' || true + cmp -l -- "$1" "$2" 2>/dev/null \ + | awk -v m="$MEASURE_OFFSET_MAX" 'NR > m { exit } { print $1 }' || true return 0 } +# measure_offsets_truncated A B +# True when the two files differ in at least MEASURE_OFFSET_MAX bytes, so the +# offset list above is a prefix and cannot be subtracted from anything. +measure_offsets_truncated() { + [[ "$(measure_byte_offsets "$1" "$2" | grep -c . || true)" -ge "$MEASURE_OFFSET_MAX" ]] +} + # measure_pair_positions KIND A B measure_pair_positions() { if [[ "$1" == text ]]; then @@ -269,6 +297,10 @@ measure_verdict() { if measure_is_text "$base" && measure_is_text "$unit"; then kind=text fi + if [[ "$kind" == binary ]]; then + measure_verdict_binary "$unit" "$@" + return 0 + fi local floor unit_pos outside n floor=$(measure_floor_positions "$kind" "$@") unit_pos=$(measure_pair_positions "$kind" "$base" "$unit") @@ -278,10 +310,6 @@ measure_verdict() { "differs where the controls do not vary: $(measure_nature_list "$outside")" return 0 fi - if [[ "$kind" == binary ]]; then - measure_verdict_binary "$unit" "$@" - return 0 - fi n=$(grep -c . <<<"$unit_pos" || true) if [[ "$n" -gt "$MEASURE_POSITION_MAX" ]]; then printf 'overlapping\t%s\n' \ @@ -328,11 +356,24 @@ measure_verdict_text() { measure_verdict_binary() { local unit=$1 base=$2 shift - local delta count dmin dmax cmin cmax - read -r dmin dmax < <(measure_range size "$@") - read -r cmin cmax < <(measure_range bytes "$@") + local delta count dmin dmax cmin cmax floor outside delta=$(( $(wc -c < "$unit") - $(wc -c < "$base") )) count=$(measure_byte_offsets "$base" "$unit" | grep -c . || true) + if measure_offsets_truncated "$base" "$unit"; then + printf 'overlapping\t%s\n' \ + "at least $MEASURE_OFFSET_MAX bytes differ, too many to compare offset by offset; size delta $delta, and the evidence names the first of them" + return 0 + fi + floor=$(measure_floor_positions binary "$@") + outside=$(measure_set_minus "$(measure_byte_offsets "$base" "$unit")" "$floor") + [[ -z "$outside" ]] || outside="@$outside" + if [[ -n "$outside" ]]; then + printf 'real\t%s\n' \ + "differs at bytes the controls do not vary at: $(measure_nature_list "${outside//$'\n'/$'\n@'}")" + return 0 + fi + read -r dmin dmax < <(measure_range size "$@") + read -r cmin cmax < <(measure_range bytes "$@") if [[ "$delta" -lt "$dmin" ]] || [[ "$delta" -gt "$dmax" ]]; then printf 'real\t%s\n' \ "size moves by $delta bytes where the controls move by $dmin to $dmax" @@ -373,9 +414,42 @@ measure_evidence() { measure_pair_evidence "$base" "$f" i=$((i + 1)) done + measure_window_evidence "$unit" "$@" + return 0 +} + +# measure_window_evidence UNIT CONTROL... +# For a binary, the bytes themselves at the first offset that differs, from +# every capture. This is what settles a difference by hand: read down the +# column and a value that climbs with capture time is a clock, while a value +# that changes only in the unit is the change being measured. Without it a +# reviewer gets a list of offsets and no way to tell those apart. +measure_window_evidence() { + local unit=$1 base=$2 + shift + measure_is_text "$base" && return 0 + local off f + off=$(measure_byte_offsets "$base" "$unit" | head -1) + [[ -n "$off" ]] || return 0 + printf ' the %s bytes at offset %s, in capture order:\n' \ + "$MEASURE_WINDOW_BYTES" "$off" + for f in "$@" "$unit"; do + printf ' %-12s %s\n' "$(measure_window_label "$f" "$unit")" \ + "$(dd if="$f" bs=1 skip=$((off - 1)) count="$MEASURE_WINDOW_BYTES" \ + status=none 2>/dev/null | od -An -tx1 | tr -s ' ' | tr -d '\n')" + done return 0 } +# measure_window_label FILE UNIT +measure_window_label() { + if [[ "$1" == "$2" ]]; then + echo unit + else + echo control + fi +} + # measure_pair_evidence A B measure_pair_evidence() { local a=$1 b=$2 diff --git a/docs/layer-measure.md b/docs/layer-measure.md index 9819482..5d69cc8 100644 --- a/docs/layer-measure.md +++ b/docs/layer-measure.md @@ -224,6 +224,53 @@ path list, and are distinct from one another. The note this implements says those must hold "or nothing the comparison prints means anything"; recording them and never checking them is the same as not recording them. +## What it said about mariadb, 2026-09-28 + +The first real use, four captures on the build host, `SOURCE_DATE_EPOCH` +1700000000, `core` as parent, three of them controls. The block is +reproducible from `/mnt/builds/measure/mariadb-gate`. + + attributable differences: 0 + (also 0 against each single control pair) + state paths the unit pair differs at: 167 + noise 102, overlapping 33, real 32, not sampled 0 + subtracted as floor noise, and not state paths: 6 + verdict: FAIL (0 attributable, 32 real, 33 overlapping and uncleared) + +Zero attributable, which is what the experiment in decision 0010 concluded, +now against the union of three control pairs rather than one. What is new is +the 167 state paths, which that measurement subtracted by name without +opening. + +The 32 called `real` are almost all Aria index files, `*.MAI`, and the reason +is worth knowing before the next component is measured: the Aria index header +carries a Unix timestamp, and in 30 of the 32 it increases strictly in capture +order. Three control builds moved only its low bytes; the unit build, four +minutes later, carried into the byte above. So the difference is a clock whose +carry three controls did not span. The evidence block prints the bytes from +every capture at the first differing offset, in capture order, which is what +makes that readable rather than a list of offsets: + + the 8 bytes at offset 181, in capture order: + control b9 fe 1a 00 00 00 00 00 + control b9 fe c6 00 00 00 00 00 + control b9 ff 64 00 00 00 00 00 + unit ba 00 04 00 00 00 00 00 + +The 33 called `overlapping` are almost all `*.frm`, whose create and update +timestamps sit at fixed offsets that every build moves together. + +`mysql/user.frm` is the one that came out `noise`, and correctly: `mysql.user` +is a view over `global_priv`, its `.frm` really is a text file, and its only +difference between builds is one `timestamp=` line that the mask the three +controls establish there admits. + +This FAIL is the honest state of that measurement and not a defect in the +component. Reaching PASS needs a decision that is not the tool's to make: +either more control captures, until the floor spans the counter's carry, or a +committed `--cleared` waiver saying the Aria header clock was read and +accepted. Both are on the record; silently subtracting 167 paths was not. + ## Layout Capture and formats in `bin/layer-measure-lib`, the verdicts in diff --git a/tests/compare b/tests/compare index f643235..63f828d 100755 --- a/tests/compare +++ b/tests/compare @@ -117,6 +117,14 @@ assert_ok text-plain measure_is_text "$tmp/t" assert_fail text-nul measure_is_text "$tmp/nul" assert_fail text-nonewline measure_is_text "$tmp/noeol" assert_fail text-longline measure_is_text "$tmp/longline" +# a NUL-free binary that still carries control bytes is not text, because a +# binary masked line by line can come out as noise +printf 'header\x01\x02\x03\nbody\n' > "$tmp/ctrl" +assert_fail text-controlbytes measure_is_text "$tmp/ctrl" +# a TYPE=VIEW .frm is ASCII with newlines and really is text; mysql/user.frm +# is one, and its only difference between builds is a timestamp line +printf 'TYPE=VIEW\ntimestamp=0001790574106902134\nalgorithm=0\n' > "$tmp/viewfrm" +assert_ok text-viewfrm measure_is_text "$tmp/viewfrm" assert_fail text-missing measure_is_text "$tmp/absent" info "2: measure_line, measure_common_prefix and measure_common_suffix" @@ -266,6 +274,18 @@ v=$(verdict position-cap overlapping "$tmp/mu" "$tmp/mc1" "$tmp/mc2") assert_grep position-cap-why "too many" "$v" MEASURE_POSITION_MAX=$saved +info "16b: a binary too different to compare offset by offset" +saved_off=$MEASURE_OFFSET_MAX +MEASURE_OFFSET_MAX=4 +head -c 64 /dev/urandom > "$tmp/big1" +head -c 64 /dev/urandom > "$tmp/big2" +head -c 64 /dev/urandom > "$tmp/bigu" +v=$(verdict offset-cap overlapping "$tmp/bigu" "$tmp/big1" "$tmp/big2") +assert_grep offset-cap-why "too many to compare" "$v" +assert_ok truncated-yes measure_offsets_truncated "$tmp/big1" "$tmp/big2" +MEASURE_OFFSET_MAX=$saved_off +assert_fail truncated-no measure_offsets_truncated "$tmp/ba" "$tmp/bb" + info "17: measure_cleared_reason" cat > "$tmp/cleared" <<'EOF' # a waiver is committed, and the report prints it @@ -289,6 +309,15 @@ assert_grep evidence-noise-content "written" "$e" e=$(measure_evidence ./k "$tmp/ku" "$tmp/kc1" "$tmp/kc2") assert_grep evidence-binary "offset" "$e" assert_grep evidence-binary-size "size" "$e" +# the bytes themselves, in capture order, which is what tells a clock that +# climbs with capture time apart from a change only the unit has +assert_grep evidence-window "in capture order" "$e" +assert_grep evidence-window-unit "unit " "$e" +assert_eq evidence-window-rows \ + "$(grep -cE '^ (control|unit) ' <<<"$e")" 3 +# a text path gets no byte window, because its diff already shows the content +e=$(measure_evidence ./etc/shadow "$tmp/u-clock" "$tmp/c1" "$tmp/c2" "$tmp/c3") +assert_eq evidence-window-text "$(grep -c 'in capture order' <<<"$e")" 0 e=$(measure_evidence ./x "$tmp/absent" "$tmp/c1" "$tmp/c2") assert_grep evidence-unsampled "not sampled" "$e" From 78f6984491c3af16f16afc898cbac48503dfc458 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Mon, 28 Sep 2026 07:18:34 +0000 Subject: [PATCH 4/9] fix: pin the free region of a mask, and print a cleared path's bytes Re-review found the first fix incomplete in the same direction as the defect, and it was right. The mask pinned a line's common prefix and common suffix and left the region between them free: no length, no character class, no resemblance to anything a control put there. So wherever the controls vary at random the mask degenerates to the line's fixed framing, and any value carrying that framing was admitted as noise. A hardcoded ssh_host_ed25519_key.pub behind "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI" and " root@keel", a baked-in root password hash behind "root:$y$j9T$" and ":20000:0:99999:7:::", a free region grown by 400 characters, and a SQL payload planted on the real mysql/user.frm of the gate run all came back noise. The fix is the rule already in the file applied to the other branch. This file argues that bytes carry no shape, so a binary coincidence is never noise; a per-build random text field carries no more shape than a random byte field. So noise now requires the controls to pin the free region: they agree on its length, it is no longer than MEASURE_FREE_MAX, and every character is in the class they used there, which may only be digits and the punctuation of a date. A line a control itself produced is admitted whatever its shape, because that is a value the recipe makes on its own. That also removes the dependence on when the gate ran. The user.frm mask is 21 characters of prefix only because four builds landed inside the same thousand seconds, and across a digit carry it collapses to 9. tests/compare runs that case at both lengths: the clock is noise and the payload is not, because the criterion is now the field's width and class rather than how much framing survived. A cleared path printed no verdict line and no evidence; the waiver branch returned before both. That contradicted the section's own header and it is where the bytes matter most, because a waiver is the one place a person overrides the instrument. A cleared path now prints its verdict, the rule and justification that cleared it, and its bytes, always and not subject to the abbreviation cap. The waiver itself was unbounded and unrecorded: one "*" rule matched everything including /etc/shadow, the report never said which rule covered which path, and it printed no digest. A rule is now five fields, rule, count, max-position, justified-against and reason. A rule with too little literal path is refused, a rule that covers a different number of paths than it was written against fails the run rather than stretching, and a rule justified to one offset does not cover a path differing beyond it. The block prints the file's digest and every rule with what it covered. share/layer-waivers.example is the template and it lives in the repository. tests/measure carries the composition the review built: with a waiver naming only ./var/lib/mysql/**, a unit shipping a hardcoded host key must still fail. It does; the key is overlapping and named, and the waiver covers only the two database paths it was written for. Also: tests/measure's header comment no longer lists three functions that were removed. Coverage 204, 298, 253 and 71 lines, all four files at 100 percent, gate unchanged at 99. --- COVERAGE.md | 34 +++++- bin/layer-compare-lib | 226 ++++++++++++++++++++++++++++++++---- bin/layer-report-lib | 92 ++++++++++----- bt-layer-measure | 20 +++- docs/layer-measure.md | 53 +++++++-- share/layer-waivers.example | 27 +++++ tests/compare | 152 +++++++++++++++++++++--- tests/measure | 90 ++++++++++++-- 8 files changed, 600 insertions(+), 94 deletions(-) create mode 100644 share/layer-waivers.example diff --git a/COVERAGE.md b/COVERAGE.md index 8a597f6..194b2eb 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -132,9 +132,9 @@ measures ten files: | bin/signature-lib | 76 of 76 | 100 percent | | bin/generate-signature | 87 of 87 | 100 percent | | bin/layer-measure-lib | 204 of 204 | 100 percent | -| bin/layer-compare-lib | 216 of 216 | 100 percent | -| bin/layer-report-lib | 230 of 230 | 100 percent | -| bt-layer-measure | 70 of 70 | 100 percent | +| bin/layer-compare-lib | 298 of 298 | 100 percent | +| bin/layer-report-lib | 253 of 253 | 100 percent | +| bt-layer-measure | 71 of 71 | 100 percent | The gate stays at 99, the lowest file rounded down. All four new files are at 100 percent line and branch: every subcommand, every exit code (0, 1, 2 and @@ -165,6 +165,34 @@ binary coincidence is `overlapping` and fails until it is cleared in writing. The evidence is printed for every verdict, `noise` included, because the one mistake the scheme can still make is the one nobody would otherwise see. +## 2026-09-28: a mask that constrained only the ends of a line + +The second review found the first fix incomplete in the same direction. The +mask pinned a line's common prefix and common suffix and left the region +between them free: no length, no character class, no resemblance to anything +a control put there. So wherever the controls vary at random, the mask +degenerates to the line's fixed framing and any value carrying that framing +was admitted as `noise`. A hardcoded `ssh_host_ed25519_key.pub`, a baked-in +root password hash, a free region grown by 400 characters, and a SQL payload +planted on the real `mysql/user.frm` of the gate run all came back `noise`. + +The fix is the rule already written in the file applied to the other branch. +`bin/layer-compare-lib` argues that bytes carry no shape, so a binary +coincidence is never `noise`; a per-build random text field carries no more +shape than a random byte field. `noise` now requires the controls to pin the +free region: they must agree on its length, it must be no longer than +`MEASURE_FREE_MAX`, and every character of it must be in the class they used +there, which may only be digits and the punctuation of a date. A line a +control itself produced is admitted whatever its shape, because that is a +value the recipe makes on its own. + +This also removes the dependence the review named on when the gate happened +to run. The `user.frm` mask was 21 characters of prefix only because four +builds landed inside the same thousand seconds; across a digit carry it +collapses to 9. `tests/compare` runs that case: the clock is `noise` and the +payload is not, at either prefix length, because the criterion is now the +field's width and class rather than how much framing survived. + ## 2026-09-28: pipefail in the three older suites `tests/layer`, `tests/aplinfo` and `tests/signature` did not set `pipefail` diff --git a/bin/layer-compare-lib b/bin/layer-compare-lib index 8de4cd0..727bf51 100644 --- a/bin/layer-compare-lib +++ b/bin/layer-compare-lib @@ -49,6 +49,22 @@ MEASURE_OFFSET_MAX=20000 # The field separator inside a mask, which cannot occur in a text line. MEASURE_MASK_SEP=$'\001' +# The free region of a mask is the part between the common prefix and the +# common suffix: the part the controls actually vary. Admitting a line means +# admitting whatever it put there, so the controls have to pin it. They pin +# it by agreeing on its length, which must be no more than this, and on its +# characters, which must all be in the class below. +# +# Why this class and not a wider one. A per-build random text field carries +# no more shape than a random byte field, and this file already refuses to +# call a binary coincidence noise for exactly that reason. Install-time text +# state varies by clocks, counters and log sequence numbers, which are +# digits and the punctuation of a date. A key, a password hash, a hostname +# or a SQL fragment is not, and each of those was admitted as noise while +# this constrained only the two ends of the line. +MEASURE_FREE_MAX=24 +MEASURE_FREE_CLASS='0-9.:[:space:]-' + # How many bytes of a window the binary evidence prints from each capture at # the first differing offset. Eight covers a 32 or 64 bit counter, which is # what most of these differences turn out to be. @@ -222,42 +238,201 @@ measure_mask_at() { "$(measure_common_suffix "${lines[@]}")" } -# measure_mask_admits LINE PREFIX SUFFIX -# A mask with neither a prefix nor a suffix admits nothing: it is what two -# unrelated lines have in common, which is no evidence about a third. +# measure_free_region STRING PREFIX SUFFIX -> the part between the two +measure_free_region() { + local s=$1 n=$(( ${#1} - ${#2} - ${#3} )) + [[ $n -gt 0 ]] || return 0 + printf '%s\n' "${s:${#2}:$n}" +} + +# measure_mask_admits LINE PREFIX SUFFIX CONTROL_LINE... +# +# Admits LINE only when the controls pin what it may carry between the +# prefix and the suffix. Four ways they fail to: +# +# the mask has neither a prefix nor a suffix, so it is only what two +# unrelated lines happen to share and says nothing about a third; +# the controls all carry the same line, so they establish no varying shape; +# the free regions are not all one length, or LINE's is a different length; +# a free region is longer than MEASURE_FREE_MAX, or carries a character +# outside MEASURE_FREE_CLASS or outside what the controls themselves used. +# +# A LINE a control itself produced is pinned by observation and admitted +# whatever its shape, because it is a value this recipe makes on its own. measure_mask_admits() { local line=$1 prefix=$2 suffix=$3 + shift 3 + local v first="" same=yes + for v in "$@"; do + [[ -n "$first" ]] || first=$v + [[ "$v" == "$first" ]] || same=no + [[ "$line" != "$v" ]] || return 0 + done + [[ "$same" == no ]] || return 1 [[ -n "$prefix" ]] || [[ -n "$suffix" ]] || return 1 [[ ${#line} -ge $(( ${#prefix} + ${#suffix} )) ]] || return 1 [[ "$line" == "$prefix"* ]] || return 1 [[ -z "$suffix" ]] || [[ "$line" == *"$suffix" ]] || return 1 + local free seen="" len=-1 + for v in "$@"; do + free=$(measure_free_region "$v" "$prefix" "$suffix") + [[ $len -lt 0 ]] || [[ ${#free} -eq $len ]] || return 1 + len=${#free} + seen=$seen$free + done + [[ $len -gt 0 ]] || return 1 + [[ $len -le $MEASURE_FREE_MAX ]] || return 1 + measure_free_pinned "$seen" || return 1 + free=$(measure_free_region "$line" "$prefix" "$suffix") + [[ ${#free} -eq $len ]] || return 1 + # the same character class the controls used there, not the same + # characters: a counter reaches digits three builds did not happen to + # show, while a key reaches letters they never could + [[ -z "$(tr -d "$(measure_free_class "$seen")" <<<"$free")" ]] || return 1 return 0 } -# measure_cleared_reason FILE PATH -> the reason a waiver gives, or nothing +# measure_free_class STRING -> a tr set covering the classes STRING uses +measure_free_class() { + local out="" + [[ "$1" != *[0-9]* ]] || out=$out'0-9' + [[ "$1" != *.* ]] || out=$out'.' + [[ "$1" != *:* ]] || out=$out':' + [[ "$1" != *[[:space:]]* ]] || out=$out'[:space:]' + [[ "$1" != *-* ]] || out=$out'-' + printf '%s\n' "$out" +} + +# measure_mask_framed LINE PREFIX SUFFIX +# The line carries the framing the controls share, so the difference is +# inside the region they vary; what it put there is the open question. +measure_mask_framed() { + local line=$1 prefix=$2 suffix=$3 + [[ -n "$prefix" ]] || [[ -n "$suffix" ]] || return 0 + [[ ${#line} -ge $(( ${#prefix} + ${#suffix} )) ]] || return 1 + [[ "$line" == "$prefix"* ]] || return 1 + [[ -z "$suffix" ]] || [[ "$line" == *"$suffix" ]] || return 1 + return 0 +} + +# measure_free_summary LINE PREFIX SUFFIX -> how the free region is described +# in the one line verdict, without printing a secret into it +measure_free_summary() { + local free + free=$(measure_free_region "$1" "$2" "$3") + if [[ -z "$free" ]]; then + echo "nothing" + return 0 + fi + local kind="not a number or a timestamp" + ! measure_free_pinned "$free" || kind="a number or a timestamp" + printf '%s characters, %s\n' "${#free}" "$kind" +} + +# measure_free_pinned STRING +# Every character is a digit or the punctuation of a date or a duration. +measure_free_pinned() { + [[ -n "$1" ]] || return 1 + [[ -z "$(tr -d "$MEASURE_FREE_CLASS" <<<"$1")" ]] || return 1 + return 0 +} + +# A waiver file is five tab separated fields per rule: # -# A waiver line is PATHREASON, and PATH may be a glob, because a real -# MariaDB data directory is 173 files that one sentence explains. A glob -# waiver is only honest because the report names every path it covered, so -# it is a record of what was waived and not a way of not looking. -measure_cleared_reason() { - local file=$1 path=$2 rule reason line +# rulecountmax-positionjustified-againstreason +# +# rule an exact path, or a glob; a glob has to carry enough +# literal path to be a decision about somewhere +# count how many differing state paths it covered when it was +# written. A run where it covers a different number +# fails, so a new file appearing in a waived directory +# cannot be swept in silently +# max-position the highest differing byte offset or line number the +# waiver was justified against. A covered path differing +# beyond it is not waived, so the same path turning +# overlapping somewhere else is not quietly included +# justified-against the capture directory and recipe commit it was read +# against, so it cannot outlive its evidence +# reason what was read, and what it was found to be +# +# It lives in the repository beside the component, and the report prints its +# digest and every rule, because a waiver is the one place a person +# overrides the instrument. +MEASURE_WAIVER_FIELDS=5 + +# The least literal path a glob rule has to carry. "*" matches ./etc/shadow. +MEASURE_WAIVER_MIN_LITERAL=8 + +# measure_waiver_rules FILE -> the rules, one per line, comments gone +measure_waiver_rules() { + [[ -f "$1" ]] || return 0 + sed -e '/^[[:space:]]*#/d' -e '/^[[:space:]]*$/d' "$1" +} + +# measure_waiver_check FILE +measure_waiver_check() { + local file=$1 line rule count max rest literal rc=0 n=0 + [[ -f "$file" ]] || return 0 + local -a lines=() + mapfile -t lines < <(measure_waiver_rules "$file") + for line in "${lines[@]}"; do + n=$((n + 1)) + IFS=$'\t' read -r rule count max rest <<<"$line" + if [[ "$(awk -F'\t' '{ print NF }' <<<"$line")" -lt "$MEASURE_WAIVER_FIELDS" ]]; then + measure_error "$file line $n: a waiver has $MEASURE_WAIVER_FIELDS tab separated fields, rule, count, max-position, justified-against and reason" + rc=1 + continue + fi + literal=${rule//[*?]/} + if [[ ${#literal} -lt $MEASURE_WAIVER_MIN_LITERAL ]]; then + measure_error "$file line $n: '$rule' is too broad to be a decision about anywhere; a rule needs at least $MEASURE_WAIVER_MIN_LITERAL characters of literal path" + rc=1 + fi + if [[ ! "$count" =~ ^[0-9]+$ ]] || [[ ! "$max" =~ ^[0-9]+$ ]]; then + measure_error "$file line $n: count and max-position are numbers, not '$count' and '$max'" + rc=1 + fi + [[ -n "$rest" ]] || { measure_error "$file line $n: no justification"; rc=1; } + done + return $rc +} + +# measure_cleared_rule FILE PATH MAXPOS -> the whole rule line that covers +# PATH, or nothing. MAXPOS is the furthest this path actually differs at; a +# rule justified against less than that does not cover it. +measure_cleared_rule() { + local file=$1 path=$2 maxpos=$3 line rule count max [[ -f "$file" ]] || return 0 - local -a rules=() - mapfile -t rules < "$file" - for line in "${rules[@]}"; do - IFS=$'\t' read -r rule reason <<<"$line" + local -a lines=() + mapfile -t lines < <(measure_waiver_rules "$file") + for line in "${lines[@]}"; do + IFS=$'\t' read -r rule count max _ <<<"$line" [[ -n "$rule" ]] || continue - [[ "$rule" != \#* ]] || continue # shellcheck disable=SC2053 - if [[ "$path" == "$rule" ]] || [[ "$path" == $rule ]]; then - printf '%s\n' "$reason" - return 0 - fi + [[ "$path" == "$rule" ]] || [[ "$path" == $rule ]] || continue + [[ "$maxpos" -le "$max" ]] || continue + printf '%s\n' "$line" + return 0 done return 0 } +# measure_cleared_reason RULE_LINE -> the reason field of a rule line +measure_cleared_reason() { + cut -f5- <<<"$1" +} + +# measure_max_position UNIT BASE -> how far into the file it differs +measure_max_position() { + local unit=$1 base=$2 + if measure_is_text "$base" && measure_is_text "$unit"; then + measure_text_positions "$base" "$unit" | sed 's/^[on]//' | LC_ALL=C sort -n | tail -1 + return 0 + fi + measure_byte_offsets "$base" "$unit" | LC_ALL=C sort -n | tail -1 + return 0 +} + # --------------------------------------------------------------------------- # the verdict # --------------------------------------------------------------------------- @@ -323,7 +498,8 @@ measure_verdict() { measure_verdict_text() { local positions=$1 unit=$2 shift 2 - local token n mask prefix suffix line unexplained=0 masked=0 + local token n mask prefix suffix line f unexplained=0 masked=0 + local -a variants=() while IFS= read -r token; do [[ -n "$token" ]] || continue n=${token#[on]} @@ -331,13 +507,17 @@ measure_verdict_text() { prefix=${mask%%"$MEASURE_MASK_SEP"*} suffix=${mask#*"$MEASURE_MASK_SEP"} line=$(measure_line "$n" "$unit") - if measure_mask_admits "$line" "$prefix" "$suffix"; then + variants=() + for f in "$@"; do + variants+=("$(measure_line "$n" "$f")") + done + if measure_mask_admits "$line" "$prefix" "$suffix" "${variants[@]}"; then masked=$((masked + 1)) continue fi - if [[ -z "$prefix" ]] && [[ -z "$suffix" ]]; then + if measure_mask_framed "$line" "$prefix" "$suffix"; then printf 'overlapping\t%s\n' \ - "line $n varies in the controls with nothing in common, so nothing about it can be shown; read the evidence" + "line $n has the framing the controls vary within, and nothing pins what it put between: $(measure_free_summary "$line" "$prefix" "$suffix"); read the evidence" return 0 fi unexplained=$((unexplained + 1)) diff --git a/bin/layer-report-lib b/bin/layer-report-lib index ea3c551..4525b8c 100644 --- a/bin/layer-report-lib +++ b/bin/layer-report-lib @@ -300,7 +300,7 @@ measure_state_section() { shift 5 local -a c=("$@") local base=${c[0]} - local paths path verdict why reason tag + local paths path verdict why rule shown_verdict tag maxpos local -a samples=() local noise=0 real=0 overlapping=0 unsampled=0 none=0 waived=0 examined=0 local evidence="" shown=0 abbreviated=0 waivers="" @@ -325,22 +325,35 @@ measure_state_section() { measure_error "no verdict for $path, which must not happen" return 1 fi - reason=$(measure_cleared_reason "$cleared" "$path") - if [[ -n "$reason" ]] && [[ "$verdict" == overlapping ]]; then + rule="" + if [[ "$verdict" == overlapping ]]; then + maxpos=$(measure_max_position "$(measure_sample_path "$dir/$unit" "$path")" "${samples[0]}") + rule=$(measure_cleared_rule "$cleared" "$path" "$maxpos") + fi + shown_verdict=$verdict + if [[ -n "$rule" ]]; then waived=$((waived + 1)) - waivers="$waivers$(printf '%s\t%s' "$path" "$reason")"$'\n' - continue + shown_verdict=cleared + waivers="$waivers$(printf '%s\t%s' "$path" "$(cut -f1 <<<"$rule")")"$'\n' + else + case "$verdict" in + noise) noise=$((noise + 1)) ;; + real) real=$((real + 1)) ;; + overlapping) overlapping=$((overlapping + 1)) ;; + unsampled) unsampled=$((unsampled + 1)) ;; + none) none=$((none + 1)) ;; + esac fi - case "$verdict" in - noise) noise=$((noise + 1)) ;; - real) real=$((real + 1)) ;; - overlapping) overlapping=$((overlapping + 1)) ;; - unsampled) unsampled=$((unsampled + 1)) ;; - none) none=$((none + 1)) ;; - esac - printf ' %-11s %s\n' "$verdict" "$path" + printf ' %-11s %s\n' "$shown_verdict" "$path" printf ' %s\n' "$why" - if [[ "$verdict" == real ]] || [[ $shown -lt $MEASURE_EVIDENCE_PATHS ]]; then + if [[ -n "$rule" ]]; then + printf ' cleared by rule %s, justified against %s: %s\n' \ + "$(cut -f1 <<<"$rule")" "$(cut -f4 <<<"$rule")" \ + "$(measure_cleared_reason "$rule")" + fi + # a waiver is the one place a person overrides the instrument, so it + # is the last place to withhold the bytes + if [[ "$verdict" == real ]] || [[ -n "$rule" ]] || [[ $shown -lt $MEASURE_EVIDENCE_PATHS ]]; then shown=$((shown + 1)) evidence="$evidence$(measure_evidence "$path" "$(measure_sample_path "$dir/$unit" "$path")" "${samples[@]}")"$'\n' else @@ -353,15 +366,8 @@ measure_state_section() { fi printf ' totals: noise %s, overlapping %s, real %s, not sampled %s, unchanged %s, cleared %s\n' \ "$noise" "$overlapping" "$real" "$unsampled" "$none" "$waived" - if [[ -n "$waivers" ]]; then - echo - printf 'waived in writing (%s paths, every one of them named):\n' "$waived" - measure_lines " " "$(head -n "$MEASURE_PRINT_MAX" <<<"$waivers" \ - | sed "s/\t/ -> /")" - [[ "$waived" -le "$MEASURE_PRINT_MAX" ]] \ - || printf ' and %s more, all covered by the same waiver file\n' \ - "$((waived - MEASURE_PRINT_MAX))" - fi + local stretched=0 + measure_waiver_section "$cleared" "$waivers" "$waived" || stretched=1 if [[ -n "$evidence" ]]; then echo echo "evidence, for every state path above:" @@ -373,13 +379,13 @@ measure_state_section() { measure_subtracted_section "$dir" "$unit" "$suspects" "$globs" "${c[@]}" local suspect=$MEASURE_SUSPECT_COUNT echo - if [[ $((MEASURE_ATTRIBUTABLE + real + overlapping + unsampled + suspect)) -eq 0 ]]; then + if [[ $((MEASURE_ATTRIBUTABLE + real + overlapping + unsampled + suspect + stretched)) -eq 0 ]]; then echo "verdict: PASS" echo "============================================================================" return 0 fi - printf 'verdict: FAIL (%s attributable, %s real, %s overlapping and uncleared, %s not sampled, %s subtracted but state shaped)\n' \ - "$MEASURE_ATTRIBUTABLE" "$real" "$overlapping" "$unsampled" "$suspect" + printf 'verdict: FAIL (%s attributable, %s real, %s overlapping and uncleared, %s not sampled, %s subtracted but state shaped, %s waiver rules stretched)\n' \ + "$MEASURE_ATTRIBUTABLE" "$real" "$overlapping" "$unsampled" "$suspect" "$stretched" echo "============================================================================" return 1 } @@ -411,3 +417,37 @@ measure_subtracted_section() { MEASURE_SUSPECT_COUNT=$(grep -c . <<<"$suspect" || true) return 0 } + +# measure_waiver_section FILE WAIVED_PATHS COUNT +# The waiver file itself, its digest and every rule, then which rule covered +# which path. A rule that covered a different number of paths than it was +# written against fails the run, so a new file appearing in a waived +# directory cannot be swept in without anybody noticing. +measure_waiver_section() { + local file=$1 waivers=$2 waived=$3 rule count covered rc=0 + echo + if [[ ! -f "$file" ]]; then + echo "waivers: none" + return 0 + fi + printf 'waivers %s (sha256 %s)\n' "$file" \ + "$(sha256sum < "$file" | cut -c1-16)" + local -a lines=() + mapfile -t lines < <(measure_waiver_rules "$file") + for rule in "${lines[@]}"; do + [[ -n "$rule" ]] || continue + count=$(cut -f2 <<<"$rule") + covered=$(LC_ALL=C awk -F'\t' -v r="$(cut -f1 <<<"$rule")" '$2 == r { n++ } END { print n + 0 }' <<<"$waivers") + printf ' rule %s: %s paths (written against %s, max position %s)\n' \ + "$(cut -f1 <<<"$rule")" "$covered" "$count" "$(cut -f3 <<<"$rule")" + printf ' against %s: %s\n' "$(cut -f4 <<<"$rule")" \ + "$(measure_cleared_reason "$rule")" + if [[ "$covered" != "$count" ]]; then + measure_error "the waiver for $(cut -f1 <<<"$rule") was written against $count paths and covers $covered here; read the difference and rewrite it" + echo " THIS RULE HAS STRETCHED: written against $count, covers $covered" + rc=1 + fi + done + printf ' cleared in this run: %s paths, every one of them named above\n' "$waived" + return $rc +} diff --git a/bt-layer-measure b/bt-layer-measure index 8cf40e3..ca978bb 100755 --- a/bt-layer-measure +++ b/bt-layer-measure @@ -86,12 +86,18 @@ Options:: --unit TAG the build with the component supplied as a unit --control-again TAG another control build. Repeatable, and the floor is the union over every pair of controls - --cleared FILE waivers: lines of PATHREASON. A state path whose - difference coincides with the floor but cannot be - shown to be that variation is 'overlapping' and fails - the run; a waiver clears it and prints the reason in - the block, so the judgement is on the record. A - waiver cannot clear a 'real' difference + --cleared FILE waivers, five tab separated fields per rule: + rule, count, max-position, justified-against, reason. + A state path whose difference coincides with the + floor but cannot be shown to be that variation is + 'overlapping' and fails the run; a waiver clears it, + and the block prints the file's digest, every rule + and every path each rule covered, with its bytes. A + rule covering a different number of paths than it was + written against fails the run rather than stretching, + a rule is refused if it carries too little literal + path to be a decision about anywhere, and a waiver + can never clear a 'real' difference Environment:: @@ -232,6 +238,8 @@ do_attribute() { measure_preconditions "$MEASURE_DIR" "$MEASURE_UNIT" \ "$MEASURE_CONTROL" "${MEASURE_CONTROL2[@]}" \ || die "$EXIT_INPUT" "the captures cannot be compared" + measure_waiver_check "${MEASURE_CLEARED:-/nonexistent}" \ + || die "$EXIT_INPUT" "the waiver file is not usable" measure_attribute_report "$MEASURE_DIR" "$MEASURE_UNIT" \ "${MEASURE_CLEARED:-/nonexistent}" "$(suspect_paths_file)" \ "$MEASURE_CONTROL" "${MEASURE_CONTROL2[@]}" || exit "$EXIT_DIFFERENCES" diff --git a/docs/layer-measure.md b/docs/layer-measure.md index 5d69cc8..40b3082 100644 --- a/docs/layer-measure.md +++ b/docs/layer-measure.md @@ -164,10 +164,35 @@ they were noise: So the controls are the model, and the model has content in it. At each position the controls vary at, the longest common prefix and suffix of their -variants is the *shape* of that variation, and the unit is admitted only if -its line has that shape. `# written 1790475089` against `# written 1790475323` -gives the shape `# written ...`, which `# written 1790475511` fits and -`root:$y$hash:...` does not. +variants is the framing of that variation, and the region between them is +what they actually vary. + +Framing alone is not enough, and the first version of this made that mistake. +It pinned the two ends of the line and left the middle free, so wherever the +controls vary at random the mask degenerates to the line's fixed parts and +any value carrying them was admitted: a hardcoded `ssh_host_ed25519_key.pub` +behind `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI` and ` root@keel`, a baked-in +password hash behind `root:$y$j9T$` and `:20000:0:99999:7:::`, and a SQL +payload behind `timestamp=0001790574`. + +**So the controls have to pin the free region too.** They must agree on its +length, it must be no longer than 24 characters, and every character of it +must be in the class they used there, which may only be digits and the +punctuation of a date. That is the rule this file already applies to bytes, +which carry no shape and are therefore never `noise`; a per-build random text +field carries no more shape than a random byte field. Install-time text state +varies by clocks, counters and log sequence numbers. A key, a hash, a hostname +or a SQL fragment is not one of those. + +A line a control itself produced is admitted whatever its shape, because it is +a value this recipe makes on its own. + +One consequence worth having: the verdict no longer depends on when the gate +ran. `mysql/user.frm`'s mask is 21 characters of prefix only because four +builds landed inside the same thousand seconds, and across a digit carry it +collapses to 9. The clock is `noise` at either length and the payload is not, +because the criterion is the field's width and class rather than how much +framing happened to survive. | Verdict | Meaning | Run | |---------|---------|-----| @@ -194,15 +219,23 @@ not pass as if the file were unchanged. Raise `--state-max-bytes` (default ### Waivers -`--cleared FILE` takes lines of `PATHREASON`, and `PATH` may be a glob, -because a real data directory is 173 files that one sentence explains: +`--cleared FILE` takes five tab separated fields per rule, and the file lives +in the repository beside the component. `share/layer-waivers.example` is the +template. - ./var/lib/mysql/** InnoDB and Aria creation stamps, read byte by byte on 2026-09-28 +| Field | What it is for | +|---|---| +| `rule` | an exact path, or a glob carrying enough literal path to be a decision about somewhere. `*` is refused: it matches `./etc/shadow` | +| `count` | how many differing state paths it covered when written. A run where it covers a different number **fails**, so a new file appearing in a waived directory cannot be swept in silently | +| `max-position` | the highest differing byte offset or line number it was justified against. A covered path differing beyond it is **not** waived | +| `justified-against` | the capture directory and recipe commit it was read against, so it cannot outlive its evidence | +| `reason` | what was read, and what it was found to be | A waiver clears `overlapping` and nothing else: it cannot clear a `real` -difference. Every path a waiver covered is named in the block, so a reviewer -sees what was waived and on what grounds, and the file itself is committed -next to the component. +difference. The block prints the file's digest, every rule, how many paths +each one covered against how many it was written for, and **every cleared +path with its bytes**. A waiver is the one place a person overrides the +instrument, so it is the last place to withhold the evidence. ### What the floor removed diff --git a/share/layer-waivers.example b/share/layer-waivers.example new file mode 100644 index 0000000..da7bc83 --- /dev/null +++ b/share/layer-waivers.example @@ -0,0 +1,27 @@ +# A waiver file. It lives in the repository beside the component it is about, +# and bt-layer-measure attribute prints its digest and every rule below, with +# every path each rule covered and that path's bytes. +# +# A waiver clears `overlapping`, which is a difference that coincides with +# what the control builds vary but cannot be shown to be that variation. It +# can never clear `real`. It is the one place a person overrides the +# instrument, so it has to say what was read and what it was found to be. +# +# Five tab separated fields: +# +# rule an exact path, or a glob carrying enough literal path +# to be a decision about somewhere. "*" is refused +# count how many differing state paths it covered when it was +# written. A run where it covers a different number +# fails, so a new file appearing in a waived directory +# cannot be swept in silently +# max-position the highest differing byte offset or line number it +# was justified against. A covered path differing +# beyond it is not waived, so the same path turning +# overlapping somewhere else is not quietly included +# justified-against the capture directory and the recipe commit it was +# read against, so it cannot outlive its evidence +# reason what was read, and what it was found to be +# +# rule count max-position justified-against reason +./var/lib/mysql/** 33 16777216 /mnt/builds/measure/mariadb-gate, recipe 659eeb4 Aria and InnoDB header timestamps; the six accounts and their authentication_string values are identical in all four captures, only password_last_changed moves. Read 2026-09-28. diff --git a/tests/compare b/tests/compare index 63f828d..20ec4df 100755 --- a/tests/compare +++ b/tests/compare @@ -88,6 +88,23 @@ assert_grep() { grep -q -- "$pattern" <<<"$haystack" || fatal "$label: no '$pattern' in: $haystack" } +# not_noise LABEL UNIT CONTROL... +# What the four attacks below are actually about: whichever of `real` and +# `overlapping` a case lands on, both fail the run and both print evidence. +# Asserting one of them exactly would be asserting an accident of how much +# framing the planted line happened to keep. +not_noise() { + local label=$1 + shift + local got + got=$(measure_verdict "$@") || fatal "$label: measure_verdict failed" + case "$(cut -f1 <<<"$got")" in + real|overlapping) ;; + *) fatal "$label: got '$got', wanted real or overlapping" ;; + esac + printf '%s\n' "$got" +} + # verdict LABEL WANT UNIT CONTROL... verdict() { local label=$1 want=$2 @@ -186,11 +203,38 @@ printf 'ZZZaaa\n' > "$tmp/q1"; printf 'YYYaaa\n' > "$tmp/q2" assert_eq mask-suffixonly "$(measure_mask_at 1 "$tmp/q1" "$tmp/q2")" "$(printf '\001aaa')" printf 'abc\n' > "$tmp/r1"; printf 'xyz\n' > "$tmp/r2" assert_eq mask-neither "$(measure_mask_at 1 "$tmp/r1" "$tmp/r2")" "$(printf '\001')" -assert_ok admits-yes measure_mask_admits "# written 4000" "# written " "000" -assert_fail admits-prefix measure_mask_admits "root:x:0" "# written " "" -assert_fail admits-suffix measure_mask_admits "# written 4242" "# written " "000" -assert_fail admits-short measure_mask_admits "# wr" "# written " "" -assert_fail admits-emptymask measure_mask_admits "anything at all" "" "" +# the free region, the part between the prefix and the suffix, is the part +# the controls actually vary; admitting a line means admitting what it put +# there, so the controls have to pin it +assert_eq free-middle "$(measure_free_region "# written 4000" "# written " "000")" "4" +assert_eq free-empty "$(measure_free_region "abc" "abc" "")" "" +assert_eq free-overlap "$(measure_free_region "ab" "abc" "abc")" "" +assert_ok admits-yes measure_mask_admits "# written 4000" "# written " "000" \ + "# written 1000" "# written 2000" "# written 3000" +assert_fail admits-prefix measure_mask_admits "root:x:0" "# written " "" \ + "# written 1000" "# written 2000" +assert_fail admits-suffix measure_mask_admits "# written 4242" "# written " "000" \ + "# written 1000" "# written 2000" +assert_fail admits-short measure_mask_admits "# wr" "# written " "" \ + "# written 1000" "# written 2000" +assert_fail admits-emptymask measure_mask_admits "anything at all" "" "" \ + "abc" "xyz" +# a free region of a different length than every control's is not that field +assert_fail admits-longer measure_mask_admits "# written 40000" "# written " "" \ + "# written 1000" "# written 2000" +# a free region carrying anything but a number or a timestamp is not pinned, +# however well the line's framing matches +assert_fail admits-class measure_mask_admits "# written abcd" "# written " "" \ + "# written 1000" "# written 2000" +# controls that agree at a line establish no varying shape at all +assert_fail admits-noshape measure_mask_admits "# written 4000" "# written 1000" "" \ + "# written 1000" "# written 1000" +# a unit line one of the controls itself produced is pinned by observation +assert_ok admits-observed measure_mask_admits "# written 2000" "# written " "000" \ + "# written 1000" "# written 2000" +# and a free region longer than the cap is never pinned +long=$(head -c 40 /dev/zero | tr '\0' '7') +assert_fail admits-toolong measure_mask_admits "t=$long" "t=" "" "t=1111111111" "t=2222222222" # --------------------------------------------------------------------------- # the verdicts, and the four false negatives that made this rewrite necessary @@ -286,20 +330,96 @@ assert_ok truncated-yes measure_offsets_truncated "$tmp/big1" "$tmp/big2" MEASURE_OFFSET_MAX=$saved_off assert_fail truncated-no measure_offsets_truncated "$tmp/ba" "$tmp/bb" -info "17: measure_cleared_reason" +info "16c: the four ways a two-ended mask was made to admit a constant" +# each of these was reported as noise by the mask that constrained only the +# prefix and the suffix, with a real secret in the free region +key3() { printf 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI%s root@keel\n' "$1" > "$2"; } +key3 "$(head -c 32 /dev/urandom | base64 | tr -d '=\n' | head -c 30)" "$tmp/k1" +key3 "$(head -c 32 /dev/urandom | base64 | tr -d '=\n' | head -c 30)" "$tmp/k2" +key3 "$(head -c 32 /dev/urandom | base64 | tr -d '=\n' | head -c 30)" "$tmp/k3" +key3 "HARDCODEDKEYTHATTHEUNITSHIPSXX" "$tmp/kunit" +not_noise attack-sshkey "$tmp/kunit" "$tmp/k1" "$tmp/k2" "$tmp/k3" >/dev/null +# shellcheck disable=SC2016 # a literal crypt prefix, not an expansion +hash3() { printf 'root:$y$j9T$%s:20000:0:99999:7:::\n' "$1" > "$2"; } +hash3 "$(head -c 16 /dev/urandom | base64 | tr -d '=\n' | head -c 20)" "$tmp/h1" +hash3 "$(head -c 16 /dev/urandom | base64 | tr -d '=\n' | head -c 20)" "$tmp/h2" +hash3 "$(head -c 16 /dev/urandom | base64 | tr -d '=\n' | head -c 20)" "$tmp/h3" +hash3 "BAKEDINPASSWORDHASHX" "$tmp/hunit" +not_noise attack-shadow "$tmp/hunit" "$tmp/h1" "$tmp/h2" "$tmp/h3" >/dev/null +for n in 1 2 3; do printf '# written 179047500%s by build\n' "$n" > "$tmp/w$n"; done +printf '# written 1790475 GRANT ALL ON *.* TO evil by build\n' > "$tmp/wunit" +not_noise attack-planted "$tmp/wunit" "$tmp/w1" "$tmp/w2" "$tmp/w3" >/dev/null +for n in 1 2 3; do printf 'token=%s\n' "$n$n$n$n" > "$tmp/g$n"; done +printf 'token=%s\n' "$(head -c 400 /dev/zero | tr '\0' 'A')" > "$tmp/gunit" +not_noise attack-grown "$tmp/gunit" "$tmp/g1" "$tmp/g2" "$tmp/g3" >/dev/null +# the payload planted on the real mysql/user.frm of the gate run +for n in 1 2 3; do printf 'TYPE=VIEW\ntimestamp=000179057410690213%s\n' "$n" > "$tmp/v$n"; done +printf 'TYPE=VIEW\ntimestamp=0001790574 OR 1=1; GRANT ALL ON *.* TO evil@localhost\n' > "$tmp/vunit" +not_noise attack-userfrm "$tmp/vunit" "$tmp/v1" "$tmp/v2" "$tmp/v3" >/dev/null +# and the clock those four were hiding behind is still noise +printf 'TYPE=VIEW\ntimestamp=0001790574106902139\n' > "$tmp/vclock" +verdict clock-still-noise noise "$tmp/vclock" "$tmp/v1" "$tmp/v2" "$tmp/v3" >/dev/null + +info "16d: the verdict does not depend on where the builds fell" +# the review's point: the user.frm mask was 21 characters of prefix only +# because four builds landed inside the same thousand seconds, and a +# criterion whose answer depends on the hour the gate ran is not a +# criterion. Across a digit carry the prefix collapses to 9 characters; the +# clock must still be noise and the payload must still not be +for v in 0001790574999999999 0001790575000000001 0001790575000000009; do + printf 'TYPE=VIEW\ntimestamp=%s\n' "$v" > "$tmp/carry-$v" +done +printf 'TYPE=VIEW\ntimestamp=0001790575000000123\n' > "$tmp/carry-clock" +printf 'TYPE=VIEW\ntimestamp=0001790574 OR 1=1; GRANT ALL ON *.* TO evil\n' > "$tmp/carry-payload" +assert_eq carry-prefix \ + "$(measure_common_prefix 0001790574999999999 0001790575000000001 0001790575000000009)" \ + "000179057" +verdict carry-clock noise "$tmp/carry-clock" "$tmp/carry-0001790574999999999" \ + "$tmp/carry-0001790575000000001" "$tmp/carry-0001790575000000009" >/dev/null +not_noise carry-payload "$tmp/carry-payload" "$tmp/carry-0001790574999999999" \ + "$tmp/carry-0001790575000000001" "$tmp/carry-0001790575000000009" >/dev/null + +info "17: the waiver file" cat > "$tmp/cleared" <<'EOF' -# a waiver is committed, and the report prints it -./var/lib/mysql/ibdata1 InnoDB creation stamps, checked by hand on 2026-09-28 +# rule, count, max-position, justified-against, reason +./var/lib/mysql/ibdata1 1 600 captures at /m/gate, recipe 659eeb4 InnoDB creation stamps, checked by hand on 2026-09-28 +./var/lib/postgresql/** 6 4096 captures at /m/gate, recipe 659eeb4 initdb output, read on 2026-09-28 EOF -assert_eq cleared-hit "$(measure_cleared_reason "$tmp/cleared" ./var/lib/mysql/ibdata1)" \ +assert_ok waiver-ok measure_waiver_check "$tmp/cleared" +assert_ok waiver-nofile measure_waiver_check "$tmp/absent" +assert_eq waiver-rules "$(measure_waiver_rules "$tmp/cleared" | wc -l)" 2 +rule=$(measure_cleared_rule "$tmp/cleared" ./var/lib/mysql/ibdata1 600) +assert_eq cleared-hit "$(measure_cleared_reason "$rule")" \ "InnoDB creation stamps, checked by hand on 2026-09-28" -assert_eq cleared-miss "$(measure_cleared_reason "$tmp/cleared" ./etc/shadow)" "" -assert_eq cleared-nofile "$(measure_cleared_reason "$tmp/absent" ./etc/shadow)" "" -# a glob waiver, because a real data directory is 173 files and one sentence -printf './var/lib/postgresql/**\tinitdb output, read on 2026-09-28\n' >> "$tmp/cleared" -assert_eq cleared-glob "$(measure_cleared_reason "$tmp/cleared" ./var/lib/postgresql/17/main/PG_VERSION)" \ - "initdb output, read on 2026-09-28" -assert_eq cleared-glob-miss "$(measure_cleared_reason "$tmp/cleared" ./var/lib/other/x)" "" +# a rule justified only as far as offset 600 does not cover a difference at 601 +assert_eq cleared-beyond "$(measure_cleared_rule "$tmp/cleared" ./var/lib/mysql/ibdata1 601)" "" +assert_eq cleared-miss "$(measure_cleared_rule "$tmp/cleared" ./etc/shadow 1)" "" +assert_eq cleared-nofile "$(measure_cleared_rule "$tmp/absent" ./etc/shadow 1)" "" +# a glob covers a subtree, because a data directory is many files and one +# sentence, and the report names every path it covered +assert_grep cleared-glob "initdb output" \ + "$(measure_cleared_reason "$(measure_cleared_rule "$tmp/cleared" ./var/lib/postgresql/17/main/PG_VERSION 10)")" +# a rule with too little literal path is a decision about nowhere +printf '*\t1\t1\tnowhere\tanything\n' > "$tmp/broad" +assert_fail waiver-broad measure_waiver_check "$tmp/broad" +printf './etc/*\t1\t1\tnowhere\tstill too broad\n' > "$tmp/broad2" +assert_fail waiver-broad2 measure_waiver_check "$tmp/broad2" +printf './var/lib/mysql/**\ttwo fields\n' > "$tmp/short" +assert_fail waiver-fields measure_waiver_check "$tmp/short" +printf './var/lib/mysql/**\tmany\t1\tnowhere\tcount is not a number\n' > "$tmp/nan" +assert_fail waiver-nan measure_waiver_check "$tmp/nan" +printf './var/lib/mysql/**\t1\t1\t\t\n' > "$tmp/nojust" +assert_fail waiver-nojust measure_waiver_check "$tmp/nojust" + +assert_eq free-summary-none "$(measure_free_summary "abc" "abc" "")" "nothing" +assert_grep free-summary-num "a number or a timestamp" \ + "$(measure_free_summary "t=4000" "t=" "")" +assert_grep free-summary-not "not a number" \ + "$(measure_free_summary "t=abcd" "t=" "")" + +info "17b: measure_max_position" +assert_eq maxpos-text "$(measure_max_position "$tmp/c2" "$tmp/c1")" 1 +assert_eq maxpos-binary "$(measure_max_position "$tmp/bb" "$tmp/ba")" 3 info "18: measure_evidence is printed for every verdict, noise included" e=$(measure_evidence ./etc/shadow "$tmp/u-clock" "$tmp/c1" "$tmp/c2" "$tmp/c3") diff --git a/tests/measure b/tests/measure index 69cd246..e352d4a 100755 --- a/tests/measure +++ b/tests/measure @@ -4,7 +4,8 @@ # place, so every capture below is a real capture of a real tree. # # Code paths covered: -# layer-measure-lib: measure_path_safe (plain, space, tab, newline); +# layer-measure-lib: measure_path_safe (plain, space, tab, newline, +# backslash); # measure_tree_record (sorted, tab keyed, size and mode, a path with a # space, a rejected tab, a missing root); measure_links_record (target # with a space, none); measure_pkgs_record (installed only, dpkg-query @@ -15,13 +16,16 @@ # measure_capture_check (complete, incomplete, missing); measure_meta_get # (present, absent); measure_only_in (some, none); measure_changed (some, # none); measure_value (present, absent); measure_is_text (text, NUL, -# missing); measure_signature (text same, text changed line, text added -# line, binary same, binary offsets, binary length, missing side); # measure_set_minus (empty, residue); measure_sample_path; -# measure_state_verdict (text noise, text real by content, text real by -# an extra changed line, binary noise, binary real offset, binary real -# length, not sampled on one side, absent on one side); -# measure_state_evidence (text, binary, unsampled); measure_count. +# measure_nature_list (empty, every token kind, unknown, overflow); +# measure_lines; measure_count. The verdicts themselves are +# tests/compare. +# layer-report-lib: measure_preconditions (ok, missing, duplicate, epoch +# mismatch, epoch none, parent mismatch, a tampered state list); the +# attribute and compare blocks; the waiver section (digest, rules, +# covered counts, a stretched rule, a rule too broad, malformed fields, +# a max-position that does not reach); the subtracted section and the +# suspect net. # bt-layer-measure: --help; no subcommand; unknown subcommand; capture # without --dir, without --layer, without --tag, with a bad tag, with a # failing build, with no rootfs, ok, and refusing to overwrite a tag; @@ -450,17 +454,45 @@ assert_grep attr-evidence-for-noise "control vs unit" "$r" assert_grep attr-subtracted-named "./var/log/alternatives.log" "$r" info "15: a waiver in writing clears an overlapping path and is printed" +# rule, count, max-position, justified-against, reason cat > "$tmp/cleared" <<'EOF' # checked by hand: both are install-time database state -./var/lib/mysql/mysql/global_priv.MAD creation stamps, read byte by byte on 2026-09-28 -./var/lib/mysql/ibdata1 InnoDB page header, read byte by byte on 2026-09-28 +./var/lib/mysql/mysql/global_priv.MAD 1 64 fixtures, recipe deadbeef creation stamps, read byte by byte on 2026-09-28 +./var/lib/mysql/ibdata1 1 600 fixtures, recipe deadbeef InnoDB page header, read byte by byte on 2026-09-28 EOF r=$(attr u1 --cleared "$tmp/cleared") assert_grep attr-pass "verdict: PASS" "$r" -assert_grep attr-waiver-printed "waived in writing" "$r" +assert_grep attr-waiver-digest "waivers .*sha256" "$r" +assert_grep attr-waiver-rule "rule ./var/lib/mysql/ibdata1: 1 paths" "$r" assert_grep attr-waiver-reason "read byte by byte" "$r" +assert_grep attr-waiver-verdict "cleared .*global_priv.MAD" "$r" +assert_grep attr-waiver-against "justified against" "$r" +# a waiver is the one place a person overrides the instrument, so the bytes +# are printed for a cleared path too +assert_grep attr-waiver-evidence "sizes .* differing bytes" \ + "$(sed -n '/evidence, for every state path/,$p' <<<"$r" | grep -A3 'global_priv.MAD')" assert_status attr-pass-status 0 attr u1 --cleared "$tmp/cleared" +info "15b: a waiver that is too broad, stretched, or malformed" +printf '*\t1\t64\tnowhere\tanything at all\n' > "$tmp/broad" +assert_status waiver-broad 3 attr u1 --cleared "$tmp/broad" +printf './var/lib/mysql/**\tonly two fields\n' > "$tmp/short" +assert_status waiver-fields 3 attr u1 --cleared "$tmp/short" +printf './var/lib/mysql/**\tmany\t64\tnowhere\tnot a number\n' > "$tmp/nan" +assert_status waiver-count-nan 3 attr u1 --cleared "$tmp/nan" +# a glob written against one path that now covers two fails rather than +# quietly taking the new one in +printf './var/lib/mysql/**\t1\t99999\tfixtures\twritten when this matched one file\n' > "$tmp/stretch" +r=$(attr u1 --cleared "$tmp/stretch" || true) +assert_grep waiver-stretched "THIS RULE HAS STRETCHED" "$r" +assert_grep waiver-stretched-fail "verdict: FAIL" "$r" +assert_status waiver-stretched-status 1 attr u1 --cleared "$tmp/stretch" +# and a rule justified against a smaller max position does not cover a path +# that differs beyond it +printf './var/lib/mysql/ibdata1\t1\t4\tfixtures\tjustified only at the head of the file\n' > "$tmp/narrowmax" +r=$(attr u1 --cleared "$tmp/narrowmax" || true) +assert_grep waiver-maxpos "overlapping .*ibdata1" "$r" + info "16: a dirty unit, and a real difference inside the floor" assert_status attr-dirty 1 attr u2 --cleared "$tmp/cleared" r=$(attr u2 --cleared "$tmp/cleared" || true) @@ -476,6 +508,44 @@ assert_grep attr-real-row "mysql" "$r" # a waiver cannot wave away a real difference, only an overlapping one assert_grep attr-real-binary "real .*global_priv" "$r" +info "16b: a waiver over the database cannot green a hardcoded host key" +# the composition the review found: a waiver naming ./var/lib/mysql/** is +# exactly the route under discussion for this component, and with it the +# run must still fail on a unit that bakes in an SSH host key, because the +# controls vary that file at random and random is not a shape +randkey() { + printf 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI%s root@keel\n' \ + "$(head -c 24 /dev/urandom | base64 | tr -d '=\n/+' | head -c 22)" +} +for pair in "c1 kc1" "c2 kc2" "c3 kc3" "u1 ku1"; do + set -- $pair + rm -rf "$fix/$2"; cp -a "$fix/$1" "$fix/$2" + mkdir -p "$fix/$2/etc/ssh" + randkey > "$fix/$2/etc/ssh/ssh_host_ed25519_key.pub" +done +printf 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHARDCODEDKEYINTHEUNIT root@keel\n' \ + > "$fix/ku1/etc/ssh/ssh_host_ed25519_key.pub" +cat > "$tmp/state-plus" <<'EOF' +./etc/shadow +./var/lib/mysql/** +./etc/ssh/ssh_host_* +EOF +for t in kc1 kc2 kc3 ku1; do + echo "$t" > "$layers/NEXT" + MEASURE_STATE_PATHS=$tmp/state-plus "$M" capture --dir "$caps" --layer mariadb \ + --parent core --tag "$t" > /dev/null +done +cat > "$tmp/db-waiver" <<'EOF' +./var/lib/mysql/** 2 99999 fixtures, recipe deadbeef database creation stamps, read on 2026-09-28 +EOF +r=$("$M" attribute --dir "$caps" --unit ku1 --control kc1 --control-again kc2 \ + --control-again kc3 --cleared "$tmp/db-waiver" || true) +assert_grep compose-key-named "ssh_host_ed25519_key.pub" "$r" +assert_grep compose-key-fails "verdict: FAIL" "$r" +grep -qE '^ noise .*ssh_host' <<<"$r" && fatal "compose: a hardcoded host key came back as noise" +assert_status compose-status 1 "$M" attribute --dir "$caps" --unit ku1 \ + --control kc1 --control-again kc2 --control-again kc3 --cleared "$tmp/db-waiver" + info "17: blind because nothing was sampled" capture c1 s1 --state-max-bytes 8 > /dev/null capture c2 s2 --state-max-bytes 8 > /dev/null From b7aac6914a190edaf714b878b971cb32ebdc8c0e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Mon, 28 Sep 2026 08:15:06 +0000 Subject: [PATCH 5/9] fix: delete the control-produced exemption, and measure glob breadth properly Two findings from the third review, and the first was one line undoing the rule it sat inside. measure_mask_admits admitted a unit line identical to any control's line at that position, ahead of every pin, on the grounds that it was a value the recipe makes on its own. The base capture is the first control, so a value copied from the second or third differs from the base, is flagged as a differing position, and was then waved through. Against three controls: a unit shipping the host key control 2 produced came out noise, the machine-id control 3 produced came out noise, a replayed root hash came out noise, and a fresh random key at the same path came out overlapping. The safe case was the one that failed, which is the property this file exists to have, inverted. The rationale was true and did not support the conclusion. What makes a generated secret safe is that it is different every build, so matching a control is evidence of pinning rather than of freshness. docs/traps.md, "Every appliance built from core has the same machine-id", is that defect already recorded here. The early return is deleted and it cost nothing: a genuinely pinned field still passes on length and class, and two independent random values do not collide, so the branch only ever admitted replays. All four cases now come out overlapping, and tests/compare carries the three replays and the fresh-key control. MEASURE_WAIVER_MIN_LITERAL counted the literal characters of a glob, and ./etc/ss* has exactly the eight it required while covering an SSH host key and a TLS private key under one sentence; ./var/lib/* has ten and covers every state tree. A rule is now an exact path, or a named directory and its subtree written DIR/**, with at least two components below ./, and the paths it actually covers must sit under one immediate child of that directory. So *, ./etc/*, ./etc/ss*, ./var/lib/*, ./etc/** and ./var/lib/my*/** are all refused, and one sentence justifies one subtree. A directory is a thing somebody chose; a character count is not. Three more from the same round: justified-against was printed and never read, which is the shape state_paths_sha256 had two rounds ago. The capture directory it names is now compared with --dir and the run fails when they differ. share/layer-waivers.example set max-position to the sample cap, which is past the end of anything that was sampled and so bounds nothing, on the line a maintainer copies. The template now carries real observed maxima, one rule per file shape, and a value at or past the cap is refused. The abbreviation cap hid 115 of the 135 non-real paths, and noise passes, so the one remaining route to a bad noise landed where no bytes were printed. Only overlapping is abbreviated now. And the correction to the plan, which was the reviewer's and is right: a waiver is consulted only for overlapping, so granting one over the 32 real paths would leave them untouched and the run still FAIL. It is more control captures and then a waiver, not one or the other. docs/layer-measure.md says so now. shellcheck -S style exits 0, which it did not before: SC2115 and SC2086 in tests/measure. Coverage 204, 316, 264 and 71 lines, all four files at 100 percent, gate unchanged at 99. --- COVERAGE.md | 46 +++++++++++++++++++++++ bin/layer-compare-lib | 74 ++++++++++++++++++++++++++++++++----- bin/layer-report-lib | 27 ++++++++++++-- bt-layer-measure | 1 + docs/layer-measure.md | 41 +++++++++++++------- share/layer-waivers.example | 47 +++++++++++++++-------- tests/compare | 66 +++++++++++++++++++++++++++++---- tests/measure | 52 ++++++++++++++++++++------ 8 files changed, 293 insertions(+), 61 deletions(-) diff --git a/COVERAGE.md b/COVERAGE.md index 194b2eb..4bec15f 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -193,6 +193,52 @@ collapses to 9. `tests/compare` runs that case: the clock is `noise` and the payload is not, at either prefix length, because the criterion is now the field's width and class rather than how much framing survived. +## 2026-09-28: the exemption that inverted the property + +The third review found one line that undid the pinning rule it sat inside. +`measure_mask_admits` admitted a unit line identical to *any* control's line +at that position, ahead of every pin, on the grounds that it was "a value this +recipe makes on its own". The base capture is the first control, so a value +copied from the second or third differs from the base, is flagged as a +differing position, and was then waved through. Verified against three +controls: a unit shipping the host key control 2 produced came out `noise`, +the machine-id control 3 produced came out `noise`, a replayed root hash came +out `noise`, and a *fresh random* key at the same path came out +`overlapping`. The safe case was the one that failed. + +The rationale was true and did not support the conclusion. What makes a +generated secret safe is that it is different every build, so matching a +control is evidence of pinning rather than of freshness; `docs/traps.md`, +"Every appliance built from `core` has the same machine-id", is that defect +already recorded in this project. The early return is deleted. It cost +nothing: a genuinely pinned field still passes on length and class, and two +independent random values do not collide, so the branch only ever admitted +replays. `tests/compare` carries the three replays and the fresh-key control. + +Also in that round, and each of them a measure that was easier to write than +to defend: + +`MEASURE_WAIVER_MIN_LITERAL` counted the literal characters of a glob, and +`./etc/ss*` has exactly the eight it required while covering an SSH host key +and a TLS private key together. A rule is now an exact path or a named +directory's subtree written `DIR/**`, with at least two components below +`./`, and the paths it actually covers must sit under one immediate child of +that directory. A directory is a thing somebody chose; a character count is +not. + +`justified-against` was printed and never read, which is the shape +`state_paths_sha256` had two rounds ago. The capture directory it names is now +compared with `--dir` and the run fails when they differ. + +`share/layer-waivers.example` set `max-position` to the sample cap, which is +past the end of anything that was sampled and so bounds nothing, on the line +a maintainer copies. The template now carries real observed maxima, one rule +per file shape, and a value at or past the cap is refused. + +The abbreviation cap hid 115 of the 135 non-`real` paths, and `noise` passes, +so the one remaining route to a bad `noise` landed where no bytes were +printed. Only `overlapping` is abbreviated now. + ## 2026-09-28: pipefail in the three older suites `tests/layer`, `tests/aplinfo` and `tests/signature` did not set `pipefail` diff --git a/bin/layer-compare-lib b/bin/layer-compare-lib index 727bf51..052bb58 100644 --- a/bin/layer-compare-lib +++ b/bin/layer-compare-lib @@ -257,8 +257,16 @@ measure_free_region() { # a free region is longer than MEASURE_FREE_MAX, or carries a character # outside MEASURE_FREE_CLASS or outside what the controls themselves used. # -# A LINE a control itself produced is pinned by observation and admitted -# whatever its shape, because it is a value this recipe makes on its own. +# There is deliberately no exemption for a line a control itself produced. +# There was one, and it inverted the property this whole file exists to +# have: the base capture is the first control, so a value copied from the +# second or third differs from the base, reaches here, and was waved +# through, while a fresh random value at the same path was flagged. The safe +# case was the one that failed. What makes a generated secret safe is that +# it is different every build, so matching a control is evidence of pinning +# rather than of freshness; docs/traps.md, "Every appliance built from core +# has the same machine-id", is that defect already recorded. A genuinely +# pinned field still passes below, on length and class, like any other. measure_mask_admits() { local line=$1 prefix=$2 suffix=$3 shift 3 @@ -266,7 +274,6 @@ measure_mask_admits() { for v in "$@"; do [[ -n "$first" ]] || first=$v [[ "$v" == "$first" ]] || same=no - [[ "$line" != "$v" ]] || return 0 done [[ "$same" == no ]] || return 1 [[ -n "$prefix" ]] || [[ -n "$suffix" ]] || return 1 @@ -360,8 +367,17 @@ measure_free_pinned() { # overrides the instrument. MEASURE_WAIVER_FIELDS=5 -# The least literal path a glob rule has to carry. "*" matches ./etc/shadow. -MEASURE_WAIVER_MIN_LITERAL=8 +# A rule is an exact path, or a named directory and its whole subtree +# written as DIR/**, and nothing else. Counting the literal characters of a +# glob was the previous test and it was the wrong measure: ./etc/ss* has +# eight of them and waives an SSH host key and a TLS private key together. +# A directory is a thing somebody chose; a character count is not. +# +# The directory also has to be at least this many components below ./, so +# ./etc/** and ./** cannot be written, and the paths a rule actually covers +# have to sit under one immediate child of it, which is what "one rule per +# subtree" means and is checked against the run in bin/layer-report-lib. +MEASURE_WAIVER_MIN_DEPTH=2 # measure_waiver_rules FILE -> the rules, one per line, comments gone measure_waiver_rules() { @@ -369,9 +385,12 @@ measure_waiver_rules() { sed -e '/^[[:space:]]*#/d' -e '/^[[:space:]]*$/d' "$1" } -# measure_waiver_check FILE +# measure_waiver_check FILE [SAMPLE_CAP] +# SAMPLE_CAP is the capture's --state-max-bytes. A max-position at or past +# it bounds nothing, because nothing larger was ever sampled, and it is the +# figure a template invites somebody to copy. measure_waiver_check() { - local file=$1 line rule count max rest literal rc=0 n=0 + local file=$1 cap=${2:-0} line rule count max rest rc=0 n=0 [[ -f "$file" ]] || return 0 local -a lines=() mapfile -t lines < <(measure_waiver_rules "$file") @@ -383,9 +402,8 @@ measure_waiver_check() { rc=1 continue fi - literal=${rule//[*?]/} - if [[ ${#literal} -lt $MEASURE_WAIVER_MIN_LITERAL ]]; then - measure_error "$file line $n: '$rule' is too broad to be a decision about anywhere; a rule needs at least $MEASURE_WAIVER_MIN_LITERAL characters of literal path" + if ! measure_waiver_rule_shape "$rule"; then + measure_error "$file line $n: '$rule' is not an exact path nor a named directory's subtree; write either ./a/b/c or ./a/b/** with at least $MEASURE_WAIVER_MIN_DEPTH components below ./, so that what it covers is a directory somebody chose" rc=1 fi if [[ ! "$count" =~ ^[0-9]+$ ]] || [[ ! "$max" =~ ^[0-9]+$ ]]; then @@ -393,10 +411,46 @@ measure_waiver_check() { rc=1 fi [[ -n "$rest" ]] || { measure_error "$file line $n: no justification"; rc=1; } + if [[ "$cap" -gt 0 ]] && [[ "$max" =~ ^[0-9]+$ ]] && [[ "$max" -ge "$cap" ]]; then + measure_error "$file line $n: max-position $max is at or past the sample cap $cap, so it bounds nothing; use the largest offset or line the block actually reported for that path" + rc=1 + fi done return $rc } +# measure_waiver_rule_shape RULE +# An exact path with no wildcard at all, or DIR/** where DIR has none. +measure_waiver_rule_shape() { + local rule=$1 dir + [[ "$rule" == ./* ]] || return 1 + if [[ "$rule" != *[*?]* ]]; then + return 0 + fi + [[ "$rule" == */'**' ]] || return 1 + dir=${rule%/'**'} + [[ "$dir" != *[*?]* ]] || return 1 + [[ "$(measure_waiver_depth "$dir")" -ge "$MEASURE_WAIVER_MIN_DEPTH" ]] || return 1 + return 0 +} + +# measure_waiver_depth DIR -> how many components it names below ./ +measure_waiver_depth() { + local d=${1#./} + d=${d%/} + if [[ -z "$d" ]]; then + echo 0 + return 0 + fi + awk -F/ '{ print NF }' <<<"$d" +} + +# measure_waiver_prefix RULE -> the directory a subtree rule names, or nothing +measure_waiver_prefix() { + [[ "$1" == */'**' ]] || return 0 + printf '%s\n' "${1%/'**'}" +} + # measure_cleared_rule FILE PATH MAXPOS -> the whole rule line that covers # PATH, or nothing. MAXPOS is the furthest this path actually differs at; a # rule justified against less than that does not cover it. diff --git a/bin/layer-report-lib b/bin/layer-report-lib index 4525b8c..4e49101 100644 --- a/bin/layer-report-lib +++ b/bin/layer-report-lib @@ -353,7 +353,10 @@ measure_state_section() { fi # a waiver is the one place a person overrides the instrument, so it # is the last place to withhold the bytes - if [[ "$verdict" == real ]] || [[ -n "$rule" ]] || [[ $shown -lt $MEASURE_EVIDENCE_PATHS ]]; then + # `overlapping` is named in the totals and fails the run, so it is + # the only column the cap may abbreviate: `noise` passes, and a + # mistaken `noise` is the one error this scheme can still make + if [[ "$verdict" != overlapping ]] || [[ -n "$rule" ]] || [[ $shown -lt $MEASURE_EVIDENCE_PATHS ]]; then shown=$((shown + 1)) evidence="$evidence$(measure_evidence "$path" "$(measure_sample_path "$dir/$unit" "$path")" "${samples[@]}")"$'\n' else @@ -367,7 +370,7 @@ measure_state_section() { printf ' totals: noise %s, overlapping %s, real %s, not sampled %s, unchanged %s, cleared %s\n' \ "$noise" "$overlapping" "$real" "$unsampled" "$none" "$waived" local stretched=0 - measure_waiver_section "$cleared" "$waivers" "$waived" || stretched=1 + measure_waiver_section "$cleared" "$waivers" "$waived" "$dir" || stretched=1 if [[ -n "$evidence" ]]; then echo echo "evidence, for every state path above:" @@ -424,7 +427,7 @@ measure_subtracted_section() { # written against fails the run, so a new file appearing in a waived # directory cannot be swept in without anybody noticing. measure_waiver_section() { - local file=$1 waivers=$2 waived=$3 rule count covered rc=0 + local file=$1 waivers=$2 waived=$3 dir=$4 rule count covered rc=0 prefix kids echo if [[ ! -f "$file" ]]; then echo "waivers: none" @@ -442,6 +445,24 @@ measure_waiver_section() { "$(cut -f1 <<<"$rule")" "$covered" "$count" "$(cut -f3 <<<"$rule")" printf ' against %s: %s\n' "$(cut -f4 <<<"$rule")" \ "$(measure_cleared_reason "$rule")" + # it cannot outlive its evidence: the captures it was read against + # have to be the captures being measured + if [[ "$(cut -f4 <<<"$rule")" != *"$dir"* ]]; then + measure_error "the waiver for $(cut -f1 <<<"$rule") was justified against '$(cut -f4 <<<"$rule")', which does not name $dir; a waiver is a reading of particular captures and does not carry to others" + echo " THIS RULE NAMES OTHER CAPTURES: $dir is not in its justification" + rc=1 + fi + # one rule per subtree: everything it covered has to sit under one + # immediate child of the directory it names + prefix=$(measure_waiver_prefix "$(cut -f1 <<<"$rule")") + if [[ -n "$prefix" ]]; then + kids=$(LC_ALL=C awk -F'\t' -v r="$(cut -f1 <<<"$rule")" -v p="$prefix/" '$2 == r { s = substr($1, length(p) + 1); sub(/\/.*$/, "", s); print s }' <<<"$waivers" | LC_ALL=C sort -u | grep -c . || true) + if [[ "$kids" -gt 1 ]]; then + measure_error "the waiver for $(cut -f1 <<<"$rule") covers $kids different subtrees of $prefix; write one rule per subtree so that one sentence justifies one thing" + echo " THIS RULE SPANS $kids SUBTREES of $prefix" + rc=1 + fi + fi if [[ "$covered" != "$count" ]]; then measure_error "the waiver for $(cut -f1 <<<"$rule") was written against $count paths and covers $covered here; read the difference and rewrite it" echo " THIS RULE HAS STRETCHED: written against $count, covers $covered" diff --git a/bt-layer-measure b/bt-layer-measure index ca978bb..762b219 100755 --- a/bt-layer-measure +++ b/bt-layer-measure @@ -239,6 +239,7 @@ do_attribute() { "$MEASURE_CONTROL" "${MEASURE_CONTROL2[@]}" \ || die "$EXIT_INPUT" "the captures cannot be compared" measure_waiver_check "${MEASURE_CLEARED:-/nonexistent}" \ + "$(measure_meta_get "$MEASURE_DIR/$MEASURE_CONTROL" state_max_bytes)" \ || die "$EXIT_INPUT" "the waiver file is not usable" measure_attribute_report "$MEASURE_DIR" "$MEASURE_UNIT" \ "${MEASURE_CLEARED:-/nonexistent}" "$(suspect_paths_file)" \ diff --git a/docs/layer-measure.md b/docs/layer-measure.md index 40b3082..6155464 100644 --- a/docs/layer-measure.md +++ b/docs/layer-measure.md @@ -184,8 +184,15 @@ field carries no more shape than a random byte field. Install-time text state varies by clocks, counters and log sequence numbers. A key, a hash, a hostname or a SQL fragment is not one of those. -A line a control itself produced is admitted whatever its shape, because it is -a value this recipe makes on its own. +There is deliberately no exemption for a line a control itself produced. There +was one, and it inverted the property the tool exists to have: the base capture +is the first control, so a value copied from the second or third differs from +the base, reaches the mask, and was waved through, while a *fresh* random value +at the same path was flagged. The safe case was the one that failed. What makes +a generated secret safe is that it is different every build, so matching a +control is evidence of pinning rather than of freshness. `docs/traps.md`, +"Every appliance built from `core` has the same machine-id", is that defect +already recorded. One consequence worth having: the verdict no longer depends on when the gate ran. `mysql/user.frm`'s mask is 21 characters of prefix only because four @@ -225,17 +232,18 @@ template. | Field | What it is for | |---|---| -| `rule` | an exact path, or a glob carrying enough literal path to be a decision about somewhere. `*` is refused: it matches `./etc/shadow` | +| `rule` | an exact path, or a named directory and its subtree as `DIR/**`, and nothing else. A glob that can reach a second subtree is refused, and so is one naming fewer than two components below `./`: counting literal characters said `./etc/ss*` was specific enough to waive an SSH host key and a TLS private key under one sentence | | `count` | how many differing state paths it covered when written. A run where it covers a different number **fails**, so a new file appearing in a waived directory cannot be swept in silently | -| `max-position` | the highest differing byte offset or line number it was justified against. A covered path differing beyond it is **not** waived | -| `justified-against` | the capture directory and recipe commit it was read against, so it cannot outlive its evidence | +| `max-position` | the largest position actually reported: a line number for text, a byte offset for a binary. A covered path differing beyond it is **not** waived, and a value at or past the sample cap is refused, because it bounds nothing | +| `justified-against` | the capture directory and recipe commit it was read against. The directory is compared with `--dir` and the run fails if it does not match, so it cannot outlive its evidence | | `reason` | what was read, and what it was found to be | A waiver clears `overlapping` and nothing else: it cannot clear a `real` -difference. The block prints the file's digest, every rule, how many paths -each one covered against how many it was written for, and **every cleared -path with its bytes**. A waiver is the one place a person overrides the -instrument, so it is the last place to withhold the evidence. +difference, and it is not a route to PASS on its own. The block prints the +file's digest, every rule, how many paths each one covered against how many it +was written for, and **every cleared path with its bytes**. A waiver is the one +place a person overrides the instrument, so it is the last place to withhold +the evidence. ### What the floor removed @@ -299,10 +307,17 @@ difference between builds is one `timestamp=` line that the mask the three controls establish there admits. This FAIL is the honest state of that measurement and not a defect in the -component. Reaching PASS needs a decision that is not the tool's to make: -either more control captures, until the floor spans the counter's carry, or a -committed `--cleared` waiver saying the Aria header clock was read and -accepted. Both are on the record; silently subtracting 167 paths was not. +component. Reaching PASS takes two steps in order, and a waiver alone is not +one of them: a waiver is consulted only for `overlapping`, and these 32 are +`real`, so granting one would turn `FAIL (32 real, 33 overlapping)` into +`FAIL (32 real, 0 overlapping)` and leave the 32 untouched. + +So it is more control captures **and then** a waiver. More controls, until the +floor spans the Aria counter's carry, is the only lever that moves those 32 +from `real` to `overlapping`; only then is there anything for a waiver to +clear. That the lever is luck about where the builds fall is a fair objection, +and it is still the only one available, because teaching the tool to recognise +a monotone run and downgrade it was declined on purpose. ## Layout diff --git a/share/layer-waivers.example b/share/layer-waivers.example index da7bc83..849e2ae 100644 --- a/share/layer-waivers.example +++ b/share/layer-waivers.example @@ -4,24 +4,39 @@ # # A waiver clears `overlapping`, which is a difference that coincides with # what the control builds vary but cannot be shown to be that variation. It -# can never clear `real`. It is the one place a person overrides the -# instrument, so it has to say what was read and what it was found to be. +# can never clear `real`, and it is not a route to PASS on its own: a `real` +# path stays `real` and the run stays FAIL. It is the one place a person +# overrides the instrument, so it has to say what was read and what it was +# found to be. # # Five tab separated fields: # -# rule an exact path, or a glob carrying enough literal path -# to be a decision about somewhere. "*" is refused -# count how many differing state paths it covered when it was -# written. A run where it covers a different number -# fails, so a new file appearing in a waived directory -# cannot be swept in silently -# max-position the highest differing byte offset or line number it -# was justified against. A covered path differing -# beyond it is not waived, so the same path turning -# overlapping somewhere else is not quietly included -# justified-against the capture directory and the recipe commit it was -# read against, so it cannot outlive its evidence -# reason what was read, and what it was found to be +# rule an exact path, or a named directory and its whole +# subtree written DIR/**. Nothing else: a glob that can +# reach a second subtree is refused, and so is one that +# names fewer than two components below ./, because +# ./etc/ss* covers an SSH host key and a TLS private +# key under one sentence. One rule per subtree. +# count how many `overlapping` paths it covered in the run it +# was written against, read off that run's own block +# and not estimated. A run where it covers a different +# number fails, so a new file appearing in a waived +# directory cannot be swept in silently. +# max-position the largest position actually reported for those +# paths: a line number for a text file, a byte offset +# for a binary. NEVER the sample cap, which is past the +# end of anything and so bounds nothing; a value at or +# past it is refused. One rule per file shape, because +# one number cannot bound two of them. +# justified-against the capture directory this was read against, and the +# recipe commit. The directory is compared with --dir +# and the run fails if it does not match, so a waiver +# cannot outlive its evidence. +# reason what was read, and what it was found to be. +# +# The numbers below are from one particular run and are examples, not values +# to copy: take yours from your own block. # # rule count max-position justified-against reason -./var/lib/mysql/** 33 16777216 /mnt/builds/measure/mariadb-gate, recipe 659eeb4 Aria and InnoDB header timestamps; the six accounts and their authentication_string values are identical in all four captures, only password_last_changed moves. Read 2026-09-28. +./var/lib/mysql/mysql/global_priv.MAD 1 16384 /mnt/builds/measure/mariadb-gate, recipe 659eeb4 Aria data page header timestamps. The six accounts and their authentication_string values are identical in all four captures; only password_last_changed moves, 1790574121 to 1790574610, climbing with capture time. Read 2026-09-28. +./var/lib/mysql/mysql/global_priv.frm 1 76 /mnt/builds/measure/mariadb-gate, recipe 659eeb4 .frm create and update timestamps at fixed offsets 67-76, the same offsets every control pair moves. Read 2026-09-28. diff --git a/tests/compare b/tests/compare index 20ec4df..8132bcc 100755 --- a/tests/compare +++ b/tests/compare @@ -229,9 +229,15 @@ assert_fail admits-class measure_mask_admits "# written abcd" "# written " "" \ # controls that agree at a line establish no varying shape at all assert_fail admits-noshape measure_mask_admits "# written 4000" "# written 1000" "" \ "# written 1000" "# written 1000" -# a unit line one of the controls itself produced is pinned by observation -assert_ok admits-observed measure_mask_admits "# written 2000" "# written " "000" \ +# a unit line one of the controls produced is NOT admitted for being +# observed: a generated secret is safe because it is different every build, +# so matching a control is evidence of pinning, not of freshness. It is +# admitted here only because a one digit decimal is pinned by length and +# class like any other +assert_ok admits-pinned-digit measure_mask_admits "# written 2000" "# written " "000" \ "# written 1000" "# written 2000" +assert_fail admits-replay measure_mask_admits "k=AAAABBBBCCCC" "k=" "" \ + "k=AAAABBBBCCCC" "k=DDDDEEEEFFFF" # and a free region longer than the cap is never pinned long=$(head -c 40 /dev/zero | tr '\0' '7') assert_fail admits-toolong measure_mask_admits "t=$long" "t=" "" "t=1111111111" "t=2222222222" @@ -360,6 +366,36 @@ not_noise attack-userfrm "$tmp/vunit" "$tmp/v1" "$tmp/v2" "$tmp/v3" >/dev/null printf 'TYPE=VIEW\ntimestamp=0001790574106902139\n' > "$tmp/vclock" verdict clock-still-noise noise "$tmp/vclock" "$tmp/v1" "$tmp/v2" "$tmp/v3" >/dev/null +info "16c2: a value copied from a control is a replay, not freshness" +# docs/traps.md, "Every appliance built from core has the same machine-id": +# a generated value baked into a layer is a defect, not a licence. The unit +# ships, byte for byte, what the second or third control produced, which +# differs from the base capture and so reaches the mask +replay() { printf '%s\n' "$2" > "$1"; } +replay "$tmp/rk1" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIRANDOMONEEEEEEEEEEE root@keel" +replay "$tmp/rk2" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIRANDOMTWOOOOOOOOOOO root@keel" +replay "$tmp/rk3" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIRANDOMTHREEEEEEEEEE root@keel" +cp "$tmp/rk2" "$tmp/rkunit" +not_noise replay-hostkey "$tmp/rkunit" "$tmp/rk1" "$tmp/rk2" "$tmp/rk3" >/dev/null +replay "$tmp/rm1" "1f0c1d4a6b7e4f0a9c2d3e4f5a6b7c8d" +replay "$tmp/rm2" "2a1b2c3d4e5f60718293a4b5c6d7e8f9" +replay "$tmp/rm3" "3b2c3d4e5f60718293a4b5c6d7e8f9a0" +cp "$tmp/rm3" "$tmp/rmunit" +not_noise replay-machineid "$tmp/rmunit" "$tmp/rm1" "$tmp/rm2" "$tmp/rm3" >/dev/null +# shellcheck disable=SC2016 # literal crypt prefixes, not expansions +i=0 +for h in Kq7xPnZa2WdLbTvE Rm3JcYs8HgUpNfXo Zt9BwQe4VkAiDrLs; do + i=$((i + 1)) + # shellcheck disable=SC2016 # a literal crypt prefix, not an expansion + printf 'root:$y$j9T$%s:20000:0:99999:7:::\n' "$h" > "$tmp/rs$i" +done +cp "$tmp/rs2" "$tmp/rsunit" +not_noise replay-shadow "$tmp/rsunit" "$tmp/rs1" "$tmp/rs2" "$tmp/rs3" >/dev/null +# and a fresh random value at the same path is flagged too, so the safe case +# is no longer the only one that is +replay "$tmp/rkfresh" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIRANDOMFOURRRRRRRRRR root@keel" +not_noise fresh-hostkey "$tmp/rkfresh" "$tmp/rk1" "$tmp/rk2" "$tmp/rk3" >/dev/null + info "16d: the verdict does not depend on where the builds fell" # the review's point: the user.frm mask was 21 characters of prefix only # because four builds landed inside the same thousand seconds, and a @@ -399,11 +435,17 @@ assert_eq cleared-nofile "$(measure_cleared_rule "$tmp/absent" ./etc/shadow 1)" # sentence, and the report names every path it covered assert_grep cleared-glob "initdb output" \ "$(measure_cleared_reason "$(measure_cleared_rule "$tmp/cleared" ./var/lib/postgresql/17/main/PG_VERSION 10)")" -# a rule with too little literal path is a decision about nowhere -printf '*\t1\t1\tnowhere\tanything\n' > "$tmp/broad" -assert_fail waiver-broad measure_waiver_check "$tmp/broad" -printf './etc/*\t1\t1\tnowhere\tstill too broad\n' > "$tmp/broad2" -assert_fail waiver-broad2 measure_waiver_check "$tmp/broad2" +# a glob is an exact path or a named directory and its subtree, and nothing +# else: counting literal characters said ./etc/ss* was specific enough to +# waive an SSH host key and a TLS private key under one sentence +for bad in '*' './etc/*' './etc/ss*' './var/lib/*' './var/lib/my*/**' './etc/**' './**' './var/lib/mysql/*.MAD'; do + printf '%s\t1\t1\tnowhere\ttoo broad or not a subtree\n' "$bad" > "$tmp/broad" + assert_fail "waiver-broad-$bad" measure_waiver_check "$tmp/broad" +done +for good in './var/lib/mysql/**' './var/lib/mysql/mysql/global_priv.MAD' './etc/ssh/**'; do + printf '%s\t1\t1\tnowhere\ta named subtree or an exact path\n' "$good" > "$tmp/ok" + assert_ok "waiver-ok-$good" measure_waiver_check "$tmp/ok" +done printf './var/lib/mysql/**\ttwo fields\n' > "$tmp/short" assert_fail waiver-fields measure_waiver_check "$tmp/short" printf './var/lib/mysql/**\tmany\t1\tnowhere\tcount is not a number\n' > "$tmp/nan" @@ -417,6 +459,16 @@ assert_grep free-summary-num "a number or a timestamp" \ assert_grep free-summary-not "not a number" \ "$(measure_free_summary "t=abcd" "t=" "")" +# the shape helpers, and the cap check the executable passes a cap to +assert_eq depth-none "$(measure_waiver_depth ./)" 0 +assert_eq depth-two "$(measure_waiver_depth ./var/lib)" 2 +assert_eq prefix-subtree "$(measure_waiver_prefix './var/lib/mysql/**')" "./var/lib/mysql" +assert_eq prefix-exact "$(measure_waiver_prefix ./var/lib/mysql/ibdata1)" "" +printf './var/lib/mysql/**\t1\t16777216\there\tthe sample cap bounds nothing\n' > "$tmp/capmax" +assert_ok capmax-nocap measure_waiver_check "$tmp/capmax" +assert_fail capmax-withcap measure_waiver_check "$tmp/capmax" 16777216 +assert_ok capmax-under measure_waiver_check "$tmp/capmax" 16777217 + info "17b: measure_max_position" assert_eq maxpos-text "$(measure_max_position "$tmp/c2" "$tmp/c1")" 1 assert_eq maxpos-binary "$(measure_max_position "$tmp/bb" "$tmp/ba")" 3 diff --git a/tests/measure b/tests/measure index e352d4a..76fa521 100755 --- a/tests/measure +++ b/tests/measure @@ -454,11 +454,12 @@ assert_grep attr-evidence-for-noise "control vs unit" "$r" assert_grep attr-subtracted-named "./var/log/alternatives.log" "$r" info "15: a waiver in writing clears an overlapping path and is printed" -# rule, count, max-position, justified-against, reason -cat > "$tmp/cleared" <<'EOF' +# rule, count, max-position, justified-against, reason. The justification +# has to name the captures being measured, so a waiver cannot outlive them. +cat > "$tmp/cleared" < "$tmp/broad" assert_status waiver-broad 3 attr u1 --cleared "$tmp/broad" +printf './etc/ss*\t1\t64\tnowhere\tan SSH key and a TLS key under one sentence\n' > "$tmp/ss" +assert_status waiver-ss 3 attr u1 --cleared "$tmp/ss" printf './var/lib/mysql/**\tonly two fields\n' > "$tmp/short" assert_status waiver-fields 3 attr u1 --cleared "$tmp/short" printf './var/lib/mysql/**\tmany\t64\tnowhere\tnot a number\n' > "$tmp/nan" assert_status waiver-count-nan 3 attr u1 --cleared "$tmp/nan" +# a max-position at or past the sample cap bounds nothing, and it is the +# figure a template invites somebody to copy +printf './var/lib/mysql/**\t1\t16777216\t%s\tthe sample cap, which bounds nothing\n' "$caps" > "$tmp/capmax" +assert_status waiver-capmax 3 attr u1 --cleared "$tmp/capmax" +# a waiver read against other captures does not carry to these +printf './var/lib/mysql/**\t2\t600\t/somewhere/else, recipe deadbeef\tread elsewhere\n' > "$tmp/elsewhere" +r=$(attr u1 --cleared "$tmp/elsewhere" || true) +assert_grep waiver-elsewhere "THIS RULE NAMES OTHER CAPTURES" "$r" +assert_status waiver-elsewhere-status 1 attr u1 --cleared "$tmp/elsewhere" # a glob written against one path that now covers two fails rather than # quietly taking the new one in printf './var/lib/mysql/**\t1\t99999\tfixtures\twritten when this matched one file\n' > "$tmp/stretch" @@ -494,8 +506,12 @@ r=$(attr u1 --cleared "$tmp/narrowmax" || true) assert_grep waiver-maxpos "overlapping .*ibdata1" "$r" info "16: a dirty unit, and a real difference inside the floor" -assert_status attr-dirty 1 attr u2 --cleared "$tmp/cleared" -r=$(attr u2 --cleared "$tmp/cleared" || true) +# the dirty unit turns global_priv.MAD real, which no waiver can clear, so +# its waiver names only the path that is still overlapping +printf './var/lib/mysql/ibdata1\t1\t600\t%s, recipe deadbeef\tInnoDB page header\n' \ + "$caps" > "$tmp/cleared-u2" +assert_status attr-dirty 1 attr u2 --cleared "$tmp/cleared-u2" +r=$(attr u2 --cleared "$tmp/cleared-u2" || true) assert_grep attr-dirty-fail "verdict: FAIL" "$r" assert_grep attr-attributable "./etc/keel/component.conf" "$r" assert_grep attr-added "./tmp/new-file" "$r" @@ -507,6 +523,18 @@ assert_grep attr-real-shadow "real .*/etc/shadow" "$r" assert_grep attr-real-row "mysql" "$r" # a waiver cannot wave away a real difference, only an overlapping one assert_grep attr-real-binary "real .*global_priv" "$r" +grep -q 'cleared .*global_priv' <<<"$r" && fatal "a waiver cleared a real difference" + +info "15c: one rule per subtree" +# a rule naming a directory whose covered paths sit under two different +# children of it is one sentence justifying two things +printf './var/lib/**\t2\t600\t%s\tone rule spanning two subtrees\n' "$caps" > "$tmp/twosub" +two=$(printf './var/lib/mysql/ibdata1\t./var/lib/**\n./var/lib/redis/dump.rdb\t./var/lib/**\n') +r=$(measure_waiver_section "$tmp/twosub" "$two" 2 "$caps" 2>&1 || true) +assert_grep waiver-twosub "SPANS 2 SUBTREES" "$r" +one=$(printf './var/lib/mysql/ibdata1\t./var/lib/**\n./var/lib/mysql/x\t./var/lib/**\n') +r=$(measure_waiver_section "$tmp/twosub" "$one" 2 "$caps" 2>&1) +assert_grep waiver-onesub "rule ./var/lib/\*\*: 2 paths" "$r" info "16b: a waiver over the database cannot green a hardcoded host key" # the composition the review found: a waiver naming ./var/lib/mysql/** is @@ -518,10 +546,10 @@ randkey() { "$(head -c 24 /dev/urandom | base64 | tr -d '=\n/+' | head -c 22)" } for pair in "c1 kc1" "c2 kc2" "c3 kc3" "u1 ku1"; do - set -- $pair - rm -rf "$fix/$2"; cp -a "$fix/$1" "$fix/$2" - mkdir -p "$fix/$2/etc/ssh" - randkey > "$fix/$2/etc/ssh/ssh_host_ed25519_key.pub" + read -r from to <<<"$pair" + rm -rf "${fix:?}/${to:?}"; cp -a "$fix/$from" "$fix/$to" + mkdir -p "$fix/$to/etc/ssh" + randkey > "$fix/$to/etc/ssh/ssh_host_ed25519_key.pub" done printf 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHARDCODEDKEYINTHEUNIT root@keel\n' \ > "$fix/ku1/etc/ssh/ssh_host_ed25519_key.pub" @@ -535,8 +563,8 @@ for t in kc1 kc2 kc3 ku1; do MEASURE_STATE_PATHS=$tmp/state-plus "$M" capture --dir "$caps" --layer mariadb \ --parent core --tag "$t" > /dev/null done -cat > "$tmp/db-waiver" <<'EOF' -./var/lib/mysql/** 2 99999 fixtures, recipe deadbeef database creation stamps, read on 2026-09-28 +cat > "$tmp/db-waiver" < Date: Mon, 28 Sep 2026 08:51:54 +0000 Subject: [PATCH 6/9] fix: license a waiver's directory from the state path list Two measures of how broad a waiver rule is have now been tried and both were wrong the same way, by being easier to write than to defend. Counting the literal characters of the glob let ./etc/ss* through, which waives an SSH host key and a TLS private key under one sentence. Counting its components below ./ let ./var/lib/** through, and ./var/lib is the common parent of the six subtrees share/layer-state-paths declares separately: mysql, postgresql, redis, couchdb, mongodb, nodebb. The "one immediate child" check only sees the run in front of it, so such a rule passes whenever a single service's state differs, and a justification about MariaDB accounts clears PostgreSQL state in the next run. The count check does not catch it because the counts coincide, and that is not luck, because the same component shape produces the same number of state files. So the measure is now the project's own statement of what state is. A glob rule's directory has to be a directory the capture's own state path list declares, or below one. ./var/lib/mysql/** is allowed because the list names that directory, ./var/lib/** is refused because nothing names ./var/lib, and ./etc/ssh/** stays allowed because ./etc/ssh/ssh_host_* names it. A glob with no wildcard in its last component declares a file and not a directory, so ./etc/shadow does not license a waiver over ./etc, and ./etc/ssl/** is refused where ./etc/ssl/private/** is the declaration. ./var/cache/**, ./usr/share/** and ./etc/webmin/** are refused too, being declared nowhere. MEASURE_WAIVER_MIN_DEPTH is gone. The list is already compared byte for byte across the captures and its digest already checked, so this adds nothing new to trust. justified-against was a substring test, so a waiver justified against mariadb-gate2 was accepted for a run on mariadb-gate. Its first comma separated component is now compared for equality. docs/layer-measure.md gains a section on what this cannot see, which the review drew out of the one remaining comparison against a control, the cmp -s for the `none` verdict. State identical in every capture is invisible to a differential measurement by construction, including a secret copied out of the base control's own build tree, and the suspect net does not cover it either because such a path never differs and so is never subtracted. A PASS means nothing the change did shows up as a difference from the control; it does not mean the layer ships no secret it should not. --- COVERAGE.md | 31 ++++++++++++++-- bin/layer-compare-lib | 84 +++++++++++++++++++++++++++---------------- bin/layer-report-lib | 5 ++- bt-layer-measure | 1 + docs/layer-measure.md | 21 +++++++++-- tests/compare | 46 ++++++++++++++++-------- tests/measure | 6 ++++ 7 files changed, 144 insertions(+), 50 deletions(-) diff --git a/COVERAGE.md b/COVERAGE.md index 4bec15f..4d46259 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -132,8 +132,8 @@ measures ten files: | bin/signature-lib | 76 of 76 | 100 percent | | bin/generate-signature | 87 of 87 | 100 percent | | bin/layer-measure-lib | 204 of 204 | 100 percent | -| bin/layer-compare-lib | 298 of 298 | 100 percent | -| bin/layer-report-lib | 253 of 253 | 100 percent | +| bin/layer-compare-lib | 322 of 322 | 100 percent | +| bin/layer-report-lib | 264 of 264 | 100 percent | | bt-layer-measure | 71 of 71 | 100 percent | The gate stays at 99, the lowest file rounded down. All four new files are at @@ -239,6 +239,33 @@ The abbreviation cap hid 115 of the 135 non-`real` paths, and `noise` passes, so the one remaining route to a bad `noise` landed where no bytes were printed. Only `overlapping` is abbreviated now. +## 2026-09-28: the third measure of glob breadth + +Two measures of how broad a waiver rule is were tried and both were wrong the +same way, by being easier to write than to defend. Counting the literal +characters of the glob let `./etc/ss*` through, which waives an SSH host key +and a TLS private key under one sentence. Counting its components below `./` +let `./var/lib/**` through, which is the common parent of the six subtrees +`share/layer-state-paths` declares separately, so a justification about MariaDB +accounts could clear PostgreSQL state in a later run whose count happened to +match; and the counts coinciding is not luck, because the same component shape +produces the same number of state files. + +The measure is now the project's own statement of what state is: a glob rule's +directory has to be a directory the capture's state path list declares, or +below one. `./var/lib/mysql/**` is allowed because the list names that +directory, `./var/lib/**` is refused because nothing names `./var/lib`, and +`./etc/ssh/**` stays allowed because `./etc/ssh/ssh_host_*` names it. A glob +with no wildcard in its last component declares a file and no directory, so +`./etc/shadow` does not license a waiver over `./etc`. `MEASURE_WAIVER_MIN_DEPTH` +is gone, and the waiver vocabulary is the vocabulary of the thing being waived. +The list is already digest checked across the captures, so this adds no new +thing to trust. + +`justified-against` was a substring test, which accepted a waiver justified +against `mariadb-gate2` for a run on `mariadb-gate`. Its first comma separated +component is now compared for equality. + ## 2026-09-28: pipefail in the three older suites `tests/layer`, `tests/aplinfo` and `tests/signature` did not set `pipefail` diff --git a/bin/layer-compare-lib b/bin/layer-compare-lib index 052bb58..ca14fa7 100644 --- a/bin/layer-compare-lib +++ b/bin/layer-compare-lib @@ -368,16 +368,21 @@ measure_free_pinned() { MEASURE_WAIVER_FIELDS=5 # A rule is an exact path, or a named directory and its whole subtree -# written as DIR/**, and nothing else. Counting the literal characters of a -# glob was the previous test and it was the wrong measure: ./etc/ss* has -# eight of them and waives an SSH host key and a TLS private key together. -# A directory is a thing somebody chose; a character count is not. +# written as DIR/**, and nothing else. # -# The directory also has to be at least this many components below ./, so -# ./etc/** and ./** cannot be written, and the paths a rule actually covers -# have to sit under one immediate child of it, which is what "one rule per -# subtree" means and is checked against the run in bin/layer-report-lib. -MEASURE_WAIVER_MIN_DEPTH=2 +# Counting the literal characters of a glob was the first test and it was the +# wrong measure: ./etc/ss* has eight of them and waives an SSH host key and a +# TLS private key together. Counting components below ./ was the second and +# it was wrong the same way: ./var/lib has two, and ./var/lib is the common +# parent of six subtrees share/layer-state-paths declares separately, so one +# sentence about MariaDB accounts could clear PostgreSQL state. +# +# So the measure is the project's own statement of what state is. A glob +# rule's directory has to be a directory the capture's own state path list +# declares, or below one, which makes the waiver vocabulary exactly the +# vocabulary of the thing being waived. The paths a rule actually covers +# also have to sit under one immediate child of it, which is what "one rule +# per subtree" means and is checked against the run in bin/layer-report-lib. # measure_waiver_rules FILE -> the rules, one per line, comments gone measure_waiver_rules() { @@ -385,12 +390,35 @@ measure_waiver_rules() { sed -e '/^[[:space:]]*#/d' -e '/^[[:space:]]*$/d' "$1" } -# measure_waiver_check FILE [SAMPLE_CAP] +# measure_state_dirs GLOBS -> the directories a state path list declares +# +# A glob whose last component carries a wildcard declares the directory above +# it: ./var/lib/mysql/** declares ./var/lib/mysql, and ./etc/ssh/ssh_host_* +# declares ./etc/ssh. A glob with no wildcard at all declares a file and no +# directory, so ./etc/shadow does not license a waiver over ./etc. +measure_state_dirs() { + local glob last + while IFS= read -r glob; do + [[ -n "$glob" ]] || continue + last=${glob##*/} + [[ "$last" == *[*?]* ]] || continue + [[ "$glob" == */* ]] || continue + printf '%s\n' "${glob%/*}" + done <<<"$1" | LC_ALL=C sort -u +} + +# measure_dir_at_or_below DIR CANDIDATE +measure_dir_at_or_below() { + [[ "$1" == "$2" ]] || [[ "$1" == "$2"/* ]] +} + +# measure_waiver_check FILE [SAMPLE_CAP [STATE_GLOBS]] # SAMPLE_CAP is the capture's --state-max-bytes. A max-position at or past # it bounds nothing, because nothing larger was ever sampled, and it is the -# figure a template invites somebody to copy. +# figure a template invites somebody to copy. STATE_GLOBS is the capture's +# own state path list, which is what licenses a glob rule's directory. measure_waiver_check() { - local file=$1 cap=${2:-0} line rule count max rest rc=0 n=0 + local file=$1 cap=${2:-0} globs=${3:-} line rule count max rest rc=0 n=0 [[ -f "$file" ]] || return 0 local -a lines=() mapfile -t lines < <(measure_waiver_rules "$file") @@ -402,8 +430,8 @@ measure_waiver_check() { rc=1 continue fi - if ! measure_waiver_rule_shape "$rule"; then - measure_error "$file line $n: '$rule' is not an exact path nor a named directory's subtree; write either ./a/b/c or ./a/b/** with at least $MEASURE_WAIVER_MIN_DEPTH components below ./, so that what it covers is a directory somebody chose" + if ! measure_waiver_rule_shape "$rule" "$globs"; then + measure_error "$file line $n: '$rule' is not an exact path, nor the subtree of a directory the state path list declares; write ./a/b/c, or ./a/b/** where ./a/b is one of: $(measure_state_dirs "$globs" | paste -sd' ' -)" rc=1 fi if [[ ! "$count" =~ ^[0-9]+$ ]] || [[ ! "$max" =~ ^[0-9]+$ ]]; then @@ -419,10 +447,11 @@ measure_waiver_check() { return $rc } -# measure_waiver_rule_shape RULE -# An exact path with no wildcard at all, or DIR/** where DIR has none. +# measure_waiver_rule_shape RULE [STATE_GLOBS] +# An exact path with no wildcard at all, or DIR/** where DIR has none and is +# a directory the state path list declares, or below one. measure_waiver_rule_shape() { - local rule=$1 dir + local rule=$1 globs=${2:-} dir declared [[ "$rule" == ./* ]] || return 1 if [[ "$rule" != *[*?]* ]]; then return 0 @@ -430,19 +459,14 @@ measure_waiver_rule_shape() { [[ "$rule" == */'**' ]] || return 1 dir=${rule%/'**'} [[ "$dir" != *[*?]* ]] || return 1 - [[ "$(measure_waiver_depth "$dir")" -ge "$MEASURE_WAIVER_MIN_DEPTH" ]] || return 1 - return 0 -} - -# measure_waiver_depth DIR -> how many components it names below ./ -measure_waiver_depth() { - local d=${1#./} - d=${d%/} - if [[ -z "$d" ]]; then - echo 0 - return 0 - fi - awk -F/ '{ print NF }' <<<"$d" + [[ -n "$globs" ]] || return 1 + local -a dirs=() + mapfile -t dirs < <(measure_state_dirs "$globs") + for declared in "${dirs[@]}"; do + [[ -n "$declared" ]] || continue + measure_dir_at_or_below "$dir" "$declared" && return 0 + done + return 1 } # measure_waiver_prefix RULE -> the directory a subtree rule names, or nothing diff --git a/bin/layer-report-lib b/bin/layer-report-lib index 4e49101..b481ed3 100644 --- a/bin/layer-report-lib +++ b/bin/layer-report-lib @@ -447,7 +447,10 @@ measure_waiver_section() { "$(measure_cleared_reason "$rule")" # it cannot outlive its evidence: the captures it was read against # have to be the captures being measured - if [[ "$(cut -f4 <<<"$rule")" != *"$dir"* ]]; then + # the first comma separated component, compared for equality: a + # substring test accepted a waiver justified against mariadb-gate2 + # for a run on mariadb-gate + if [[ "$(cut -f4 <<<"$rule" | cut -d, -f1)" != "$dir" ]]; then measure_error "the waiver for $(cut -f1 <<<"$rule") was justified against '$(cut -f4 <<<"$rule")', which does not name $dir; a waiver is a reading of particular captures and does not carry to others" echo " THIS RULE NAMES OTHER CAPTURES: $dir is not in its justification" rc=1 diff --git a/bt-layer-measure b/bt-layer-measure index 762b219..5431c81 100755 --- a/bt-layer-measure +++ b/bt-layer-measure @@ -240,6 +240,7 @@ do_attribute() { || die "$EXIT_INPUT" "the captures cannot be compared" measure_waiver_check "${MEASURE_CLEARED:-/nonexistent}" \ "$(measure_meta_get "$MEASURE_DIR/$MEASURE_CONTROL" state_max_bytes)" \ + "$(measure_state_patterns "$MEASURE_DIR/$MEASURE_CONTROL/state-paths.txt")" \ || die "$EXIT_INPUT" "the waiver file is not usable" measure_attribute_report "$MEASURE_DIR" "$MEASURE_UNIT" \ "${MEASURE_CLEARED:-/nonexistent}" "$(suspect_paths_file)" \ diff --git a/docs/layer-measure.md b/docs/layer-measure.md index 6155464..08eaac9 100644 --- a/docs/layer-measure.md +++ b/docs/layer-measure.md @@ -232,10 +232,10 @@ template. | Field | What it is for | |---|---| -| `rule` | an exact path, or a named directory and its subtree as `DIR/**`, and nothing else. A glob that can reach a second subtree is refused, and so is one naming fewer than two components below `./`: counting literal characters said `./etc/ss*` was specific enough to waive an SSH host key and a TLS private key under one sentence | +| `rule` | an exact path, or a named directory and its subtree as `DIR/**`, where `DIR` is a directory `share/layer-state-paths` itself declares, or below one. So `./var/lib/mysql/**` and `./etc/ssh/**` are allowed and `./var/lib/**` is not, because nothing declares `./var/lib`: it is the common parent of six subtrees the list names separately. Two measures were tried and both were wrong the same way, counting the literal characters of the glob (`./etc/ss*` has enough of them to waive an SSH host key and a TLS private key together) and counting its components below `./` (`./var/lib` has enough). The waiver vocabulary is now the vocabulary of the thing being waived | | `count` | how many differing state paths it covered when written. A run where it covers a different number **fails**, so a new file appearing in a waived directory cannot be swept in silently | | `max-position` | the largest position actually reported: a line number for text, a byte offset for a binary. A covered path differing beyond it is **not** waived, and a value at or past the sample cap is refused, because it bounds nothing | -| `justified-against` | the capture directory and recipe commit it was read against. The directory is compared with `--dir` and the run fails if it does not match, so it cannot outlive its evidence | +| `justified-against` | the capture directory and recipe commit it was read against. The first comma separated component is compared with `--dir` for equality and the run fails if it differs, so it cannot outlive its evidence | | `reason` | what was read, and what it was found to be | A waiver clears `overlapping` and nothing else: it cannot clear a `real` @@ -319,6 +319,23 @@ clear. That the lever is luck about where the builds fall is a fair objection, and it is still the only one available, because teaching the tool to recognise a monotone run and downgrade it was declined on purpose. +## What this cannot see + +A differential measurement sees differences. State that is **identical in +every capture** is invisible to it by construction, and that includes the case +that matters: a secret copied out of the base control's own build tree and +baked into the component. Such a path never differs from the control, so it +never enters the differing set, never reaches a verdict and never appears in +the block. `share/layer-state-suspect` does not catch it either, for the same +reason: that net reads the paths the floor subtracted, and a path that does +not differ was never subtracted. + +So a PASS here means "nothing the change did shows up as a difference from the +control". It does not mean "this layer ships no secret it should not". The +instrument for that is a different one, reading a single image rather than +comparing four, and `docs/traps.md`, "Every appliance built from `core` has +the same machine-id", is what it would be looking for. + ## Layout Capture and formats in `bin/layer-measure-lib`, the verdicts in diff --git a/tests/compare b/tests/compare index 8132bcc..788e472 100755 --- a/tests/compare +++ b/tests/compare @@ -416,12 +416,16 @@ not_noise carry-payload "$tmp/carry-payload" "$tmp/carry-0001790574999999999" \ "$tmp/carry-0001790575000000001" "$tmp/carry-0001790575000000009" >/dev/null info "17: the waiver file" +# the list a capture carries, which is what licenses a glob rule's directory +statelist=$(printf '%s\n' './etc/shadow' './etc/ssh/ssh_host_*' \ + './etc/ssl/private/**' './var/lib/mysql/**' './var/lib/postgresql/**' \ + './home/**/.ssh/**') cat > "$tmp/cleared" <<'EOF' # rule, count, max-position, justified-against, reason ./var/lib/mysql/ibdata1 1 600 captures at /m/gate, recipe 659eeb4 InnoDB creation stamps, checked by hand on 2026-09-28 ./var/lib/postgresql/** 6 4096 captures at /m/gate, recipe 659eeb4 initdb output, read on 2026-09-28 EOF -assert_ok waiver-ok measure_waiver_check "$tmp/cleared" +assert_ok waiver-ok measure_waiver_check "$tmp/cleared" 0 "$statelist" assert_ok waiver-nofile measure_waiver_check "$tmp/absent" assert_eq waiver-rules "$(measure_waiver_rules "$tmp/cleared" | wc -l)" 2 rule=$(measure_cleared_rule "$tmp/cleared" ./var/lib/mysql/ibdata1 600) @@ -438,20 +442,34 @@ assert_grep cleared-glob "initdb output" \ # a glob is an exact path or a named directory and its subtree, and nothing # else: counting literal characters said ./etc/ss* was specific enough to # waive an SSH host key and a TLS private key under one sentence -for bad in '*' './etc/*' './etc/ss*' './var/lib/*' './var/lib/my*/**' './etc/**' './**' './var/lib/mysql/*.MAD'; do - printf '%s\t1\t1\tnowhere\ttoo broad or not a subtree\n' "$bad" > "$tmp/broad" - assert_fail "waiver-broad-$bad" measure_waiver_check "$tmp/broad" +# a glob rule's directory has to be one the state path list itself declares, +# or below one. Counting components said ./var/lib was specific enough, and +# ./var/lib is the common parent of six subtrees the list declares separately, +# so one sentence about MariaDB accounts could clear PostgreSQL state. +assert_eq state-dirs "$(measure_state_dirs "$statelist" | paste -sd, -)" \ + "./etc/ssh,./etc/ssl/private,./home/**/.ssh,./var/lib/mysql,./var/lib/postgresql" +assert_ok below-same measure_dir_at_or_below ./var/lib/mysql ./var/lib/mysql +assert_ok below-deeper measure_dir_at_or_below ./var/lib/mysql/mysql ./var/lib/mysql +assert_fail below-above measure_dir_at_or_below ./var/lib ./var/lib/mysql +assert_fail below-sibling measure_dir_at_or_below ./var/lib/mysqlx ./var/lib/mysql +for bad in '*' './etc/*' './etc/ss*' './var/lib/*' './var/lib/my*/**' './etc/**' './**' \ + './var/lib/mysql/*.MAD' './var/lib/**' './var/**' './etc/ssl/**' \ + './var/cache/**' './usr/share/**' './etc/webmin/**'; do + printf '%s\t1\t1\tnowhere\ttoo broad or not declared state\n' "$bad" > "$tmp/broad" + assert_fail "waiver-broad-$bad" measure_waiver_check "$tmp/broad" 0 "$statelist" done -for good in './var/lib/mysql/**' './var/lib/mysql/mysql/global_priv.MAD' './etc/ssh/**'; do - printf '%s\t1\t1\tnowhere\ta named subtree or an exact path\n' "$good" > "$tmp/ok" - assert_ok "waiver-ok-$good" measure_waiver_check "$tmp/ok" +for good in './var/lib/mysql/**' './var/lib/mysql/mysql/**' './etc/ssh/**' \ + './etc/ssl/private/**' './var/lib/mysql/mysql/global_priv.MAD' \ + './some/path/nobody/declared'; do + printf '%s\t1\t1\tnowhere\ta declared subtree or an exact path\n' "$good" > "$tmp/ok" + assert_ok "waiver-ok-$good" measure_waiver_check "$tmp/ok" 0 "$statelist" done printf './var/lib/mysql/**\ttwo fields\n' > "$tmp/short" -assert_fail waiver-fields measure_waiver_check "$tmp/short" +assert_fail waiver-fields measure_waiver_check "$tmp/short" 0 "$statelist" printf './var/lib/mysql/**\tmany\t1\tnowhere\tcount is not a number\n' > "$tmp/nan" -assert_fail waiver-nan measure_waiver_check "$tmp/nan" +assert_fail waiver-nan measure_waiver_check "$tmp/nan" 0 "$statelist" printf './var/lib/mysql/**\t1\t1\t\t\n' > "$tmp/nojust" -assert_fail waiver-nojust measure_waiver_check "$tmp/nojust" +assert_fail waiver-nojust measure_waiver_check "$tmp/nojust" 0 "$statelist" assert_eq free-summary-none "$(measure_free_summary "abc" "abc" "")" "nothing" assert_grep free-summary-num "a number or a timestamp" \ @@ -460,14 +478,12 @@ assert_grep free-summary-not "not a number" \ "$(measure_free_summary "t=abcd" "t=" "")" # the shape helpers, and the cap check the executable passes a cap to -assert_eq depth-none "$(measure_waiver_depth ./)" 0 -assert_eq depth-two "$(measure_waiver_depth ./var/lib)" 2 assert_eq prefix-subtree "$(measure_waiver_prefix './var/lib/mysql/**')" "./var/lib/mysql" assert_eq prefix-exact "$(measure_waiver_prefix ./var/lib/mysql/ibdata1)" "" printf './var/lib/mysql/**\t1\t16777216\there\tthe sample cap bounds nothing\n' > "$tmp/capmax" -assert_ok capmax-nocap measure_waiver_check "$tmp/capmax" -assert_fail capmax-withcap measure_waiver_check "$tmp/capmax" 16777216 -assert_ok capmax-under measure_waiver_check "$tmp/capmax" 16777217 +assert_ok capmax-nocap measure_waiver_check "$tmp/capmax" 0 "$statelist" +assert_fail capmax-withcap measure_waiver_check "$tmp/capmax" 16777216 "$statelist" +assert_ok capmax-under measure_waiver_check "$tmp/capmax" 16777217 "$statelist" info "17b: measure_max_position" assert_eq maxpos-text "$(measure_max_position "$tmp/c2" "$tmp/c1")" 1 diff --git a/tests/measure b/tests/measure index 76fa521..a101fb4 100755 --- a/tests/measure +++ b/tests/measure @@ -492,6 +492,12 @@ printf './var/lib/mysql/**\t2\t600\t/somewhere/else, recipe deadbeef\tread elsew r=$(attr u1 --cleared "$tmp/elsewhere" || true) assert_grep waiver-elsewhere "THIS RULE NAMES OTHER CAPTURES" "$r" assert_status waiver-elsewhere-status 1 attr u1 --cleared "$tmp/elsewhere" +# and a directory that merely has the run's as a prefix is not the run's: +# a substring test accepted mariadb-gate2 for a run on mariadb-gate +printf './var/lib/mysql/**\t2\t600\t%s2, recipe deadbeef\tread next door\n' "$caps" > "$tmp/nextdoor" +r=$(attr u1 --cleared "$tmp/nextdoor" || true) +assert_grep waiver-nextdoor "THIS RULE NAMES OTHER CAPTURES" "$r" +assert_status waiver-nextdoor-status 1 attr u1 --cleared "$tmp/nextdoor" # a glob written against one path that now covers two fails rather than # quietly taking the new one in printf './var/lib/mysql/**\t1\t99999\tfixtures\twritten when this matched one file\n' > "$tmp/stretch" From ef655958985ad70d93182471010b7594e8553cb5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Mon, 28 Sep 2026 09:54:24 +0000 Subject: [PATCH 7/9] fix: anchor the state path list to the committed one The licensing rule of the previous commit made state-paths.txt load bearing twice: it decides which bytes a capture keeps, and, since a waiver rule's directory has to be one the list declares, it now also decides which waivers are legal. measure_preconditions checked that every capture carried a byte identical list whose recorded digest matched, which says the operator was consistent with themselves and nothing more. So four captures taken against a list with ./var/lib/** added agree with each other, the block reports that they agree, a ./var/lib/** waiver is licensed, and one sentence about MariaDB accounts clears PostgreSQL state on the next run. share/layer-state-suspect cannot catch it and reports 0 subtracted but state shaped throughout, because widening the list makes more paths state paths, so nothing new is subtracted. What changed is that widening the list used to be harmless: it only ever increased what was compared by content. The same knob now points both ways. attribute compares the capture's copy with the committed share/layer-state-paths and refuses on mismatch. A run against another list declares a reason with --state-paths-override, which the block prints as OVERRIDE, and the block prints both digests in any case, because one digest with nothing to check it against is the shape of the finding two rounds ago where state_paths_sha256 was recorded and never read. Refusal rather than a printed warning, for that same reason: every other precondition here refuses, and a warning relies on somebody noticing. The anchor has no environment override. MEASURE_STATE_PATHS is a capture time knob and attribute ignores it entirely, including as a way of moving the anchor, so the property that pointing it at a narrower file changes nothing after the fact is unchanged and is still tested. tests/measure now takes its captures against the committed list rather than a fixture list, which is the real configuration and removes a difference between the suite and a real run. Two smaller things from the same review. A declaration with a wildcard in a non-final component, ./home/**/.ssh/**, yields the declared directory ./home/**/.ssh, which is compared literally and which no wildcard-free rule can sit below, so it licenses nothing. That is intended and now says so in the doc: it fails closed, the route for those paths is an exact path waiver, and matching a glob against a glob here would put back the breadth ambiguity the rule exists to remove. docs/layer-measure.md repeats, in the section about what a pull request quotes, that a PASS does not mean the layer ships no secret it should not, because the person over-reading a PASS is reading the block and not the section that explains it. And the slash guard in measure_state_dirs, which skips a malformed list entry with no / in it, was executing without a test behind it. It has one now, state-dirs-noslash. COVERAGE.md records which of the six branches added last commit a mutation test kills and which survive because they are redundant, since 100 percent line coverage with an untested line in it claims more than it has. --- COVERAGE.md | 56 +++++++++++++++++++++++++++++++++++++--- bin/layer-compare-lib | 25 ++++++++++++++++++ bin/layer-measure-lib | 4 ++- bin/layer-report-lib | 15 +++++++++++ bt-layer-measure | 26 ++++++++++++++++++- docs/layer-measure.md | 25 ++++++++++++++---- tests/compare | 20 +++++++++++++++ tests/measure | 59 ++++++++++++++++++++++++++++--------------- 8 files changed, 199 insertions(+), 31 deletions(-) diff --git a/COVERAGE.md b/COVERAGE.md index 4d46259..48363f5 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -131,10 +131,10 @@ measures ten files: | bt-aplinfo | 47 of 47 | 100 percent | | bin/signature-lib | 76 of 76 | 100 percent | | bin/generate-signature | 87 of 87 | 100 percent | -| bin/layer-measure-lib | 204 of 204 | 100 percent | -| bin/layer-compare-lib | 322 of 322 | 100 percent | -| bin/layer-report-lib | 264 of 264 | 100 percent | -| bt-layer-measure | 71 of 71 | 100 percent | +| bin/layer-measure-lib | 206 of 206 | 100 percent | +| bin/layer-compare-lib | 330 of 330 | 100 percent | +| bin/layer-report-lib | 270 of 270 | 100 percent | +| bt-layer-measure | 77 of 77 | 100 percent | The gate stays at 99, the lowest file rounded down. All four new files are at 100 percent line and branch: every subcommand, every exit code (0, 1, 2 and @@ -266,6 +266,54 @@ thing to trust. against `mariadb-gate2` for a run on `mariadb-gate`. Its first comma separated component is now compared for equality. +## 2026-09-28: anchoring the state path list to the repository + +The licensing rule of the previous section made `state-paths.txt` load bearing +twice: it decides which bytes a capture keeps, and it decides which waiver +rules are legal. `measure_preconditions` checked that every capture carried a +byte identical list whose recorded digest matched, which says the operator was +consistent with themselves and nothing more. Four captures taken against a +list with `./var/lib/**` added agree with each other, license a `./var/lib/**` +waiver, and let one sentence about MariaDB accounts clear PostgreSQL state. +`share/layer-state-suspect` cannot catch it, and reports `0 subtracted but +state shaped` throughout, because widening the list makes more paths state +paths so nothing new is subtracted. + +`attribute` now also compares the capture's copy with the committed +`share/layer-state-paths` and refuses on mismatch. A run against another list +has to declare a reason with `--state-paths-override`, which the block prints +as `OVERRIDE` beside both digests, and the block prints both digests in any +case, because one digest with nothing to check it against is what the round +about `state_paths_sha256` was about. + +Refusal rather than a printed warning, for the same reason: every other +precondition here refuses, and the defect being closed is a field that was +recorded and never read. A warning relies on somebody noticing. + +The anchor has no environment override. `MEASURE_STATE_PATHS` is a capture +time knob and `attribute` ignores it entirely, including as a way of moving +the anchor, so the property that pointing it at a narrower file changes +nothing after the fact is unchanged. `tests/measure` now takes its captures +against the committed list rather than a fixture list, which is the real +configuration. + +### What the number does and does not say + +A mutation test of the six branches added with the licensing rule: three fail +the suite when removed, which is coverage doing work (the +wildcard-in-last-component guard in `measure_state_dirs`, the equality clause +in `measure_dir_at_or_below`, and loosening that function's path boundary to a +prefix match). Two survive because they are redundant, since removing them the +licensing loop refuses the same inputs anyway: the no-wildcard-in-directory +and empty-list guards in `measure_waiver_rule_shape`. They stay as defence in +depth and a surviving mutant is the correct result for them. + +The sixth, the `*/*` guard in `measure_state_dirs` that skips a malformed +entry with no slash in it, executed without being tested. It is tested now, +`state-dirs-noslash` in `tests/compare`, because 100 percent line coverage +with one line in it that no assertion would miss is a number claiming more +than it has. + ## 2026-09-28: pipefail in the three older suites `tests/layer`, `tests/aplinfo` and `tests/signature` did not set `pipefail` diff --git a/bin/layer-compare-lib b/bin/layer-compare-lib index ca14fa7..990c523 100644 --- a/bin/layer-compare-lib +++ b/bin/layer-compare-lib @@ -390,6 +390,31 @@ measure_waiver_rules() { sed -e '/^[[:space:]]*#/d' -e '/^[[:space:]]*$/d' "$1" } +# measure_list_anchored CAPTURE COMMITTED +# The state path list decides two things: which bytes a capture kept, and +# which waiver rules are legal. Checking it is identical across the captures +# only says the operator was consistent with themselves. This says it is the +# list the repository declares. +# +# Widening the list used to be harmless, because it only ever increased what +# was compared by content. Once a waiver rule's directory has to be one the +# list declares, widening it also widens the waiver vocabulary, so the same +# knob acquired a second effect pointing the other way: adding ./var/lib/** +# to the list makes a ./var/lib/** waiver legal, and one sentence about +# MariaDB accounts then clears PostgreSQL state. +measure_list_anchored() { + local capture=$1 committed=$2 + if [[ ! -f "$committed" ]]; then + measure_error "no committed state path list at $committed" + return 1 + fi + if [[ ! -f "$capture/state-paths.txt" ]]; then + measure_error "$capture carries no state path list" + return 1 + fi + cmp -s -- "$capture/state-paths.txt" "$committed" +} + # measure_state_dirs GLOBS -> the directories a state path list declares # # A glob whose last component carries a wildcard declares the directory above diff --git a/bin/layer-measure-lib b/bin/layer-measure-lib index f694f2e..4ed665d 100644 --- a/bin/layer-measure-lib +++ b/bin/layer-measure-lib @@ -336,6 +336,7 @@ measure_parse_args() { MEASURE_UNIT= MEASURE_CONTROL2=() MEASURE_CLEARED= + MEASURE_LIST_OVERRIDE= MEASURE_HELP= MEASURE_CAP=$MEASURE_STATE_MAX_BYTES_DEFAULT local opt val @@ -347,7 +348,7 @@ measure_parse_args() { shift continue ;; - --dir|--layer|--parent|--tag|--a|--b|--control|--unit|--control-again|--state-max-bytes|--cleared) + --dir|--layer|--parent|--tag|--a|--b|--control|--unit|--control-again|--state-max-bytes|--cleared|--state-paths-override) if [[ $# -lt 2 ]]; then measure_error "$opt needs a value" return 1 @@ -380,6 +381,7 @@ measure_parse_args() { --unit) MEASURE_UNIT=$val ;; --control-again) MEASURE_CONTROL2+=("$val") ;; --cleared) MEASURE_CLEARED=$val ;; + --state-paths-override) MEASURE_LIST_OVERRIDE=$val ;; --state-max-bytes) MEASURE_CAP=$val ;; esac done diff --git a/bin/layer-report-lib b/bin/layer-report-lib index b481ed3..c7ab520 100644 --- a/bin/layer-report-lib +++ b/bin/layer-report-lib @@ -34,6 +34,13 @@ MEASURE_PRINT_MAX=40 MEASURE_ATTRIBUTABLE=0 MEASURE_SUSPECT_COUNT=0 +# The committed state path list the captures are anchored to, and the reason +# given if they deliberately are not. Both are printed in the block, because +# a digest with nothing to check it against is what the round about +# state_paths_sha256 was about. +MEASURE_COMMITTED_LIST= +MEASURE_LIST_OVERRIDE= + # --------------------------------------------------------------------------- # printing helpers # --------------------------------------------------------------------------- @@ -251,6 +258,14 @@ measure_attribute_report() { printf '\n' printf 'state paths %s captures agree on the list (sha256 %s)\n' \ "$(( ${#c[@]} + 1 ))" "$(measure_meta_get "$dir/$base" state_paths_sha256 | cut -c1-16)" + if [[ -f "$MEASURE_COMMITTED_LIST" ]]; then + printf 'committed %s (sha256 %s)\n' "$MEASURE_COMMITTED_LIST" \ + "$(sha256sum < "$MEASURE_COMMITTED_LIST" | cut -c1-16)" + fi + if [[ -n "$MEASURE_LIST_OVERRIDE" ]]; then + printf 'OVERRIDE the captures were taken against a list that is not the committed one, declared: %s\n' \ + "$MEASURE_LIST_OVERRIDE" + fi printf 'unit %s\n' "$(measure_describe "$dir" "$unit")" for tag in "${c[@]}"; do printf 'control %s\n' "$(measure_describe "$dir" "$tag")" diff --git a/bt-layer-measure b/bt-layer-measure index 5431c81..5c607e5 100755 --- a/bt-layer-measure +++ b/bt-layer-measure @@ -86,6 +86,13 @@ Options:: --unit TAG the build with the component supplied as a unit --control-again TAG another control build. Repeatable, and the floor is the union over every pair of controls + --state-paths-override REASON + run against captures whose state path list is not the + committed \$BT/share/layer-state-paths. That list + decides both which bytes were kept and which waiver + rules are legal, so a mismatch is refused unless the + run declares a reason, which the block prints as + OVERRIDE beside both digests --cleared FILE waivers, five tab separated fields per rule: rule, count, max-position, justified-against, reason. A state path whose difference coincides with the @@ -106,7 +113,9 @@ Environment:: list is copied into each capture and 'attribute' reads the copy, never this variable, so what a measurement was blind to cannot be changed after the - fact + fact. 'attribute' also anchors that copy to the + committed \$BT/share/layer-state-paths, which has no + environment override at all MEASURE_SUSPECT_PATHS names that look like state (default \$BT/share/layer-state-suspect). A path the floor @@ -152,6 +161,14 @@ state_paths_file() { printf '%s\n' "${MEASURE_STATE_PATHS:-$BT/share/layer-state-paths}" } +# The committed list, with no environment override on purpose. `attribute` +# ignores MEASURE_STATE_PATHS entirely: that variable is a capture time knob, +# and reading it here would let the anchor be moved to whatever the captures +# happen to carry, which is the check disabling itself. +committed_paths_file() { + printf '%s\n' "$BT/share/layer-state-paths" +} + suspect_paths_file() { printf '%s\n' "${MEASURE_SUSPECT_PATHS:-$BT/share/layer-state-suspect}" } @@ -238,6 +255,13 @@ do_attribute() { measure_preconditions "$MEASURE_DIR" "$MEASURE_UNIT" \ "$MEASURE_CONTROL" "${MEASURE_CONTROL2[@]}" \ || die "$EXIT_INPUT" "the captures cannot be compared" + MEASURE_COMMITTED_LIST=$(committed_paths_file) + if ! measure_list_anchored "$MEASURE_DIR/$MEASURE_CONTROL" "$MEASURE_COMMITTED_LIST"; then + if [[ -z "$MEASURE_LIST_OVERRIDE" ]]; then + die "$EXIT_INPUT" "the captures were taken against a state path list that is not $MEASURE_COMMITTED_LIST; that list decides both which bytes were kept and which waiver rules are legal, so a run against another one has to say so with --state-paths-override REASON, which the block then prints" + fi + warning "the captures were taken against a list that is not the committed one; declared: $MEASURE_LIST_OVERRIDE" + fi measure_waiver_check "${MEASURE_CLEARED:-/nonexistent}" \ "$(measure_meta_get "$MEASURE_DIR/$MEASURE_CONTROL" state_max_bytes)" \ "$(measure_state_patterns "$MEASURE_DIR/$MEASURE_CONTROL/state-paths.txt")" \ diff --git a/docs/layer-measure.md b/docs/layer-measure.md index 08eaac9..c7fb1ca 100644 --- a/docs/layer-measure.md +++ b/docs/layer-measure.md @@ -84,6 +84,11 @@ floor, the attributable count and the verdict, so a claim like "0 attributable differences" says which command produced it and a reviewer can run that command again. +A PASS in that block means nothing the change did shows up as a difference +from the control. **It does not mean the layer ships no secret it should +not**: see "What this cannot see" below, because the person over-reading a +PASS is reading the block rather than that section. + Exit status: 0 when the verdict is PASS, 1 when it is FAIL, 2 on a usage error, 3 when a capture, a build or a tool is missing. @@ -116,10 +121,20 @@ A `mariadb` capture with the cap raised past `ib_logfile0` is about 150 MiB, so four of them are roughly 600 MiB. `attribute` reads the state path list from the captures, never from the -environment. The list is what decides which bytes were kept, so a run cannot -be made to look clean afterwards by pointing `MEASURE_STATE_PATHS` at a -narrower file: all the captures must carry the same list, and their recorded -digests must match it. +environment, and anchors it to the committed `share/layer-state-paths`. Three +things have to agree: every capture carries the same list, each capture's +recorded digest matches the copy it carries, and that copy is the committed +file. `MEASURE_STATE_PATHS` is a capture time knob and `attribute` ignores it +entirely, including as a way of moving the anchor. + +The third of those exists because the list decides two things now. It decides +which bytes a capture keeps, and, since a waiver rule's directory has to be one +the list declares, it also decides which waivers are legal. Widening it used to +be harmless in the only direction it acted; now the same knob points both ways, +and adding `./var/lib/**` to it would make a `./var/lib/**` waiver legal. +Captures taken against another list are refused unless the run declares a +reason with `--state-paths-override`, which the block prints as `OVERRIDE` +beside both digests. ### Why a tab, and why the path comes first @@ -232,7 +247,7 @@ template. | Field | What it is for | |---|---| -| `rule` | an exact path, or a named directory and its subtree as `DIR/**`, where `DIR` is a directory `share/layer-state-paths` itself declares, or below one. So `./var/lib/mysql/**` and `./etc/ssh/**` are allowed and `./var/lib/**` is not, because nothing declares `./var/lib`: it is the common parent of six subtrees the list names separately. Two measures were tried and both were wrong the same way, counting the literal characters of the glob (`./etc/ss*` has enough of them to waive an SSH host key and a TLS private key together) and counting its components below `./` (`./var/lib` has enough). The waiver vocabulary is now the vocabulary of the thing being waived | +| `rule` | an exact path, or a named directory and its subtree as `DIR/**`, where `DIR` is a directory `share/layer-state-paths` itself declares, or below one. A declaration whose last component has no wildcard names a file and declares no directory, so `./etc/shadow` licenses nothing; and one with a wildcard in a *non-final* component, such as `./home/**/.ssh/**`, declares `./home/**/.ssh`, which is compared literally and which no wildcard-free rule can sit below, so it licenses nothing either. Both fail closed on purpose: the route for those paths is an exact-path waiver, and matching a glob against a glob here would put back the breadth ambiguity this rule exists to remove. So `./var/lib/mysql/**` and `./etc/ssh/**` are allowed and `./var/lib/**` is not, because nothing declares `./var/lib`: it is the common parent of six subtrees the list names separately. Two measures were tried and both were wrong the same way, counting the literal characters of the glob (`./etc/ss*` has enough of them to waive an SSH host key and a TLS private key together) and counting its components below `./` (`./var/lib` has enough). The waiver vocabulary is now the vocabulary of the thing being waived | | `count` | how many differing state paths it covered when written. A run where it covers a different number **fails**, so a new file appearing in a waived directory cannot be swept in silently | | `max-position` | the largest position actually reported: a line number for text, a byte offset for a binary. A covered path differing beyond it is **not** waived, and a value at or past the sample cap is refused, because it bounds nothing | | `justified-against` | the capture directory and recipe commit it was read against. The first comma separated component is compared with `--dir` for equality and the run fails if it differs, so it cannot outlive its evidence | diff --git a/tests/compare b/tests/compare index 788e472..c6f0298 100755 --- a/tests/compare +++ b/tests/compare @@ -448,6 +448,16 @@ assert_grep cleared-glob "initdb output" \ # so one sentence about MariaDB accounts could clear PostgreSQL state. assert_eq state-dirs "$(measure_state_dirs "$statelist" | paste -sd, -)" \ "./etc/ssh,./etc/ssl/private,./home/**/.ssh,./var/lib/mysql,./var/lib/postgresql" +# a malformed entry with no slash in it declares nothing: without this guard +# ${glob%/*} would hand back the entry itself as a directory +assert_eq state-dirs-noslash "$(measure_state_dirs "$(printf 'shadow*\n./var/lib/mysql/**')")" \ + "./var/lib/mysql" +assert_eq state-dirs-none "$(measure_state_dirs "$(printf './etc/shadow\n./etc/group')")" "" +# a declaration with a wildcard in a non-final component licenses nothing, +# because the comparison is literal. It fails closed on purpose: an exact +# path waiver is the route for those, and a glob comparison here would put +# the breadth ambiguity back +assert_fail below-wildcard-decl measure_dir_at_or_below ./home/alice/.ssh './home/**/.ssh' assert_ok below-same measure_dir_at_or_below ./var/lib/mysql ./var/lib/mysql assert_ok below-deeper measure_dir_at_or_below ./var/lib/mysql/mysql ./var/lib/mysql assert_fail below-above measure_dir_at_or_below ./var/lib ./var/lib/mysql @@ -485,6 +495,16 @@ assert_ok capmax-nocap measure_waiver_check "$tmp/capmax" 0 "$statelist" assert_fail capmax-withcap measure_waiver_check "$tmp/capmax" 16777216 "$statelist" assert_ok capmax-under measure_waiver_check "$tmp/capmax" 16777217 "$statelist" +info "17a: the list a capture carries is anchored to the committed one" +printf './etc/shadow\n./var/lib/mysql/**\n' > "$tmp/committed" +mkdir -p "$tmp/cap-ok" "$tmp/cap-wide" +cp "$tmp/committed" "$tmp/cap-ok/state-paths.txt" +printf './etc/shadow\n./var/lib/mysql/**\n./var/lib/**\n' > "$tmp/cap-wide/state-paths.txt" +assert_ok anchored-same measure_list_anchored "$tmp/cap-ok" "$tmp/committed" +assert_fail anchored-wide measure_list_anchored "$tmp/cap-wide" "$tmp/committed" +assert_fail anchored-nocommitted measure_list_anchored "$tmp/cap-ok" "$tmp/absent" +assert_fail anchored-nocapture measure_list_anchored "$tmp/absent" "$tmp/committed" + info "17b: measure_max_position" assert_eq maxpos-text "$(measure_max_position "$tmp/c2" "$tmp/c1")" 1 assert_eq maxpos-binary "$(measure_max_position "$tmp/bb" "$tmp/ba")" 3 diff --git a/tests/measure b/tests/measure index a101fb4..8cc4ed1 100755 --- a/tests/measure +++ b/tests/measure @@ -207,19 +207,15 @@ echo "stub built \$layer from \$next" EOF chmod 755 "$builder" -patterns=$tmp/state-paths -cat > "$patterns" <<'EOF' -# accounts, credentials and database state -./etc/shadow -./var/lib/mysql/** -EOF +# the committed list, which is what a capture is taken against in earnest +# and what attribute anchors the capture's copy to +patterns=$BT/share/layer-state-paths M=$BT/bt-layer-measure caps=$tmp/captures export MEASURE_BT_LAYER=$builder export MEASURE_LAYERS_DIR=$layers export MEASURE_LOCK=$tmp/build.lock -export MEASURE_STATE_PATHS=$patterns capture() { # capture FIXTURE TAG [extra args...] local fixture=$1 tag=$2 @@ -283,8 +279,8 @@ assert_fail pkgs-notool env MEASURE_DPKG_QUERY=no-such-dpkg-query bash -c \ "source '$BT/bin/layer-measure-lib'; measure_pkgs_record '$fix/c1'" info "5: measure_state_patterns" -assert_eq patterns-two "$(measure_state_patterns "$patterns" | wc -l)" 2 -assert_eq patterns-first "$(measure_state_patterns "$patterns" | head -1)" "./etc/shadow" +assert_grep patterns-shadow "^\./etc/shadow$" "$(measure_state_patterns "$patterns")" +assert_grep patterns-mysql "^\./var/lib/mysql/\*\*$" "$(measure_state_patterns "$patterns")" assert_fail patterns-missing measure_state_patterns "$tmp/absent" : > "$tmp/empty-patterns" assert_fail patterns-empty measure_state_patterns "$tmp/empty-patterns" @@ -559,26 +555,49 @@ for pair in "c1 kc1" "c2 kc2" "c3 kc3" "u1 ku1"; do done printf 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHARDCODEDKEYINTHEUNIT root@keel\n' \ > "$fix/ku1/etc/ssh/ssh_host_ed25519_key.pub" -cat > "$tmp/state-plus" <<'EOF' -./etc/shadow -./var/lib/mysql/** -./etc/ssh/ssh_host_* -EOF for t in kc1 kc2 kc3 ku1; do echo "$t" > "$layers/NEXT" - MEASURE_STATE_PATHS=$tmp/state-plus "$M" capture --dir "$caps" --layer mariadb \ - --parent core --tag "$t" > /dev/null + "$M" capture --dir "$caps" --layer mariadb --parent core --tag "$t" > /dev/null done cat > "$tmp/db-waiver" < "$tmp/state-wide" +for t in wc1 wc2 wc3 wu1; do + echo "${t/w/k}" > "$layers/NEXT" + MEASURE_STATE_PATHS=$tmp/state-wide "$M" capture --dir "$caps" --layer mariadb \ + --parent core --tag "$t" > /dev/null +done +wattr() { "$M" attribute --dir "$caps" --unit wu1 --control wc1 \ + --control-again wc2 --control-again wc3 "$@"; } +assert_status anchor-refused 3 wattr +r=$(wattr 2>&1 || true) +assert_grep anchor-why "state path list that is not" "$r" +# and pointing the variable at the widened list does not move the anchor +assert_status anchor-nomove 3 env "MEASURE_STATE_PATHS=$tmp/state-wide" \ + "$M" attribute --dir "$caps" --unit wu1 --control wc1 --control-again wc2 \ + --control-again wc3 +# declared, it runs, and the block says so beside both digests +r=$(wattr --state-paths-override "captures predate the widened list, read by hand" || true) +assert_grep anchor-override "OVERRIDE" "$r" +assert_grep anchor-override-why "read by hand" "$r" +assert_grep anchor-committed "^committed " "$r" +assert_grep anchor-carried "captures agree on the list" "$r" info "17: blind because nothing was sampled" capture c1 s1 --state-max-bytes 8 > /dev/null From 0a0cea1d323e29ffde6229370fb10cb4aa664d8f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Tue, 29 Sep 2026 03:19:27 +0000 Subject: [PATCH 8/9] fix: print OVERRIDE only when the list actually diverges The OVERRIDE line asserted "the captures were taken against a list that is not the committed one" whenever --state-paths-override was given, whether or not the anchor check had failed. On captures that carry the committed list the block then printed two identical digests and, directly beneath them, a line saying they differ. It erred in the safe direction, but the block exists so a reader can check each claim against the evidence printed next to it. attribute now records that the anchor check failed, and the report prints OVERRIDE only then. A reason given on captures that do carry the committed list is still printed, on an "override" line saying it was declared and not needed, so nothing the operator passed is dropped from the block. Coverage: layer-report-lib 273 of 273, bt-layer-measure 78 of 78. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01SfWQScmDZ94KCS5DMYrfe6 --- COVERAGE.md | 4 ++-- bin/layer-report-lib | 11 ++++++++++- bt-layer-measure | 1 + docs/layer-measure.md | 4 +++- tests/measure | 8 ++++++++ 5 files changed, 24 insertions(+), 4 deletions(-) diff --git a/COVERAGE.md b/COVERAGE.md index 48363f5..8d67ab7 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -133,8 +133,8 @@ measures ten files: | bin/generate-signature | 87 of 87 | 100 percent | | bin/layer-measure-lib | 206 of 206 | 100 percent | | bin/layer-compare-lib | 330 of 330 | 100 percent | -| bin/layer-report-lib | 270 of 270 | 100 percent | -| bt-layer-measure | 77 of 77 | 100 percent | +| bin/layer-report-lib | 273 of 273 | 100 percent | +| bt-layer-measure | 78 of 78 | 100 percent | The gate stays at 99, the lowest file rounded down. All four new files are at 100 percent line and branch: every subcommand, every exit code (0, 1, 2 and diff --git a/bin/layer-report-lib b/bin/layer-report-lib index c7ab520..3b84be4 100644 --- a/bin/layer-report-lib +++ b/bin/layer-report-lib @@ -40,6 +40,9 @@ MEASURE_SUSPECT_COUNT=0 # state_paths_sha256 was about. MEASURE_COMMITTED_LIST= MEASURE_LIST_OVERRIDE= +# Non-empty when the anchor check found the captures' list is not the +# committed one, which is the only case the OVERRIDE line may assert. +MEASURE_LIST_DIVERGES= # --------------------------------------------------------------------------- # printing helpers @@ -262,9 +265,15 @@ measure_attribute_report() { printf 'committed %s (sha256 %s)\n' "$MEASURE_COMMITTED_LIST" \ "$(sha256sum < "$MEASURE_COMMITTED_LIST" | cut -c1-16)" fi - if [[ -n "$MEASURE_LIST_OVERRIDE" ]]; then + # OVERRIDE asserts a mismatch, so it is printed only when the anchor + # check found one; a reason given on captures that do carry the + # committed list is still printed, as what it is + if [[ -n "$MEASURE_LIST_DIVERGES" ]]; then printf 'OVERRIDE the captures were taken against a list that is not the committed one, declared: %s\n' \ "$MEASURE_LIST_OVERRIDE" + elif [[ -n "$MEASURE_LIST_OVERRIDE" ]]; then + printf 'override declared and not needed, the captures carry the committed list: %s\n' \ + "$MEASURE_LIST_OVERRIDE" fi printf 'unit %s\n' "$(measure_describe "$dir" "$unit")" for tag in "${c[@]}"; do diff --git a/bt-layer-measure b/bt-layer-measure index 5c607e5..718b130 100755 --- a/bt-layer-measure +++ b/bt-layer-measure @@ -257,6 +257,7 @@ do_attribute() { || die "$EXIT_INPUT" "the captures cannot be compared" MEASURE_COMMITTED_LIST=$(committed_paths_file) if ! measure_list_anchored "$MEASURE_DIR/$MEASURE_CONTROL" "$MEASURE_COMMITTED_LIST"; then + MEASURE_LIST_DIVERGES=1 if [[ -z "$MEASURE_LIST_OVERRIDE" ]]; then die "$EXIT_INPUT" "the captures were taken against a state path list that is not $MEASURE_COMMITTED_LIST; that list decides both which bytes were kept and which waiver rules are legal, so a run against another one has to say so with --state-paths-override REASON, which the block then prints" fi diff --git a/docs/layer-measure.md b/docs/layer-measure.md index c7fb1ca..1b70490 100644 --- a/docs/layer-measure.md +++ b/docs/layer-measure.md @@ -134,7 +134,9 @@ be harmless in the only direction it acted; now the same knob points both ways, and adding `./var/lib/**` to it would make a `./var/lib/**` waiver legal. Captures taken against another list are refused unless the run declares a reason with `--state-paths-override`, which the block prints as `OVERRIDE` -beside both digests. +beside both digests. A reason given on captures that do carry the committed +list is printed on an `override` line as declared and not needed, so the block never +asserts a mismatch its own two digests contradict. ### Why a tab, and why the path comes first diff --git a/tests/measure b/tests/measure index 8cc4ed1..3b1f609 100755 --- a/tests/measure +++ b/tests/measure @@ -598,6 +598,14 @@ assert_grep anchor-override "OVERRIDE" "$r" assert_grep anchor-override-why "read by hand" "$r" assert_grep anchor-committed "^committed " "$r" assert_grep anchor-carried "captures agree on the list" "$r" +# declared on captures that carry the committed list, the override was not +# needed: the block says so and does not claim a mismatch its own two digests +# contradict +r=$("$M" attribute --dir "$caps" --unit u1 --control c1 --control-again c2 \ + --control-again c3 --state-paths-override "declared for no reason" || true) +assert_no_grep anchor-unneeded-claim "^OVERRIDE " "$r" +assert_grep anchor-unneeded "^override declared and not needed" "$r" +assert_grep anchor-unneeded-why "declared for no reason" "$r" info "17: blind because nothing was sampled" capture c1 s1 --state-max-bytes 8 > /dev/null From fb6c1edc79e3c315bd135cc099ee70edf7538865 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Tue, 29 Sep 2026 03:19:27 +0000 Subject: [PATCH 9/9] test: assert which fixture paths were sampled, not how many tests/measure asserted that exactly four fixture files matched the committed state path list. Since the suite takes its captures against the committed list, adding a state path that happens to match a fixture file, ./etc/keel/** or ./var/lib/dpkg/** for instance, broke the suite with a count mismatch that said nothing about sampling. It now asserts the four paths that must be sampled and two that must not, so the list can grow without touching the suite. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01SfWQScmDZ94KCS5DMYrfe6 --- tests/measure | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/tests/measure b/tests/measure index 3b1f609..5526a00 100755 --- a/tests/measure +++ b/tests/measure @@ -300,7 +300,14 @@ s=$tmp/sample mkdir -p "$s" measure_tree_record "$fix/c1" > "$s/tree.tsv" measure_state_sample "$fix/c1" "$s/tree.tsv" "$s" 100000 "$globs" -assert_eq sample-count "$(wc -l < "$s/state.tsv")" 4 +# which fixture paths were sampled, not how many, so a state path added to +# the committed list that happens to match a fixture file does not break this +assert_grep sample-has-shadow "^\./etc/shadow sampled " "$(cat "$s/state.tsv")" +assert_grep sample-has-priv "^\./var/lib/mysql/mysql/global_priv\.MAD sampled " "$(cat "$s/state.tsv")" +assert_grep sample-has-fixed "^\./var/lib/mysql/fixed\.cnf sampled " "$(cat "$s/state.tsv")" +assert_grep sample-has-ibdata "^\./var/lib/mysql/ibdata1 sampled " "$(cat "$s/state.tsv")" +assert_no_grep sample-not-hostname "^\./etc/hostname " "$(cat "$s/state.tsv")" +assert_no_grep sample-not-log "^\./var/log/" "$(cat "$s/state.tsv")" assert_eq sample-shadow "$(cat "$s/state/etc/shadow")" "$(cat "$fix/c1/etc/shadow")" assert_grep sample-flag " sampled " "$(cat "$s/state.tsv")" s2=$tmp/sample-cap