diff --git a/docs/decisions/0010-composable-units.md b/docs/decisions/0010-composable-units.md index 36ef379..82964ab 100644 --- a/docs/decisions/0010-composable-units.md +++ b/docs/decisions/0010-composable-units.md @@ -131,8 +131,23 @@ Differences attributable to the unit form, that is the control-vs-unit set minus the control-vs-control set: **zero**. The unit build differs from the control by exactly the files by which the control differs from itself. +> Superseded by "What a recount found" below. The captures are still on the +> build host and a recount of them gives 182, 179 and 182, not 173 three +> times, and a residue of three files against the control pair this section +> used. The conclusion of zero survives, against the union of both control +> pairs; the counts in this table do not. The numbers are left as they were +> reported, because what this note is now about is that nobody could check +> them. + ### The 173, named and explained +> Superseded, with the table above, by "What a recount found" below. The +> heading's own number is one of the wrong ones: the recount gives 182 files +> for the control-unit pair and 173 for `/var/lib/mysql/**` alone, not 173 and +> 167. The explanations in this table still hold for the paths they name; the +> counts do not, and the row for `/var/lib/mysql/**` describes files nobody +> opened. + | Count | Path | Difference | | --- | --- | --- | | 167 | `/var/lib/mysql/**` | the MariaDB data directory created when the conf script starts the server. Every file is the same size and 8 to 11 bytes differ. In `mysql/user.frm` the bytes are the table creation time to the microsecond, `1790475089620377` against `1790475323514142`, the two wall clocks four minutes apart. The `.MAI`, `.MAD`, `.ibd`, `aria_log_control` and `ib_logfile0` differences are the matching creation stamps and log sequence numbers. | @@ -263,9 +278,176 @@ The order to do this in, if the maintainer wants it: this note already lists as the main cost, or rule that a component needing either one stays in `common`. The second is cheaper and covers `mysql`, whose removelist is empty and whose variable is dead. -3. Only then move components one at a time as we touch them, re-running this +3. Only then move components one at a time as we touch them, re-running the three-build comparison for each, including the control-vs-control run. - Never as a migration. + Never as a migration. The command is `bt-layer-measure`, in `buildtasks` + (see below); a component extraction is not reviewable without the block it + prints. + +### The command, and what a pull request has to quote + +Added 2026-09-28, closing the gap this note left. Step 3 above asked for "this +three-build comparison" and named nothing runnable: the capture was +`/root/measure/build.sh` on the build host, a scratch file in root's home +directory on one machine, and the comparison was ad hoc shell that was never +committed. Two extractions, keel-mariadb#11 and keel-postgresql#7, were +therefore approved on a number only their author could produce, which is +testimony and not evidence. The comparison is now `bt-layer-measure` in +`buildtasks`, documented in that repository's `docs/layer-measure.md`: + + M=/mnt/builds/measure/-$(date +%Y%m%d) + bt-layer-measure capture --dir $M --layer --parent --tag control + # check out the branch that supplies the component through unit.d + bt-layer-measure capture --dir $M --layer --parent --tag unit + # check out the control tree again, and capture it twice + bt-layer-measure capture --dir $M --layer --parent --tag control-again + bt-layer-measure capture --dir $M --layer --parent --tag control-third + bt-layer-measure attribute --dir $M --control control --unit unit \ + --control-again control-again --control-again control-third + +Three controls, not two: `--control-again` is repeatable and the floor is the +union over every pair of controls, for the reason under "One control pair is +not a floor" below. Every capture at one `SOURCE_DATE_EPOCH`, one `common` +commit and one parent layer, or nothing the comparison prints means anything; +`attribute` refuses captures that disagree on layer, parent or epoch, or were +taken against a state path list other than the committed one unless the run +declares why with `--state-paths-override`. `capture` takes +`/run/lock/keel-build.lock` with a bounded wait and writes into the directory +named on the command line, so a reviewer re-runs the same command against the +same directory rather than trusting a number. + +**A pull request extracting a component quotes the block `attribute` prints, +whole.** It names the command, every capture with its epoch and capture time, +the state path list's digest beside the committed one's, what each single +control pair would have said, the attributable count, every state path with +its verdict and the evidence for it, every waiver rule with the paths it +cleared, and a PASS or FAIL line. A claim of zero attributable differences +that does not carry that block is not reviewable, and the extraction is not +approved on it. +A PASS means that nothing the change did shows up as a difference from the +controls; it does not mean the layer ships no secret it should not, because +state identical in every capture is invisible to a differential measurement. + +### What a recount found, 2026-09-28 + +The captures of both extractions are still on the build host +(`/root/tree-m-{control,unit,control2,control3}.sha256` and the `p-` set for +postgresql). Recounting them on a tab separated `pathhash` key, which is +what `bt-layer-measure` does, gives different numbers from the ones above and +from the ones the two pull requests quoted. The recount, not the earlier +count, is what the captures say: + +| pair | mariadb | of which `./var/lib/mysql/**` | postgresql | of which `./var/lib/postgresql/**` | +| --- | --- | --- | --- | --- | +| control vs unit | 182 | 173 | 17 | 6 | +| control vs control again | 179 | 173 | 13 | 6 | +| control again vs a third control | 182 | 173 | 16 | 6 | + +The captures these come from predate the format `bt-layer-measure` reads: +they are bare `sha256sum` output, `` then two spaces then the path, so +the tool cannot read them and the recount had to convert them first. Under +the rule this note introduces, that command belongs here rather than in +somebody's shell history. It is the whole of it: + +``` +key() { + sed 's/^\([0-9a-f]\{64\}\) /\1\t/' "$1" \ + | awk -F'\t' -v OFS='\t' '{ print $2, $1 }' | LC_ALL=C sort +} +differing() { + join -t"$(printf '\t')" -j1 <(key "/root/tree-$1.sha256") <(key "/root/tree-$2.sha256") \ + | awk -F'\t' '$2 != $3 { print $1 }' +} +differing m-control m-unit | wc -l +comm -23 <(differing m-control m-unit | sort -u) <(differing m-control m-control2 | sort -u) +``` + +The conversion is the point of the `key` function: it moves the path into +field 1 and separates it with a tab, which is what the captures should have +been written as in the first place. Captures taken from now on are in the +tool's own format and need none of this. + +Three things follow, and the first two are why this note now names a command. + +**One control pair is not a floor.** A union floor is more permissive than a +single pair, not more rigorous, since it subtracts strictly more; it is the +right test because its answer does not depend on which two builds were named +first. Against `control vs control again` alone, +mariadb leaves three differences unaccounted for: +`./usr/share/doc/turnkey-mariadb-19.0/changelog`, +`./var/lib/dpkg/info/turnkey-mariadb-19.0.md5sums` and +`./var/lib/dpkg/status`. All three are inside the *other* control pair's floor, +so they are install-time noise, but the pair the measurement happened to use +did not show it. Against the union of the two control pairs the mariadb residue +is zero, which is the result the pull request claimed, reached by an argument it +did not make. `bt-layer-measure attribute` takes `--control-again` as many +times as there are control captures and subtracts the union, and prints what +each single pair would have said so the swing is on the record. + +**The postgresql extraction was not at zero.** Against either control pair, and +against the union of both, one difference is left: +`./etc/postgresql/17/main/postgresql.conf`. It is one line, and it is real: + +``` +-listen_addresses = '::1,127.0.0.1' # set by conf.d/main of keel-postgresql: see the comment there ++listen_addresses = '::1,127.0.0.1' # set by the postgresql component: see the comment in its conf script +``` + +The value is identical and only the trailing comment moved, because the +component's conf script writes a different comment than `conf.d/main` did. It +is harmless, and it is exactly the sort of thing the measurement exists to +surface: a file the component writes, differing because the component wrote it. +It was reported as zero. + +**A path inside the noise floor was never evidence of noise.** 173 of the +mariadb differences are `./var/lib/mysql/**`, subtracted because of where they +are. That directory holds `mysql/global_priv` and `mysql/user.*`, and all four +of `global_priv.MAD`, `global_priv.MAI`, `global_priv.frm` and `user.frm` +differ in the control-vs-unit pair. The component's build-time job includes +deleting accounts. So the measurement subtracted, as noise, every file the +thing being measured would have shown up in. Nothing in it was wrong; nothing +in it was evidence either. + +`bt-layer-measure` closes all three. It takes the captures the step above asks +for and reports the residue against the union of every control pair; it keeps +the bytes of every path that can hold accounts, credentials, keys or database +content (`share/layer-state-paths` in `buildtasks`) and gives each state path +the unit pair differs at a verdict from those bytes, not from its name. A +position the controls do not vary at is `real`. A text line at a position they +do vary at is `noise` only if it has the shape they vary with there and the +part that varies is pinned: the same short length in every control, digits and +date punctuation only, as a timestamp is. Anything else there, a password hash, +a host key, a value copied from one of the controls, is `overlapping`. Bytes +carry no shape, so a binary file is never `noise`: at best it is +`overlapping`. `real`, `overlapping` and a state path whose bytes were not kept +(`not sampled`) all fail the run, because a file the measurement could not read +is not a file that did not change. Only `overlapping` can be cleared, by a +committed waiver that names its paths, how many it covers and the captures it +was justified against, and the block prints the bytes of every path it clears. + +It also keys every record on `path...` with the path in field 1, rather +than on `sha256sum`'s default `` plus two spaces plus ``. That +default is what made +`setuptools/_vendor/jaraco/text/Lorem ipsum.txt` two fields that never matched +themselves, so the LAMP measurement of 2026-09-27 reported 199 and 183 +differing files where the true counts were 198 and 182 (`docs/traps.md`, "A +path with a space in it"). A path carrying a tab, a newline or a backslash is +refused at capture time rather than mis-counted. + +The first run of the command, against mariadb with three controls on +2026-09-28, gives 0 attributable against every single pair and against the +union, and FAIL on the state paths: 102 `noise`, 33 `overlapping`, 32 `real`. +The 32 are almost all `*.MAI`, where bytes 180 to 183 of the Aria index header +are a wall clock Unix second that `SOURCE_DATE_EPOCH` does not touch. No finite +set of controls spans its carries, so they are open in Keel-Linux/buildtasks#13 +rather than waived. + +Neither extraction is being reopened on this. Both layers are built, published +and boot-tested, the postgresql residue is a comment, and in the mariadb run +the account set in `global_priv` is identical in all four captures. What +changes is the standard: from here on the block `bt-layer-measure attribute` +prints is what a component extraction is reviewed on, and a residue is named +and explained in the pull request rather than absent from it. ### What it costs either way diff --git a/docs/traps.md b/docs/traps.md index 8187648..34502f3 100644 --- a/docs/traps.md +++ b/docs/traps.md @@ -326,6 +326,65 @@ today, which is what makes this easy to miss and easy to dismiss. **Hit** 2026-09-27, the LAMP composition measurement: 199 and 183 differing files, which are 198 and 182. +## One control pair is not a noise floor + +**Signature.** An equivalence measurement reports a small residue of files +that look like nothing to do with the change, typically a package's own +`changelog`, `md5sums` and `/var/lib/dpkg/status`. Or, worse, it reports zero +and a second control pair would not have. + +**Cause.** Two builds of the same recipe do not differ by a fixed set. The +mariadb captures of 2026-09-27 differ by 179 files in one control pair and 182 +in another, so a difference attributed to the change against the first pair may +be ordinary noise the first pair happened not to show. Subtracting one pair +answers "is this difference in that pair", not "is this difference noise". + +**Fix.** Capture at least three controls and give every one of them to +`bt-layer-measure attribute` in `buildtasks`: `--control-again` is repeatable +and the floor it subtracts is the union over every pair, so no single pair +decides the verdict. The block also prints what each pair on its own would +have said, which is where the swing shows. Name the residue in the pull +request rather than leaving it out: the postgresql extraction's residue was +one line, the trailing comment on `listen_addresses` in `postgresql.conf`, +which the component's conf script writes and `conf.d/main` used to. + +A union floor is more permissive, not more rigorous: it subtracts strictly +more than either pair, so zero against the union proves less than zero +against one pair would have. It is the right test because it is the one whose +answer does not depend on which two builds were named first, not because it +is the harder one. + +**Hit** 2026-09-28, recounting the mariadb and postgresql extractions: 3 and 1 +files that the merged pull requests reported as zero. + +## A differing path inside the noise floor is not a noise difference + +**Signature.** An equivalence measurement passes, and the change it was +measuring was one whose whole effect lands in a directory the measurement +subtracted by name. + +**Cause.** A path is put inside the noise floor by where it is. Whether its +difference is really noise is a question about its bytes. For mariadb 173 of +the 179 noise-floor files are `/var/lib/mysql/**`, which holds +`mysql/global_priv` and `mysql/user.*`, and the component's build-time job +includes deleting accounts. + +**Fix.** Keep the bytes of anything that can hold accounts, credentials, keys +or database content (`share/layer-state-paths` in `buildtasks`) and judge each +differing state path by its bytes, which is what `bt-layer-measure attribute` +does. Do not cancel a difference against the controls by position alone: two +unrelated changes at one line number or one byte offset look alike, and an +account added on the line a clock moves on would pass. A text line is noise +only if the controls pin what varies there, the same short length and only +digits and date punctuation, as a timestamp is; a binary difference is never +shown to be noise and has to be read and cleared in writing. A file the +measurement could not read is not a file that did not change, so an unsampled +state path fails the run. + +**Hit** 2026-09-27, both component extractions: `global_priv.MAD`, +`global_priv.MAI`, `global_priv.frm` and `user.frm` all differ in the +control-unit pair and all were subtracted unexamined. + ## Two machines answer to `tkldev`, and they run different fab **Signature.** A claim about the build order, or about anything else in