Name the command decision 0010 step 3 asks for, and what a recount of the two extractions found

docs/decisions/0010-composable-units.md

@@ -131,8 +131,23 @@ Differences attributable to the unit form, that is the control-vs-unit set
 minus the control-vs-control set: **zero**. The unit build differs from the
 control by exactly the files by which the control differs from itself.
 
+> Superseded by "What a recount found" below. The captures are still on the
+> build host and a recount of them gives 182, 179 and 182, not 173 three
+> times, and a residue of three files against the control pair this section
+> used. The conclusion of zero survives, against the union of both control
+> pairs; the counts in this table do not. The numbers are left as they were
+> reported, because what this note is now about is that nobody could check
+> them.
+
 ### The 173, named and explained
 
+> Superseded, with the table above, by "What a recount found" below. The
+> heading's own number is one of the wrong ones: the recount gives 182 files
+> for the control-unit pair and 173 for `/var/lib/mysql/**` alone, not 173 and
+> 167. The explanations in this table still hold for the paths they name; the
+> counts do not, and the row for `/var/lib/mysql/**` describes files nobody
+> opened.
+
 | Count | Path | Difference |
 | --- | --- | --- |
 | 167 | `/var/lib/mysql/**` | the MariaDB data directory created when the conf script starts the server. Every file is the same size and 8 to 11 bytes differ. In `mysql/user.frm` the bytes are the table creation time to the microsecond, `1790475089620377` against `1790475323514142`, the two wall clocks four minutes apart. The `.MAI`, `.MAD`, `.ibd`, `aria_log_control` and `ib_logfile0` differences are the matching creation stamps and log sequence numbers. |
@@ -263,9 +278,176 @@ The order to do this in, if the maintainer wants it:
    this note already lists as the main cost, or rule that a component needing
    either one stays in `common`. The second is cheaper and covers `mysql`,
    whose removelist is empty and whose variable is dead.
-3. Only then move components one at a time as we touch them, re-running this
+3. Only then move components one at a time as we touch them, re-running the
    three-build comparison for each, including the control-vs-control run.
-   Never as a migration.
+   Never as a migration. The command is `bt-layer-measure`, in `buildtasks`
+   (see below); a component extraction is not reviewable without the block it
+   prints.
+
+### The command, and what a pull request has to quote
+
+Added 2026-09-28, closing the gap this note left. Step 3 above asked for "this
+three-build comparison" and named nothing runnable: the capture was
+`/root/measure/build.sh` on the build host, a scratch file in root's home
+directory on one machine, and the comparison was ad hoc shell that was never
+committed. Two extractions, keel-mariadb#11 and keel-postgresql#7, were
+therefore approved on a number only their author could produce, which is
+testimony and not evidence. The comparison is now `bt-layer-measure` in
+`buildtasks`, documented in that repository's `docs/layer-measure.md`:
+
+    M=/mnt/builds/measure/<component>-$(date +%Y%m%d)
+    bt-layer-measure capture --dir $M --layer <layer> --parent <parent> --tag control
+    # check out the branch that supplies the component through unit.d
+    bt-layer-measure capture --dir $M --layer <layer> --parent <parent> --tag unit
+    # check out the control tree again, and capture it twice
+    bt-layer-measure capture --dir $M --layer <layer> --parent <parent> --tag control-again
+    bt-layer-measure capture --dir $M --layer <layer> --parent <parent> --tag control-third
+    bt-layer-measure attribute --dir $M --control control --unit unit \
+        --control-again control-again --control-again control-third
+
+Three controls, not two: `--control-again` is repeatable and the floor is the
+union over every pair of controls, for the reason under "One control pair is
+not a floor" below. Every capture at one `SOURCE_DATE_EPOCH`, one `common`
+commit and one parent layer, or nothing the comparison prints means anything;
+`attribute` refuses captures that disagree on layer, parent or epoch, or were
+taken against a state path list other than the committed one unless the run
+declares why with `--state-paths-override`. `capture` takes
+`/run/lock/keel-build.lock` with a bounded wait and writes into the directory
+named on the command line, so a reviewer re-runs the same command against the
+same directory rather than trusting a number.
+
+**A pull request extracting a component quotes the block `attribute` prints,
+whole.** It names the command, every capture with its epoch and capture time,
+the state path list's digest beside the committed one's, what each single
+control pair would have said, the attributable count, every state path with
+its verdict and the evidence for it, every waiver rule with the paths it
+cleared, and a PASS or FAIL line. A claim of zero attributable differences
+that does not carry that block is not reviewable, and the extraction is not
+approved on it.
+A PASS means that nothing the change did shows up as a difference from the
+controls; it does not mean the layer ships no secret it should not, because
+state identical in every capture is invisible to a differential measurement.
+
+### What a recount found, 2026-09-28
+
+The captures of both extractions are still on the build host
+(`/root/tree-m-{control,unit,control2,control3}.sha256` and the `p-` set for
+postgresql). Recounting them on a tab separated `path<TAB>hash` key, which is
+what `bt-layer-measure` does, gives different numbers from the ones above and
+from the ones the two pull requests quoted. The recount, not the earlier
+count, is what the captures say:
+
+| pair | mariadb | of which `./var/lib/mysql/**` | postgresql | of which `./var/lib/postgresql/**` |
+| --- | --- | --- | --- | --- |
+| control vs unit | 182 | 173 | 17 | 6 |
+| control vs control again | 179 | 173 | 13 | 6 |
+| control again vs a third control | 182 | 173 | 16 | 6 |
+
+The captures these come from predate the format `bt-layer-measure` reads:
+they are bare `sha256sum` output, `<hash>` then two spaces then the path, so
+the tool cannot read them and the recount had to convert them first. Under
+the rule this note introduces, that command belongs here rather than in
+somebody's shell history. It is the whole of it:
+
+```
+key() {
+    sed 's/^\([0-9a-f]\{64\}\)  /\1\t/' "$1" \
+        | awk -F'\t' -v OFS='\t' '{ print $2, $1 }' | LC_ALL=C sort
+}
+differing() {
+    join -t"$(printf '\t')" -j1 <(key "/root/tree-$1.sha256") <(key "/root/tree-$2.sha256") \
+        | awk -F'\t' '$2 != $3 { print $1 }'
+}
+differing m-control m-unit | wc -l
+comm -23 <(differing m-control m-unit | sort -u) <(differing m-control m-control2 | sort -u)
+```
+
+The conversion is the point of the `key` function: it moves the path into
+field 1 and separates it with a tab, which is what the captures should have
+been written as in the first place. Captures taken from now on are in the
+tool's own format and need none of this.
+
+Three things follow, and the first two are why this note now names a command.
+
+**One control pair is not a floor.** A union floor is more permissive than a
+single pair, not more rigorous, since it subtracts strictly more; it is the
+right test because its answer does not depend on which two builds were named
+first. Against `control vs control again` alone,
+mariadb leaves three differences unaccounted for:
+`./usr/share/doc/turnkey-mariadb-19.0/changelog`,
+`./var/lib/dpkg/info/turnkey-mariadb-19.0.md5sums` and
+`./var/lib/dpkg/status`. All three are inside the *other* control pair's floor,
+so they are install-time noise, but the pair the measurement happened to use
+did not show it. Against the union of the two control pairs the mariadb residue
+is zero, which is the result the pull request claimed, reached by an argument it
+did not make. `bt-layer-measure attribute` takes `--control-again` as many
+times as there are control captures and subtracts the union, and prints what
+each single pair would have said so the swing is on the record.
+
+**The postgresql extraction was not at zero.** Against either control pair, and
+against the union of both, one difference is left:
+`./etc/postgresql/17/main/postgresql.conf`. It is one line, and it is real:
+
+```
+-listen_addresses = '::1,127.0.0.1'   # set by conf.d/main of keel-postgresql: see the comment there
++listen_addresses = '::1,127.0.0.1'   # set by the postgresql component: see the comment in its conf script
+```
+
+The value is identical and only the trailing comment moved, because the
+component's conf script writes a different comment than `conf.d/main` did. It
+is harmless, and it is exactly the sort of thing the measurement exists to
+surface: a file the component writes, differing because the component wrote it.
+It was reported as zero.
+
+**A path inside the noise floor was never evidence of noise.** 173 of the
+mariadb differences are `./var/lib/mysql/**`, subtracted because of where they
+are. That directory holds `mysql/global_priv` and `mysql/user.*`, and all four
+of `global_priv.MAD`, `global_priv.MAI`, `global_priv.frm` and `user.frm`
+differ in the control-vs-unit pair. The component's build-time job includes
+deleting accounts. So the measurement subtracted, as noise, every file the
+thing being measured would have shown up in. Nothing in it was wrong; nothing
+in it was evidence either.
+
+`bt-layer-measure` closes all three. It takes the captures the step above asks
+for and reports the residue against the union of every control pair; it keeps
+the bytes of every path that can hold accounts, credentials, keys or database
+content (`share/layer-state-paths` in `buildtasks`) and gives each state path
+the unit pair differs at a verdict from those bytes, not from its name. A
+position the controls do not vary at is `real`. A text line at a position they
+do vary at is `noise` only if it has the shape they vary with there and the
+part that varies is pinned: the same short length in every control, digits and
+date punctuation only, as a timestamp is. Anything else there, a password hash,
+a host key, a value copied from one of the controls, is `overlapping`. Bytes
+carry no shape, so a binary file is never `noise`: at best it is
+`overlapping`. `real`, `overlapping` and a state path whose bytes were not kept
+(`not sampled`) all fail the run, because a file the measurement could not read
+is not a file that did not change. Only `overlapping` can be cleared, by a
+committed waiver that names its paths, how many it covers and the captures it
+was justified against, and the block prints the bytes of every path it clears.
+
+It also keys every record on `path<TAB>...` with the path in field 1, rather
+than on `sha256sum`'s default `<hash>` plus two spaces plus `<path>`. That
+default is what made
+`setuptools/_vendor/jaraco/text/Lorem ipsum.txt` two fields that never matched
+themselves, so the LAMP measurement of 2026-09-27 reported 199 and 183
+differing files where the true counts were 198 and 182 (`docs/traps.md`, "A
+path with a space in it"). A path carrying a tab, a newline or a backslash is
+refused at capture time rather than mis-counted.
+
+The first run of the command, against mariadb with three controls on
+2026-09-28, gives 0 attributable against every single pair and against the
+union, and FAIL on the state paths: 102 `noise`, 33 `overlapping`, 32 `real`.
+The 32 are almost all `*.MAI`, where bytes 180 to 183 of the Aria index header
+are a wall clock Unix second that `SOURCE_DATE_EPOCH` does not touch. No finite
+set of controls spans its carries, so they are open in Keel-Linux/buildtasks#13
+rather than waived.
+
+Neither extraction is being reopened on this. Both layers are built, published
+and boot-tested, the postgresql residue is a comment, and in the mariadb run
+the account set in `global_priv` is identical in all four captures. What
+changes is the standard: from here on the block `bt-layer-measure attribute`
+prints is what a component extraction is reviewed on, and a residue is named
+and explained in the pull request rather than absent from it.
 
 ### What it costs either way
 

docs/traps.md

@@ -326,6 +326,65 @@ today, which is what makes this easy to miss and easy to dismiss.
 **Hit** 2026-09-27, the LAMP composition measurement: 199 and 183 differing
 files, which are 198 and 182.
 
+## One control pair is not a noise floor
+
+**Signature.** An equivalence measurement reports a small residue of files
+that look like nothing to do with the change, typically a package's own
+`changelog`, `md5sums` and `/var/lib/dpkg/status`. Or, worse, it reports zero
+and a second control pair would not have.
+
+**Cause.** Two builds of the same recipe do not differ by a fixed set. The
+mariadb captures of 2026-09-27 differ by 179 files in one control pair and 182
+in another, so a difference attributed to the change against the first pair may
+be ordinary noise the first pair happened not to show. Subtracting one pair
+answers "is this difference in that pair", not "is this difference noise".
+
+**Fix.** Capture at least three controls and give every one of them to
+`bt-layer-measure attribute` in `buildtasks`: `--control-again` is repeatable
+and the floor it subtracts is the union over every pair, so no single pair
+decides the verdict. The block also prints what each pair on its own would
+have said, which is where the swing shows. Name the residue in the pull
+request rather than leaving it out: the postgresql extraction's residue was
+one line, the trailing comment on `listen_addresses` in `postgresql.conf`,
+which the component's conf script writes and `conf.d/main` used to.
+
+A union floor is more permissive, not more rigorous: it subtracts strictly
+more than either pair, so zero against the union proves less than zero
+against one pair would have. It is the right test because it is the one whose
+answer does not depend on which two builds were named first, not because it
+is the harder one.
+
+**Hit** 2026-09-28, recounting the mariadb and postgresql extractions: 3 and 1
+files that the merged pull requests reported as zero.
+
+## A differing path inside the noise floor is not a noise difference
+
+**Signature.** An equivalence measurement passes, and the change it was
+measuring was one whose whole effect lands in a directory the measurement
+subtracted by name.
+
+**Cause.** A path is put inside the noise floor by where it is. Whether its
+difference is really noise is a question about its bytes. For mariadb 173 of
+the 179 noise-floor files are `/var/lib/mysql/**`, which holds
+`mysql/global_priv` and `mysql/user.*`, and the component's build-time job
+includes deleting accounts.
+
+**Fix.** Keep the bytes of anything that can hold accounts, credentials, keys
+or database content (`share/layer-state-paths` in `buildtasks`) and judge each
+differing state path by its bytes, which is what `bt-layer-measure attribute`
+does. Do not cancel a difference against the controls by position alone: two
+unrelated changes at one line number or one byte offset look alike, and an
+account added on the line a clock moves on would pass. A text line is noise
+only if the controls pin what varies there, the same short length and only
+digits and date punctuation, as a timestamp is; a binary difference is never
+shown to be noise and has to be read and cleared in writing. A file the
+measurement could not read is not a file that did not change, so an unsampled
+state path fails the run.
+
+**Hit** 2026-09-27, both component extractions: `global_priv.MAD`,
+`global_priv.MAI`, `global_priv.frm` and `user.frm` all differ in the
+control-unit pair and all were subtracted unexamined.
+
 ## Two machines answer to `tkldev`, and they run different fab
 
 **Signature.** A claim about the build order, or about anything else in