report: catalog the 7.15 seed-generation evidence and state the report's scope - #30
Open
BitHighlander wants to merge 3 commits into
Open
report: catalog the 7.15 seed-generation evidence and state the report's scope#30BitHighlander wants to merge 3 commits into
BitHighlander wants to merge 3 commits into
Conversation
…t's scope
The PDF is the artifact a release review actually reads, and it was quietly
claiming more than it knew. Two defects, one visible consequence.
parse_junit only emitted a 'mod::meth' key when the JUnit classname contained
a dotted test_msg_*/test_sign_*/test_verify_* module. Native gtest suites carry
a bare classname ("Dice", "Storage"), so they produced no such key, and _lookup
has no bare-method fallback by design. CI merged the firmware-unit XMLs into the
report input and every one of the 432 native tests was then structurally
impossible to reference from SECTIONS. Bare classnames are now keyed as
'Suite::Test'.
The header reported "N/N PASSED" against the catalog with nothing saying the
catalog is a subset. A 7.15 RC audit grepped this PDF for feature keywords,
found no hits for dice and PIN KDF, and reported both as having zero coverage.
Both had in fact run green in the same CI run: test_reset_device_dice passed,
and so did all five Dice unit tests and the PIN KDF rewrap tests. The header now
states catalogued-vs-executed and says outright that absence here is not
evidence of absence.
New section K catalogues what that audit went looking for: the dice flow
end-to-end (digest equals SHA256 of exactly the injected rolls, then the
mnemonic is derived from post-mix internal entropy, which is what proves the
rolls reached the seed), the aborted-reset EntropyAck disarm regression, the
five Dice known-answer and independence vectors, and the v16->v19 PIN KDF
rewrap plus storage migration.
Verified against the 7.15.0 RC artifacts from run 31284108490: dice went from 0
to 13 occurrences in the rendered PDF, section K renders 11/11 passed, and
poisoning Dice::MixDependsOnRolls in the merged JUnit turns the header red and
fails --validate-junit, so the entries are wired to real results.
Needs the companion firmware change: the CI trigger validated against the
Python JUnit alone, where every native entry resolves to "missing".
Chasing a rendering defect on the per-output shielded confirm turned up something worse than a missing screenshot: ZcashSignPCZT is never sent to a device anywhere in this suite. Every test in test_msg_zcash_sign_pczt drives a ScriptedTransport with canned responses -- they are offline contract tests that prove the client builds and orders its messages correctly, and prove nothing whatsoever about firmware behaviour. The device-driven Zcash tests cover transparent signing, display-address, FVK derivation and the seed fingerprint. None of them signs a shielded output. So the on-device shielded path -- including the confirm screen that fsm_msg_zcash.h designates as the verification gate for Orchard output values, since total_amount is "a summary prompt" taken from the host -- has no automated coverage at all. The catalog gave no hint of this. The section Z entries read exactly like the device tests around them, and that is how a screen nobody has ever rendered sat behind seven green checks. Say it in the entry instead. No screenshot hint: requesting frames from a test that cannot reach a device would produce silently zero of them, which is the same class of empty-but-green evidence this whole pass exists to remove. Verified: --screenshot-filter does not select it, and the report still renders 325 tests and passes --validate-junit against the RC artifacts.
BitHighlander
force-pushed
the
fix/report-catalog-coverage
branch
from
August 9, 2026 04:49
b99e287 to
5761392
Compare
ZcashSignPCZT had never been sent to a device by anything in this suite.
Every test in test_msg_zcash_sign_pczt drives a ScriptedTransport with canned
responses; the device-driven Zcash tests cover transparent signing,
display-address, FVK derivation and the seed fingerprint. So the on-device
shielded path had no automated coverage at all, and seven green checks in
section Z read exactly like device coverage while proving only that the
client serialises its messages in the right order.
That is how a confirm screen which cannot physically fit its amount line
shipped unnoticed. The per-output shielded confirm is the verification gate
for Orchard output values -- total_amount on the summary prompt is taken
straight from the host message -- and a unified address is 106 characters,
three full body rows, against a three-row body. The amount never rendered.
The fixtures are the firmware's own known-answer vectors from
unittests/firmware/zcash.cpp, so the device's cmx recomputation accepts them
without needing a Pallas implementation in Python. The same note under both
pools commits to a different value, which is what makes the pool tests
possible at all.
Four tests:
- the output review is two screens, and they render differently and
non-blank (read_layout returns a framebuffer, not text, so the assertions
are structural rather than OCR)
- a one-bit change to the recipient breaks the commitment
- the Orchard commitment is refused when Ironwood is declared
- the Ironwood commitment for that same note is accepted
Two gates the offline fixtures do not satisfy had to be met for real
firmware: the header digest is recomputed and compared, and for a
shielded-only transaction the verified fee reduces to orchard_value_balance
and must equal the declared fee. Both are computed here rather than canned.
VERIFIED AS A REGRESSION TEST, not just written: run against the shipped
7.15.0 RC emulator (docker image from run 31284108490, the 27970b0c6 build)
it fails with "expected 2 ConfirmOutput screens, got 1", while the
commitment-binding and pool-selection tests pass. It reproduces the defect on
the firmware that has it.
Catalogued as Z22-Z25 with screenshot hints, so the shielded confirm screens
finally appear in the report -- the RC run captured 1037 OLED frames and not
one came from a shielded flow.
BitHighlander
added a commit
to BitHighlander/keepkey-firmware
that referenced
this pull request
Aug 9, 2026
Moves deps/python-keepkey to 3bbf996, which adds the first test that actually sends ZcashSignPCZT to a device. Until now every PCZT test drove a scripted transport with canned responses, so the on-device shielded path -- including the per-output confirm this branch fixes -- was never executed by anything. The new test is a real regression test for that fix, not a companion to it: run against the shipped 7.15.0 RC emulator it fails with "expected 2 ConfirmOutput screens, got 1". This pin is what lets CI run it against the fixed firmware. Also brings the report catalog changes: native firmware unit tests can now be referenced at all, the header states how much of the run the catalog covers, and the shielded confirm screens are finally requested as screenshots. PRACTICE PIN -- 3bbf996 is on an unmerged fork branch (BitHighlander/python-keepkey#30, stacked on keepkey/python-keepkey#211). Re-pin to a merged master SHA before tagging v7.15.0, along with device-protocol and trezor-firmware.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on keepkey#211 (
fix/display-random-ignored), whose head is the exact SHA the firmware pins — so this is pin + 1 commit, 1 file, +108/-1.Why
An RC audit of 7.15 grepped
test-report.pdffor feature keywords, found no hits for dice or PIN KDF, and reported both as having zero coverage. Both had run green in the same CI run (31284108490):test_reset_device_dicetest_reset_reentry_disarms_entropy_ackDice::RollsForStrength+ 4 moreDice::*Storage::PinKdfV16RewrapsToV19AfterCorrectPinThe report was structurally incapable of showing them.
Two defects
1. Native gtests could not be catalogued at all.
parse_junitonly emitted amod::methkey when the classname contained a dottedtest_msg_*/test_sign_*/test_verify_*module. gtest suites carry a bare classname (Dice,Storage), so they produced no key, and_lookuphas no bare-method fallback by design. CI merged the firmware-unit XMLs into the report input and then could not reference a single one of the 432 tests in them. Bare classnames are now keyedSuite::Test.2. The header implied the catalog was the suite. It reported
N/N PASSEDagainst the catalog with nothing saying the catalog is a subset — it covers 314 of 1237. It now states catalogued-vs-executed and says outright that absence is not evidence of absence.New section K
The dice flow end-to-end (the device digest must equal SHA256 of exactly the injected rolls, then the mnemonic is derived from post-mix internal entropy — which is what proves the rolls reached the seed rather than being collected and discarded), the aborted-reset EntropyAck disarm regression, the five Dice known-answer/independence vectors, and the v16→v19 PIN KDF rewrap plus storage migration.
Verification
Against the real RC artifacts from run 31284108490:
\bdice\bin the rendered PDF: 0 → 13; section K renders 11/11 passedDice::MixDependsOnRollsin the merged JUnit flips the header to1 FAILED of 325and--validate-junitreportsK6 -> fail— the entries are wired to real results, not decorativeRequires the companion firmware change
The CI trigger validated against the Python JUnit alone, where every native entry resolves to
missing— observed, not predicted. Paired with BitHighlander/keepkey-firmwarefix/report-native-unit-coverage. The pyk pin must move for section K to appear.