Skip to content

fix: the seam's first hardware run - correct the burst record, and skips are not regressions - #53

Merged
jonathanstokes merged 7 commits into
mainfrom
fix/post-merge-hardware-record
Sep 8, 2026
Merged

jonathanstokes merged 7 commits into
mainfrom
fix/post-merge-hardware-record

Conversation

@jonathanstokes

@jonathanstokes jonathanstokes commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The first pytest tests/hardware --hardware after #52 merged, on the maintainer's Quad Cortex (CorOS 4.0.1 / d14e): 93 passed, 4 skipped, 2 failed. Both failures were records the seam made stale, not regressions, and the run exposed one report defect. This PR corrects all three, with the measurement.

Changes

  • connect() now reads Version before the handshake, so a listener registered through before_handshake sees three inbound Version messages, not one: the full reply to that read (+0.71 s), the unit's own Version{READ} 1 ms behind it, and the announce answer (+0.73 s). The identity entry holds serial and firmware from that reply ahead of the burst. tests/hardware/test_model_state.py, tests/hardware/test_preset_surface.py and docs/protocol.md section 4 now say so; the 2026-08-27 measurement of one Version stands for _hello() alone.
  • The report's regression line counted a skipped test as a regression (set_bypass, whose echo test skipped for want of a stored bypass entry). It now counts failures only and is labelled "VERIFIED and claimed by a test, failed"; an offline test pins the skipped-only case.
  • Changelog entry under Unreleased.

Triage

Run with the full triage-pr pass (four reviewers): 11 findings, 8 fixed on the branch, 3 deferred (a producer-layer pin of pytest's outcome vocabulary; xfail semantics; the old label in the historical plan and spec). Two re-triage rounds of the fix commits found and fixed two more (the count line dropped setup/teardown errors; claims were recorded before pytest's own -k deselection).

Test plan

  • Offline suite 2695 passed, 3 skipped; mypy clean; git diff --check clean.
  • Hardware, 2026-09-07, CorOS 4.0.1 / d14e: the full suite before this change (93 passed, 2 failed, 4 skipped); a probe with a listener on every inbound Version through connect and a 12 s burst (3 messages, composition above); the two corrected tests after the change (2 passed) with the report's last line empty; after the triage fixes, the three burst tests including the shape-based assertion (3 passed) and two narrowed runs proving the report's claims come from the tests that ran (0 of 0 claimed with no flag; 1 of 1 with set_scene_label under passed).

…ty, as measured

The first hardware run after PR #52 merged failed two tests on facts the
seam changed. connect() now reads Version before the handshake, so a
listener registered through before_handshake sees three inbound Version
messages: the full reply to that read (+0.71 s), the unit's own
Version{READ} 1 ms behind it, and the announce answer carrying
cortex_control_version_valid (+0.73 s). The state layer is listening by
then, so the identity entry holds device_serial_number and app_fw_version
ahead of the burst; the reply also carries unkept fields, so the entry
stays marked and the first read still asks the unit once.

Measured 2026-09-06 on Quad Cortex, CorOS 4.0.1 / d14e, with a listener
on every inbound Version through connect and a 12 s burst. Both tests and
docs/protocol.md section 4 now state it; the 2026-08-27 measurement of one
Version stands for _hello() alone.
On the same run the bypass echo test skipped (the loaded preset had no
stored bypass entry) and the report printed set_bypass under "regressions
by name". A skipped test measured nothing about the unit; it belongs on
the "failed or skipped" line and nowhere else. The last line now lists
only operations some test FAILED, labelled "VERIFIED and claimed by a
test, failed", and an offline test pins a skipped-only operation to the
first line alone.
…ays three Version messages, and the count's failure names its causes
… says when it measured nothing

Triage of PR #53 found four things worth fixing before merge. The burst
test counted Version messages, so a unit that was still booting - which
makes connect() retry its identity READ and adds a full reply plus the
unit's own READ per retry - would have failed a healthy run as a
regression. HandshakeBurst now records each Version's (action, fields)
and the test asserts composition: one announce answer, at least one full
reply, one unit READ per full reply, nothing else; the number of identity
reads is recorded, not asserted.

The report printed an empty regression line on an all-skip run, identical
to a healthy run. It now prints test counts and how many claimed
operations were measured, and flags a run where nothing passed. A session
that never connected gets a line saying so instead of silence; the first
run of that line was this commit's own hardware check, made with the unit
unplugged.

Also: the stale "not passed" label in two docstrings; the untested
["passed", "skipped"] cell; architecture.md's "two entries" (five);
"about 1 ms" where the older record says 0.5-0.8 ms; and the identity
assertions say a reply that stopped carrying serial or firmware is a
protocol.md finding, not a reason to loosen them.
…collected

Seen on the unit 2026-09-07 while running the three burst tests with -k:
the report said "operations measured: 0 of 16 claimed" and flagged
NOTHING PASSED, although the run had deselected every marked test on
purpose. pytest's own -k deselection runs AFTER this conftest's
collection hook, so recording claims there counted tests that never ran.
Claims are now built at report time from the node ids that produced a
report (_claimed, pure, held offline); the same narrowed run reports
"0 of 0 claimed" with no flag, and a --verifies set_scene_label run
reports 1 of 1 with the operation under passed.

Verified on Quad Cortex CorOS 4.0.1 / d14e: the shape-based burst
assertion passes against real traffic (one announce answer, one full
reply, one unit READ), and the three corrected burst tests pass.
jonathanstokes added a commit that referenced this pull request Sep 8, 2026
…ed as pytest prints it, and the rule's evidence is PR #53

Triage of the draft found: the template still carried the old optional-hardware
rule; "totals line" named output only PR #53's branch prints; the two files
disagreed on the no-unit state; nothing tied the run to a commit; the section's
parent intro still said "if you want to"; the prerequisites lived elsewhere;
"only evidence" overclaimed and dropped the manual read-back ask; and STEERING's
pattern row still said hardware verification is manual via examples/.
@jonathanstokes
jonathanstokes merged commit fec8737 into main Sep 8, 2026
5 checks passed
@jonathanstokes
jonathanstokes deleted the fix/post-merge-hardware-record branch September 8, 2026 02:16
jonathanstokes added a commit that referenced this pull request Sep 8, 2026
…ed as pytest prints it, and the rule's evidence is PR #53

Triage of the draft found: the template still carried the old optional-hardware
rule; "totals line" named output only PR #53's branch prints; the two files
disagreed on the no-unit state; nothing tied the run to a commit; the section's
parent intro still said "if you want to"; the prerequisites lived elsewhere;
"only evidence" overclaimed and dropped the manual read-back ask; and STEERING's
pattern row still said hardware verification is manual via examples/.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant