Repository navigation
fix: the seam's first hardware run - correct the burst record, and skips are not regressions - #53
Merged
Merged
Conversation
…ty, as measured The first hardware run after PR #52 merged failed two tests on facts the seam changed. connect() now reads Version before the handshake, so a listener registered through before_handshake sees three inbound Version messages: the full reply to that read (+0.71 s), the unit's own Version{READ} 1 ms behind it, and the announce answer carrying cortex_control_version_valid (+0.73 s). The state layer is listening by then, so the identity entry holds device_serial_number and app_fw_version ahead of the burst; the reply also carries unkept fields, so the entry stays marked and the first read still asks the unit once. Measured 2026-09-06 on Quad Cortex, CorOS 4.0.1 / d14e, with a listener on every inbound Version through connect and a 12 s burst. Both tests and docs/protocol.md section 4 now state it; the 2026-08-27 measurement of one Version stands for _hello() alone.
On the same run the bypass echo test skipped (the loaded preset had no stored bypass entry) and the report printed set_bypass under "regressions by name". A skipped test measured nothing about the unit; it belongs on the "failed or skipped" line and nowhere else. The last line now lists only operations some test FAILED, labelled "VERIFIED and claimed by a test, failed", and an offline test pins a skipped-only operation to the first line alone.
…ays three Version messages, and the count's failure names its causes
… says when it measured nothing Triage of PR #53 found four things worth fixing before merge. The burst test counted Version messages, so a unit that was still booting - which makes connect() retry its identity READ and adds a full reply plus the unit's own READ per retry - would have failed a healthy run as a regression. HandshakeBurst now records each Version's (action, fields) and the test asserts composition: one announce answer, at least one full reply, one unit READ per full reply, nothing else; the number of identity reads is recorded, not asserted. The report printed an empty regression line on an all-skip run, identical to a healthy run. It now prints test counts and how many claimed operations were measured, and flags a run where nothing passed. A session that never connected gets a line saying so instead of silence; the first run of that line was this commit's own hardware check, made with the unit unplugged. Also: the stale "not passed" label in two docstrings; the untested ["passed", "skipped"] cell; architecture.md's "two entries" (five); "about 1 ms" where the older record says 0.5-0.8 ms; and the identity assertions say a reply that stopped carrying serial or firmware is a protocol.md finding, not a reason to loosen them.
…ytest files apart from failures
…collected Seen on the unit 2026-09-07 while running the three burst tests with -k: the report said "operations measured: 0 of 16 claimed" and flagged NOTHING PASSED, although the run had deselected every marked test on purpose. pytest's own -k deselection runs AFTER this conftest's collection hook, so recording claims there counted tests that never ran. Claims are now built at report time from the node ids that produced a report (_claimed, pure, held offline); the same narrowed run reports "0 of 0 claimed" with no flag, and a --verifies set_scene_label run reports 1 of 1 with the operation under passed. Verified on Quad Cortex CorOS 4.0.1 / d14e: the shape-based burst assertion passes against real traffic (one announce answer, one full reply, one unit READ), and the three corrected burst tests pass.
jonathanstokes
added a commit
that referenced
this pull request
Sep 8, 2026
…ed as pytest prints it, and the rule's evidence is PR #53 Triage of the draft found: the template still carried the old optional-hardware rule; "totals line" named output only PR #53's branch prints; the two files disagreed on the no-unit state; nothing tied the run to a commit; the section's parent intro still said "if you want to"; the prerequisites lived elsewhere; "only evidence" overclaimed and dropped the manual read-back ask; and STEERING's pattern row still said hardware verification is manual via examples/.
jonathanstokes
added a commit
that referenced
this pull request
Sep 8, 2026
…ed as pytest prints it, and the rule's evidence is PR #53 Triage of the draft found: the template still carried the old optional-hardware rule; "totals line" named output only PR #53's branch prints; the two files disagreed on the no-unit state; nothing tied the run to a commit; the section's parent intro still said "if you want to"; the prerequisites lived elsewhere; "only evidence" overclaimed and dropped the manual read-back ask; and STEERING's pattern row still said hardware verification is manual via examples/.
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The first
pytest tests/hardware --hardwareafter #52 merged, on the maintainer's Quad Cortex (CorOS 4.0.1 / d14e): 93 passed, 4 skipped, 2 failed. Both failures were records the seam made stale, not regressions, and the run exposed one report defect. This PR corrects all three, with the measurement.Changes
connect()now readsVersionbefore the handshake, so a listener registered throughbefore_handshakesees three inboundVersionmessages, not one: the full reply to that read (+0.71 s), the unit's ownVersion{READ}1 ms behind it, and the announce answer (+0.73 s). The identity entry holds serial and firmware from that reply ahead of the burst.tests/hardware/test_model_state.py,tests/hardware/test_preset_surface.pyanddocs/protocol.mdsection 4 now say so; the 2026-08-27 measurement of oneVersionstands for_hello()alone.set_bypass, whose echo test skipped for want of a stored bypass entry). It now counts failures only and is labelled "VERIFIED and claimed by a test, failed"; an offline test pins the skipped-only case.Triage
Run with the full triage-pr pass (four reviewers): 11 findings, 8 fixed on the branch, 3 deferred (a producer-layer pin of pytest's outcome vocabulary; xfail semantics; the old label in the historical plan and spec). Two re-triage rounds of the fix commits found and fixed two more (the count line dropped setup/teardown errors; claims were recorded before pytest's own -k deselection).
Test plan
git diff --checkclean.Versionthrough connect and a 12 s burst (3 messages, composition above); the two corrected tests after the change (2 passed) with the report's last line empty; after the triage fixes, the three burst tests including the shape-based assertion (3 passed) and two narrowed runs proving the report's claims come from the tests that ran (0 of 0 claimed with no flag; 1 of 1 with set_scene_label under passed).