From 710eedce5e7483d35075e7ddcc0197cb4ad2826a Mon Sep 17 00:00:00 2001 From: Jonathan Stokes Date: Mon, 7 Sep 2026 18:22:47 -0500 Subject: [PATCH 1/7] fix: the burst record says three Version messages and a warmed identity, as measured The first hardware run after PR #52 merged failed two tests on facts the seam changed. connect() now reads Version before the handshake, so a listener registered through before_handshake sees three inbound Version messages: the full reply to that read (+0.71 s), the unit's own Version{READ} 1 ms behind it, and the announce answer carrying cortex_control_version_valid (+0.73 s). The state layer is listening by then, so the identity entry holds device_serial_number and app_fw_version ahead of the burst; the reply also carries unkept fields, so the entry stays marked and the first read still asks the unit once. Measured 2026-09-06 on Quad Cortex, CorOS 4.0.1 / d14e, with a listener on every inbound Version through connect and a 12 s burst. Both tests and docs/protocol.md section 4 now state it; the 2026-08-27 measurement of one Version stands for _hello() alone. --- docs/protocol.md | 23 ++++++++---- tests/hardware/test_model_state.py | 50 +++++++++++++++------------ tests/hardware/test_preset_surface.py | 16 +++++---- 3 files changed, 54 insertions(+), 35 deletions(-) diff --git a/docs/protocol.md b/docs/protocol.md index 642773d..853688d 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -524,11 +524,20 @@ version request - READ replies carry no `request_id` to disambiguate them, so the transport hands a `Version` with no id to whichever waiter is first in line. Skipping it also means the device asks nothing back. Measured 2026-08-27 on -`d14e`: a connect through `_hello()` and its whole burst produce exactly one -inbound `Version`, an `UPDATE` carrying `cortex_control_version_valid` in answer -to the announce, and eight seconds of idling after it produce none. So step 3 -above is a consequence of step 2 rather than something the device does on -connecting. +`d14e`: `_hello()` and its whole burst produce exactly one inbound `Version`, an +`UPDATE` carrying `cortex_control_version_valid` in answer to the announce, and +eight seconds of idling after it produce none. So step 3 above is a consequence +of step 2 rather than something the device does on connecting. + +Since ADR-0020, `connect()` itself makes one `Version` READ BEFORE calling +`_hello()`, to resolve the profile. Measured 2026-09-06 on CorOS 4.0.1 / d14e: a +`connect()` and its burst now carry exactly THREE inbound `Version` messages - +the full reply to that READ (15 fields, at +0.71 s), the unit's own +`Version{READ}` 1 ms behind it (section 4.4), and the announce answer above (at ++0.73 s). The state layer listens from before the handshake, so the identity +entry holds `device_serial_number` and `app_fw_version` from that reply; the +reply also carries fields the entry does not keep, so the entry stays marked and +the first `device.firmware` read still asks the unit once. ### 4.3 Keepalive and disconnect @@ -2029,7 +2038,9 @@ transport listener registered before the handshake: `connect()` handed back its in, having seen only the `ResetCommsBuffers` echo and one `Version`. (That session read the `Version` as the unit's own READ. Re-measured 2026-08-27 on d14e it is the unit's `UPDATE` answering our version announce, carrying `cortex_control_version_valid`; `_hello()` sends no -host `Version` READ, and with none sent the unit asks nothing back. See section 4.) The +host `Version` READ, and with none sent the unit asks nothing back. Since ADR-0020 `connect()` +reads `Version` once before `_hello()`, so a listener registered before the handshake now sees +three `Version` messages - see section 4.) The ModelRepo landed at 4.9 s, the 399 `File` listings and most settings at 5.1 s, and the seed `RecallPreset` at 10.1 s - 474 messages of 24 distinct types by 15 s. So a listener attached to the client `connect()` returns is about 3 s too late for the ModelRepo and 8 s too late diff --git a/tests/hardware/test_model_state.py b/tests/hardware/test_model_state.py index 780a702..4b5c8a8 100644 --- a/tests/hardware/test_model_state.py +++ b/tests/hardware/test_model_state.py @@ -174,30 +174,36 @@ def test_nothing_the_burst_delivered_is_read_again_on_first_access( "again, which is the round trip the cache exists to avoid") -def test_the_burst_does_not_warm_what_the_unit_never_announces(burst_warmed, - handshake_burst): - """The other half, and the reason the read path exists. - - The unit does send a ``Version`` during the handshake, but it is the answer - to our version announce - it sets ``cortex_control_version_valid`` and none - of the unit's own fields. So identity is exactly the case section 9's third - column is for: where the unit does not tell us, we ask. - - The count is asserted because the empty cache below is not evidence for that - story: it was equally true of the story this replaced, which had the unit - volunteering a ``Version`` READ of its own during connect. ``_hello`` sends - no host ``Version`` READ, and the unit only asks when asked, so one is the - number - and a second would mean the handshake has changed under us. +def test_the_burst_warms_identity_from_connects_own_version_read(burst_warmed, + handshake_burst): + """Identity reaches the cache through the ONE Version READ connect() makes. + + The unit never volunteers its identity, so the model reads it (section 9's + third column). Since ADR-0020, ``connect()`` issues that read itself, before + the handshake, to resolve the profile - and the state layer listens from + before the handshake, so it sees the reply. Measured 2026-09-06 on CorOS + 4.0.1 / d14e: exactly THREE inbound ``Version`` messages through connect and + its burst - the full reply to connect's READ (15 fields, +0.71 s), the + unit's own ``Version{READ}`` 1 ms behind it (the question it asks Cortex + Control, section 4.4), and the ``UPDATE`` carrying + ``cortex_control_version_valid`` that answers our announce (+0.73 s). + ``_hello`` itself still sends no READ; the 2026-08-27 measurement of one + inbound ``Version`` stands for ``_hello`` alone. + + The count is asserted so a fourth (the handshake changed under us) or a + second (connect stopped reading identity) is loud. The cache holds the two + fields the entry keeps; the reply also carries fields it does not, so the + entry stays marked and the first read of ``device.firmware`` still goes to + the unit (see ``identity`` in device/entries.py). """ versions = handshake_burst.names().count("VersionMessage") - assert versions == 1, ( - f"the connect burst carried {versions} Version message(s), not the one " - f"answering our version announce. Section 4 of docs/protocol.md says " - f"the unit asks for our version only when we ask for its.") - assert burst_warmed["identity"] == {}, ( - f"the burst carried the unit's own identity after all, which is worth " - f"knowing - it held {burst_warmed['identity']}. If that is now true, " - f"the entry's docstring in device/entries.py is wrong.") + assert versions == 3, ( + f"the connect burst carried {versions} Version message(s); 3 is measured " + f"(connect's READ reply, the unit's own READ, the announce answer). See " + f"docs/protocol.md section 4.") + assert set(burst_warmed["identity"]) == {"device_serial_number", "app_fw_version"}, ( + f"connect's Version read should have left exactly the two kept identity " + f"fields in the cache; it held {burst_warmed['identity']}") def test_a_version_read_is_answered_and_then_questioned(qc, record_property): diff --git a/tests/hardware/test_preset_surface.py b/tests/hardware/test_preset_surface.py index aa83662..80fc71a 100644 --- a/tests/hardware/test_preset_surface.py +++ b/tests/hardware/test_preset_surface.py @@ -196,15 +196,17 @@ def test_an_inactive_scene_refuses_writes_and_reads_fine(device): def test_the_burst_delivered_every_entry_the_preset_surface_reads(burst_warmed): """Measured 2026-08-15: the burst carries RecallPreset, SetlistPosition, - PresetDirty and Scene at about 10 s, inside ten milliseconds. Identity is - NOT in it - the unit never announces its own firmware - so that one is - expected to be empty and is asserted, to keep this from passing on a run - where the burst delivered nothing at all.""" + PresetDirty and Scene at about 10 s, inside ten milliseconds. The unit never + announces its own firmware; identity is in the cache anyway since ADR-0020, + because ``connect()`` reads ``Version`` before the handshake and the state + layer is listening by then (measured 2026-09-06 on 4.0.1 / d14e). Asserted + with its two kept fields, to keep this from passing on a run where the burst + delivered nothing at all.""" for name in ("preset", "scene", "dirty", "loaded"): assert burst_warmed[name], f"the burst delivered nothing for {name}" - assert not burst_warmed["identity"], ( - "identity arrived in the burst, which contradicts what the entry's " - "docstring says the unit does") + assert set(burst_warmed["identity"]) == {"device_serial_number", "app_fw_version"}, ( + f"connect's own Version read should have warmed identity with its two " + f"kept fields; the cache held {burst_warmed['identity']}") def test_reading_the_preset_surface_costs_no_round_trip(device, model_cache): From f52770fa0537112755003e54422344504ac9fc7c Mon Sep 17 00:00:00 2001 From: Jonathan Stokes Date: Mon, 7 Sep 2026 18:22:47 -0500 Subject: [PATCH 2/7] fix: the hardware report's regression line counts failures, not skips On the same run the bypass echo test skipped (the loaded preset had no stored bypass entry) and the report printed set_bypass under "regressions by name". A skipped test measured nothing about the unit; it belongs on the "failed or skipped" line and nowhere else. The last line now lists only operations some test FAILED, labelled "VERIFIED and claimed by a test, failed", and an offline test pins a skipped-only operation to the first line alone. --- changelog.md | 17 +++++++++++++++++ tests/hardware/conftest.py | 19 +++++++++++++------ tests/test_hardware_report.py | 23 ++++++++++++++++++----- 3 files changed, 48 insertions(+), 11 deletions(-) diff --git a/changelog.md b/changelog.md index 9ea98c0..ac1e7a7 100644 --- a/changelog.md +++ b/changelog.md @@ -20,6 +20,23 @@ correction. ## Unreleased +### The first hardware run of the profile seam, and what it corrected + +Run 2026-09-06 on Quad Cortex, CorOS 4.0.1 / d14e, straight after the seam +merged: 93 passed, 4 skipped, and two tests failed on records the seam itself +made stale. `connect()` now reads `Version` before the handshake, so a listener +registered through `before_handshake` sees three inbound `Version` messages, not +one: the full reply to that read, the unit's own `Version{READ}` a millisecond +behind it, and the answer to our announce. The same reply leaves the identity +entry holding serial and firmware ahead of the burst. Both tests and +`docs/protocol.md` section 4 now say so, with the measurement. + +The run also showed the report calling a SKIPPED test a regression: the bypass +echo test declined to run (no stored bypass entry on the loaded preset) and +`set_bypass` landed on the "regressions by name" line. A skip measured nothing. +That line now counts failures only and is labelled "VERIFIED and claimed by a +test, failed"; skipped operations stay on the "failed or skipped" line. + ### BREAKING: the generators take `--snapshot`, and constants moved `scripts/generate_models.py`, `generate_params.py` and `generate_options.py` diff --git a/tests/hardware/conftest.py b/tests/hardware/conftest.py index cd6c8e7..61532af 100644 --- a/tests/hardware/conftest.py +++ b/tests/hardware/conftest.py @@ -566,23 +566,30 @@ def _report_lines(cls, outcomes, claimed): verifies, and it is what keeps the last line readable: ``QuadCortex`` verifies EVERYTHING, so the plain difference against ``VERIFIED`` names all ~89 operations no test has ever driven, each tagged as a regression. Only - something a test claims can regress - nothing else was measured. + something a test claims can regress - nothing else was measured - and only + a FAILED test regresses it; a skipped one measured nothing. """ from pyquadcortex.protocol.support import EVERYTHING passed = {op for op, seen in outcomes.items() if seen and all(o == "passed" for o in seen)} - failed = {op for op, seen in outcomes.items() - if any(o != "passed" for o in seen)} + not_passed = {op for op, seen in outcomes.items() + if any(o != "passed" for o in seen)} + # A SKIP is a test that declined to measure - a precondition the loaded + # preset did not meet, an operator-only capture - and says nothing about + # the unit. Only a FAILURE is a regression. Measured on the first post-merge + # run (2026-09-06): the bypass echo test skipped for want of a stored bypass + # entry and the old line called set_bypass a regression. + failed = {op for op, seen in outcomes.items() if "failed" in seen} verified = (set(cls.operations()) if cls.VERIFIED is EVERYTHING else set(cls.VERIFIED)) return [ ("passed", sorted(passed), ""), - ("failed or skipped", sorted(failed), ""), + ("failed or skipped", sorted(not_passed), ""), ("passed, not VERIFIED", sorted(passed - verified), "<- candidates to add"), - ("VERIFIED and claimed by a test, not passed", - sorted((verified & claimed) - passed), "<- regressions by name"), + ("VERIFIED and claimed by a test, failed", + sorted(verified & claimed & failed), "<- regressions by name"), ] diff --git a/tests/test_hardware_report.py b/tests/test_hardware_report.py index 69f89b1..bb5e4dd 100644 --- a/tests/test_hardware_report.py +++ b/tests/test_hardware_report.py @@ -117,8 +117,8 @@ def test_the_report_names_the_four_things_a_maintainer_asks_for(conftest): assert lines["failed or skipped"] == ["set_bypass"] # set_ir passed and the profile does not claim it: a candidate. assert lines["passed, not VERIFIED"] == ["set_ir"] - # set_bypass is claimed by the profile, a test names it, and it did not pass. - assert lines["VERIFIED and claimed by a test, not passed"] == ["set_bypass"] + # set_bypass is claimed by the profile, a test names it, and it FAILED. + assert lines["VERIFIED and claimed by a test, failed"] == ["set_bypass"] def test_one_failing_run_of_an_operation_sinks_all_of_them(conftest): @@ -145,17 +145,30 @@ def test_a_regression_is_only_an_operation_some_test_claims(conftest): # set_ir and set_block are VERIFIED here too - EVERYTHING says so - and no # test names either, so neither is a regression. Only set_param is. - assert lines["VERIFIED and claimed by a test, not passed"] == ["set_param"] + assert lines["VERIFIED and claimed by a test, failed"] == ["set_param"] assert lines["passed, not VERIFIED"] == [], ( "a profile that verifies everything can have no candidates") +def test_a_skipped_test_is_not_a_regression(conftest): + """Measured on the first post-merge run, 2026-09-06: the bypass echo test + skipped ("no stored bypass entry") and the report called set_bypass a + regression. A skip measured nothing; it belongs on the failed-or-skipped + line and nowhere else.""" + lines = dict(_named(conftest._report_lines( + _Everything, {"set_bypass": ["skipped"], "set_param": ["failed"]}, + claimed={"set_bypass", "set_param"}))) + + assert lines["failed or skipped"] == ["set_bypass", "set_param"] + assert lines["VERIFIED and claimed by a test, failed"] == ["set_param"] + + def test_an_operation_no_test_claims_is_never_a_regression(conftest): """The 89. Nothing names them, so the run says nothing about them.""" lines = dict(_named(conftest._report_lines( _Everything, {"set_param": ["passed"]}, claimed={"set_param"}))) - assert lines["VERIFIED and claimed by a test, not passed"] == [] + assert lines["VERIFIED and claimed by a test, failed"] == [] def test_a_profile_with_nothing_claimed_reports_nothing_as_a_regression(conftest): @@ -234,7 +247,7 @@ def test_a_deselected_test_claims_nothing(conftest, collection): claimed = set().union(*collection.values()) lines = dict(_named(conftest._report_lines( _Everything, {"set_param": ["passed"]}, claimed))) - assert lines["VERIFIED and claimed by a test, not passed"] == [], ( + assert lines["VERIFIED and claimed by a test, failed"] == [], ( "set_bypass was deselected, so this run measured nothing about it") From 81872fee52313fbc3dd98bf284675b16c5a4a520 Mon Sep 17 00:00:00 2001 From: Jonathan Stokes Date: Mon, 7 Sep 2026 18:29:13 -0500 Subject: [PATCH 3/7] docs: the run is dated 2026-09-07, the listener test's twin passage says three Version messages, and the count's failure names its causes --- changelog.md | 2 +- docs/protocol.md | 6 +++--- tests/hardware/conftest.py | 4 +++- tests/hardware/test_broadcast_listener.py | 13 +++++++------ tests/hardware/test_model_state.py | 12 ++++++++---- tests/hardware/test_preset_surface.py | 2 +- tests/test_hardware_report.py | 2 +- 7 files changed, 24 insertions(+), 17 deletions(-) diff --git a/changelog.md b/changelog.md index ac1e7a7..d8b2b27 100644 --- a/changelog.md +++ b/changelog.md @@ -22,7 +22,7 @@ correction. ### The first hardware run of the profile seam, and what it corrected -Run 2026-09-06 on Quad Cortex, CorOS 4.0.1 / d14e, straight after the seam +Run 2026-09-07 on Quad Cortex, CorOS 4.0.1 / d14e, straight after the seam merged: 93 passed, 4 skipped, and two tests failed on records the seam itself made stale. `connect()` now reads `Version` before the handshake, so a listener registered through `before_handshake` sees three inbound `Version` messages, not diff --git a/docs/protocol.md b/docs/protocol.md index 853688d..7a0be32 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -530,7 +530,7 @@ eight seconds of idling after it produce none. So step 3 above is a consequence of step 2 rather than something the device does on connecting. Since ADR-0020, `connect()` itself makes one `Version` READ BEFORE calling -`_hello()`, to resolve the profile. Measured 2026-09-06 on CorOS 4.0.1 / d14e: a +`_hello()`, to resolve the profile. Measured 2026-09-07 on CorOS 4.0.1 / d14e: a `connect()` and its burst now carry exactly THREE inbound `Version` messages - the full reply to that READ (15 fields, at +0.71 s), the unit's own `Version{READ}` 1 ms behind it (section 4.4), and the announce answer above (at @@ -2040,8 +2040,8 @@ in, having seen only the `ResetCommsBuffers` echo and one `Version`. (That sessi answering our version announce, carrying `cortex_control_version_valid`; `_hello()` sends no host `Version` READ, and with none sent the unit asks nothing back. Since ADR-0020 `connect()` reads `Version` once before `_hello()`, so a listener registered before the handshake now sees -three `Version` messages - see section 4.) The -ModelRepo landed at 4.9 s, the 399 `File` listings and most settings at 5.1 s, and the seed +three `Version` messages - see section 4.) The ModelRepo landed at 4.9 s, the 399 `File` +listings and most settings at 5.1 s, and the seed `RecallPreset` at 10.1 s - 474 messages of 24 distinct types by 15 s. So a listener attached to the client `connect()` returns is about 3 s too late for the ModelRepo and 8 s too late for the current preset, which is why `connect(before_handshake=...)` exists. diff --git a/tests/hardware/conftest.py b/tests/hardware/conftest.py index 61532af..8f1b7a3 100644 --- a/tests/hardware/conftest.py +++ b/tests/hardware/conftest.py @@ -578,8 +578,10 @@ def _report_lines(cls, outcomes, claimed): # A SKIP is a test that declined to measure - a precondition the loaded # preset did not meet, an operator-only capture - and says nothing about # the unit. Only a FAILURE is a regression. Measured on the first post-merge - # run (2026-09-06): the bypass echo test skipped for want of a stored bypass + # run (2026-09-07): the bypass echo test skipped for want of a stored bypass # entry and the old line called set_bypass a regression. + # pytest reports a setup or teardown ERROR with outcome "failed" too, so an + # errored test lands here as well (checked against pytest 9.1.1). failed = {op for op, seen in outcomes.items() if "failed" in seen} verified = (set(cls.operations()) if cls.VERIFIED is EVERYTHING else set(cls.VERIFIED)) diff --git a/tests/hardware/test_broadcast_listener.py b/tests/hardware/test_broadcast_listener.py index 59acb82..c6ec342 100644 --- a/tests/hardware/test_broadcast_listener.py +++ b/tests/hardware/test_broadcast_listener.py @@ -59,12 +59,13 @@ def test_a_listener_registered_before_connecting_sees_the_handshake_burst( """The burst is what makes a push-fed cache warm for free. Note what the wait below says about the hook: ``connect()`` returns about 2 s - in, with only the ResetCommsBuffers echo and one Version recorded - the - unit's UPDATE answering our version announce, not a READ of its own - (``docs/protocol.md`` section 4) - and the state burst starts arriving about - 5 s in. So a listener - registered on the client ``connect()`` hands back is already too late, which - is the whole reason ``before_handshake`` exists. + in, with only the ResetCommsBuffers echo and three ``Version`` messages + recorded - the full reply to connect's own identity READ, the unit's + ``Version{READ}`` a millisecond behind it, and the UPDATE answering our + version announce (``docs/protocol.md`` section 4; measured 2026-09-07 on + 4.0.1 / d14e) - and the state burst starts arriving about 5 s in. So a + listener registered on the client ``connect()`` hands back is already too + late, which is the whole reason ``before_handshake`` exists. Confirmed on this unit (2026-08-12, CorOS 4.0.1 / d14e): 474 messages of 24 distinct types by 15 s. The floors below sit well under that, because a test diff --git a/tests/hardware/test_model_state.py b/tests/hardware/test_model_state.py index 4b5c8a8..4522bef 100644 --- a/tests/hardware/test_model_state.py +++ b/tests/hardware/test_model_state.py @@ -181,7 +181,7 @@ def test_the_burst_warms_identity_from_connects_own_version_read(burst_warmed, The unit never volunteers its identity, so the model reads it (section 9's third column). Since ADR-0020, ``connect()`` issues that read itself, before the handshake, to resolve the profile - and the state layer listens from - before the handshake, so it sees the reply. Measured 2026-09-06 on CorOS + before the handshake, so it sees the reply. Measured 2026-09-07 on CorOS 4.0.1 / d14e: exactly THREE inbound ``Version`` messages through connect and its burst - the full reply to connect's READ (15 fields, +0.71 s), the unit's own ``Version{READ}`` 1 ms behind it (the question it asks Cortex @@ -191,7 +191,9 @@ def test_the_burst_warms_identity_from_connects_own_version_read(burst_warmed, inbound ``Version`` stands for ``_hello`` alone. The count is asserted so a fourth (the handshake changed under us) or a - second (connect stopped reading identity) is loud. The cache holds the two + fall to one (connect stopped reading identity, and with no READ the unit + asks nothing back) is loud. A retried identity read on a slow-to-boot unit + would also add to it; the message says so. The cache holds the two fields the entry keeps; the reply also carries fields it does not, so the entry stays marked and the first read of ``device.firmware`` still goes to the unit (see ``identity`` in device/entries.py). @@ -199,8 +201,10 @@ def test_the_burst_warms_identity_from_connects_own_version_read(burst_warmed, versions = handshake_burst.names().count("VersionMessage") assert versions == 3, ( f"the connect burst carried {versions} Version message(s); 3 is measured " - f"(connect's READ reply, the unit's own READ, the announce answer). See " - f"docs/protocol.md section 4.") + f"(connect's READ reply, the unit's own READ, the announce answer). More " + f"means the handshake changed under us, or connect's identity read " + f"retried on a unit that was still booting. See docs/protocol.md " + f"section 4.") assert set(burst_warmed["identity"]) == {"device_serial_number", "app_fw_version"}, ( f"connect's Version read should have left exactly the two kept identity " f"fields in the cache; it held {burst_warmed['identity']}") diff --git a/tests/hardware/test_preset_surface.py b/tests/hardware/test_preset_surface.py index 80fc71a..a3b5d76 100644 --- a/tests/hardware/test_preset_surface.py +++ b/tests/hardware/test_preset_surface.py @@ -199,7 +199,7 @@ def test_the_burst_delivered_every_entry_the_preset_surface_reads(burst_warmed): PresetDirty and Scene at about 10 s, inside ten milliseconds. The unit never announces its own firmware; identity is in the cache anyway since ADR-0020, because ``connect()`` reads ``Version`` before the handshake and the state - layer is listening by then (measured 2026-09-06 on 4.0.1 / d14e). Asserted + layer is listening by then (measured 2026-09-07 on 4.0.1 / d14e). Asserted with its two kept fields, to keep this from passing on a run where the burst delivered nothing at all.""" for name in ("preset", "scene", "dirty", "loaded"): diff --git a/tests/test_hardware_report.py b/tests/test_hardware_report.py index bb5e4dd..613c753 100644 --- a/tests/test_hardware_report.py +++ b/tests/test_hardware_report.py @@ -151,7 +151,7 @@ def test_a_regression_is_only_an_operation_some_test_claims(conftest): def test_a_skipped_test_is_not_a_regression(conftest): - """Measured on the first post-merge run, 2026-09-06: the bypass echo test + """Measured on the first post-merge run, 2026-09-07: the bypass echo test skipped ("no stored bypass entry") and the report called set_bypass a regression. A skip measured nothing; it belongs on the failed-or-skipped line and nowhere else.""" From e2cdde5e61adf58334a26bef92ee21f23e5b2027 Mon Sep 17 00:00:00 2001 From: Jonathan Stokes Date: Mon, 7 Sep 2026 19:05:11 -0500 Subject: [PATCH 4/7] fix: the burst test asserts the shape of each Version, and the report says when it measured nothing Triage of PR #53 found four things worth fixing before merge. The burst test counted Version messages, so a unit that was still booting - which makes connect() retry its identity READ and adds a full reply plus the unit's own READ per retry - would have failed a healthy run as a regression. HandshakeBurst now records each Version's (action, fields) and the test asserts composition: one announce answer, at least one full reply, one unit READ per full reply, nothing else; the number of identity reads is recorded, not asserted. The report printed an empty regression line on an all-skip run, identical to a healthy run. It now prints test counts and how many claimed operations were measured, and flags a run where nothing passed. A session that never connected gets a line saying so instead of silence; the first run of that line was this commit's own hardware check, made with the unit unplugged. Also: the stale "not passed" label in two docstrings; the untested ["passed", "skipped"] cell; architecture.md's "two entries" (five); "about 1 ms" where the older record says 0.5-0.8 ms; and the identity assertions say a reply that stopped carrying serial or firmware is a protocol.md finding, not a reason to loosen them. --- docs/architecture.md | 6 ++-- docs/protocol.md | 2 +- tests/hardware/conftest.py | 48 ++++++++++++++++++++++++--- tests/hardware/test_model_state.py | 47 ++++++++++++++++++-------- tests/hardware/test_preset_surface.py | 3 +- tests/test_hardware_report.py | 30 +++++++++++++++-- 6 files changed, 111 insertions(+), 25 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 1fa5a12..642cfbc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -269,9 +269,9 @@ returns immediately, which is what makes the metronome's tempo stream - a pair per beat, on every connection, forever - cost nothing. `entries.py` is the table of what is tracked: per entry, the message types that -carry it, the fields kept from each, and the read that fetches it. Two entries -today, `identity` and `dirty`; the rest of section 9's table arrives with the -surfaces that read it. +carry it, the fields kept from each, and the read that fetches it. Five entries +today - `identity`, `dirty`, `preset`, `scene` and `loaded`; the rest of section 9's +table arrives with the surfaces that read it. `watch.py` is the write side. A write updates the cache immediately and the unit's echo confirms it in the background, against one sentence: every field we diff --git a/docs/protocol.md b/docs/protocol.md index 7a0be32..cc166d7 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -533,7 +533,7 @@ Since ADR-0020, `connect()` itself makes one `Version` READ BEFORE calling `_hello()`, to resolve the profile. Measured 2026-09-07 on CorOS 4.0.1 / d14e: a `connect()` and its burst now carry exactly THREE inbound `Version` messages - the full reply to that READ (15 fields, at +0.71 s), the unit's own -`Version{READ}` 1 ms behind it (section 4.4), and the announce answer above (at +`Version{READ}` about 1 ms behind it (section 4.4), and the announce answer above (at +0.73 s). The state layer listens from before the handshake, so the identity entry holds `device_serial_number` and `app_fw_version` from that reply; the reply also carries fields the entry does not keep, so the entry stays marked and diff --git a/tests/hardware/conftest.py b/tests/hardware/conftest.py index 8f1b7a3..d255831 100644 --- a/tests/hardware/conftest.py +++ b/tests/hardware/conftest.py @@ -21,6 +21,8 @@ import pytest +from pyquadcortex.protocol.proto import ProductionAutomation_pb2 as pa + #: This directory. Everything under it drives the unit and is gated on the flag. SUITE = pathlib.Path(__file__).resolve().parent ROOT = SUITE.parent.parent @@ -76,7 +78,7 @@ def _claims(items, operations, wanted, deselect): left. The order matters and is the whole point: ``claimed`` in the end-of-run report is the union of this, and a deselected test measured nothing - recording it printed every operation the run never touched under - ``VERIFIED and claimed by a test, not passed``, which reads as a + ``VERIFIED and claimed by a test, failed``, which reads as a regression. Held offline by ``tests/test_hardware_report.py``. """ marks = {} @@ -209,6 +211,11 @@ class HandshakeBurst: def __init__(self): self._lock = threading.Lock() self._names = [] + #: ``(action, frozenset of field names)`` for every ``Version`` seen, + #: because since ADR-0020 a connect carries three of them and a test + #: that only counts cannot tell a retried identity read from a changed + #: handshake. Shapes only - the values are not this recorder's business. + self._versions = [] self._detach = None self.closed = False self.settled_in = None # seconds the burst took, or None if it timed out @@ -224,6 +231,14 @@ def __call__(self, message): # arrive after removal. It must not reopen the recording. return self._names.append(type(message).__name__) + if isinstance(message, pa.VersionMessage): + self._versions.append( + (message.action, frozenset(f.name for f, _ in message.ListFields()))) + + def versions(self): + """The shape of every ``Version`` recorded: ``(action, field names)``.""" + with self._lock: + return list(self._versions) def record_until(self, sentinel, patience): """Record until a ``sentinel``-typed message arrives, then stop. @@ -595,6 +610,18 @@ def _report_lines(cls, outcomes, claimed): ] +def _measured_nothing(outcomes, claimed): + """True when tests claimed operations and not one of them passed. + + The regression line is empty on a healthy run AND on a run where every + test skipped its precondition, so an empty line alone cannot be read as + clean. Pure, held offline like :func:`_report_lines`. + """ + passed = {op for op, seen in outcomes.items() + if seen and all(o == "passed" for o in seen)} + return bool(claimed) and not passed + + def pytest_terminal_summary(terminalreporter, exitstatus, config): """Print which operations this run measured on the connected profile. @@ -604,14 +631,27 @@ def pytest_terminal_summary(terminalreporter, exitstatus, config): """ if not config.getoption("--hardware"): return + tr = terminalreporter cls = getattr(config, "_profile", None) - if cls is None: # the session never connected, so there is nothing to report + if cls is None: + # The session never connected. pytest's own summary carries the setup + # errors; this line exists so the ABSENCE of the operations report is + # never mistaken for "nothing to report". + tr.section("operations: no report") + tr.line("the hardware session never connected, so no operation was measured; " + "see the setup errors above") return claimed = set().union(*_VERIFIES.values()) if _VERIFIES else set() lines = _report_lines(cls, _OUTCOMES, claimed) width = max(len(label) for label, _names, _note in lines) - tr = terminalreporter + counts = {k: len(tr.stats.get(k, ())) for k in ("passed", "failed", "skipped")} tr.section(f"operations on {cls.__name__} " f"(CorOS {', '.join(cls.MEASURED_ON)}, {cls.EVIDENCE.name})") + tr.line(f"tests: {counts['passed']} passed, {counts['failed']} failed, " + f"{counts['skipped']} skipped; operations measured: " + f"{len(_OUTCOMES)} of {len(claimed)} claimed") for label, names, note in lines: - tr.line(f"{label + ':':<{width + 1}} {names} {note}".rstrip()) + tr.line(f"{label + ':':<{width + 1}} {names} ({len(names)}) {note}".rstrip()) + if _measured_nothing(_OUTCOMES, claimed): + tr.line("NOTHING PASSED: every claimed operation failed or skipped, so the " + "empty regression line above says nothing about the unit") diff --git a/tests/hardware/test_model_state.py b/tests/hardware/test_model_state.py index 4522bef..db4698e 100644 --- a/tests/hardware/test_model_state.py +++ b/tests/hardware/test_model_state.py @@ -175,7 +175,8 @@ def test_nothing_the_burst_delivered_is_read_again_on_first_access( def test_the_burst_warms_identity_from_connects_own_version_read(burst_warmed, - handshake_burst): + handshake_burst, + record_property): """Identity reaches the cache through the ONE Version READ connect() makes. The unit never volunteers its identity, so the model reads it (section 9's @@ -184,30 +185,48 @@ def test_the_burst_warms_identity_from_connects_own_version_read(burst_warmed, before the handshake, so it sees the reply. Measured 2026-09-07 on CorOS 4.0.1 / d14e: exactly THREE inbound ``Version`` messages through connect and its burst - the full reply to connect's READ (15 fields, +0.71 s), the - unit's own ``Version{READ}`` 1 ms behind it (the question it asks Cortex + unit's own ``Version{READ}`` about 1 ms behind it (the question it asks Cortex Control, section 4.4), and the ``UPDATE`` carrying ``cortex_control_version_valid`` that answers our announce (+0.73 s). ``_hello`` itself still sends no READ; the 2026-08-27 measurement of one inbound ``Version`` stands for ``_hello`` alone. - The count is asserted so a fourth (the handshake changed under us) or a - fall to one (connect stopped reading identity, and with no READ the unit - asks nothing back) is loud. A retried identity read on a slow-to-boot unit - would also add to it; the message says so. The cache holds the two + Asserted by SHAPE rather than by count, because a unit that was still + booting makes ``connect()`` retry its identity READ, and each retry adds a + full reply plus the unit's own READ - a healthy run that a bare ``== 3`` + would call a regression. So: exactly one announce answer, at least one full + reply, and one unit READ per full reply, and nothing else. A message of any + other shape (the handshake changed under us) or no full reply at all + (connect stopped reading identity, and with no READ the unit asks nothing + back) is loud. The number of identity reads is recorded, not asserted. The cache holds the two fields the entry keeps; the reply also carries fields it does not, so the entry stays marked and the first read of ``device.firmware`` still goes to the unit (see ``identity`` in device/entries.py). """ - versions = handshake_burst.names().count("VersionMessage") - assert versions == 3, ( - f"the connect burst carried {versions} Version message(s); 3 is measured " - f"(connect's READ reply, the unit's own READ, the announce answer). More " - f"means the handshake changed under us, or connect's identity read " - f"retried on a unit that was still booting. See docs/protocol.md " - f"section 4.") + shapes = handshake_burst.versions() + full = [s for s in shapes if "device_serial_number" in s[1]] + own_reads = [s for s in shapes + if s[0] == pa.MessageAction.READ and s[1] == {"action"}] + announce = [s for s in shapes if "cortex_control_version_valid" in s[1]] + record_property("identity_reads", len(full)) + assert len(announce) == 1, ( + f"{len(announce)} answers to our version announce; one is measured. " + f"Shapes seen: {shapes}") + assert full, ( + f"no full Version reply reached the listener: connect() stopped reading " + f"identity before the handshake, or the unit stopped answering. Shapes " + f"seen: {shapes}") + assert len(own_reads) == len(full), ( + f"{len(full)} full replies but {len(own_reads)} unit READs; the unit asks " + f"its own question once per answer (protocol.md 4.4). Shapes: {shapes}") + assert len(shapes) == len(full) + len(own_reads) + 1, ( + f"a Version of a shape this test does not know arrived - the handshake " + f"changed under us. Shapes: {shapes}") assert set(burst_warmed["identity"]) == {"device_serial_number", "app_fw_version"}, ( f"connect's Version read should have left exactly the two kept identity " - f"fields in the cache; it held {burst_warmed['identity']}") + f"fields in the cache; it held {burst_warmed['identity']}. The unit's " + f"full reply carries both (protocol.md, version read row); if it stopped, " + f"that is a finding for protocol.md, not a reason to loosen this.") def test_a_version_read_is_answered_and_then_questioned(qc, record_property): diff --git a/tests/hardware/test_preset_surface.py b/tests/hardware/test_preset_surface.py index a3b5d76..71305db 100644 --- a/tests/hardware/test_preset_surface.py +++ b/tests/hardware/test_preset_surface.py @@ -206,7 +206,8 @@ def test_the_burst_delivered_every_entry_the_preset_surface_reads(burst_warmed): assert burst_warmed[name], f"the burst delivered nothing for {name}" assert set(burst_warmed["identity"]) == {"device_serial_number", "app_fw_version"}, ( f"connect's own Version read should have warmed identity with its two " - f"kept fields; the cache held {burst_warmed['identity']}") + f"kept fields; the cache held {burst_warmed['identity']}. If the unit's " + f"reply stopped carrying one, that is a finding for protocol.md.") def test_reading_the_preset_surface_costs_no_round_trip(device, model_cache): diff --git a/tests/test_hardware_report.py b/tests/test_hardware_report.py index 613c753..924ca93 100644 --- a/tests/test_hardware_report.py +++ b/tests/test_hardware_report.py @@ -163,6 +163,32 @@ def test_a_skipped_test_is_not_a_regression(conftest): assert lines["VERIFIED and claimed by a test, failed"] == ["set_param"] +def test_a_pass_and_a_skip_on_one_operation_is_neither_passed_nor_a_regression(conftest): + """Two tests name an operation; one skipped its precondition, one passed. + Not all of it passed, so it is not offered as VERIFIED; nothing failed, so + it is not a regression. It sits on the middle line, where a reader sees the + skip.""" + lines = dict(_named(conftest._report_lines( + _Everything, {"set_param": ["passed", "skipped"]}, claimed={"set_param"}))) + + assert lines["passed"] == [] + assert lines["failed or skipped"] == ["set_param"] + assert lines["VERIFIED and claimed by a test, failed"] == [] + + +def test_a_run_where_nothing_passed_is_flagged_not_read_as_clean(conftest): + """An all-skip run has an empty regression line, same as a healthy run. + The flag is what tells them apart.""" + assert conftest._measured_nothing( + {"set_param": ["skipped"], "set_bypass": ["skipped"]}, + claimed={"set_param", "set_bypass"}) is True + assert conftest._measured_nothing( + {"set_param": ["passed"], "set_bypass": ["skipped"]}, + claimed={"set_param", "set_bypass"}) is False + assert conftest._measured_nothing({}, claimed=set()) is False, ( + "nothing claimed is a deselected run, not a run that measured nothing") + + def test_an_operation_no_test_claims_is_never_a_regression(conftest): """The 89. Nothing names them, so the run says nothing about them.""" lines = dict(_named(conftest._report_lines( @@ -232,8 +258,8 @@ def test_a_deselected_test_claims_nothing(conftest, collection): The recording used to happen before the deselection, so `claimed` was built from every COLLECTED test - and an operation whose test never ran - was printed under `VERIFIED and claimed by a test, not passed`, which the - report labels a regression. A run of one test reported ~104 of them. + was printed under the line the report labels a regression (then + `not passed`, now `failed`). A run of one test reported ~104 of them. """ items = [_Item("t.py::a", "set_param"), _Item("t.py::b", "set_bypass")] config = _Config(verifies="set_param") From 40ea30f5e864a64b01c7296cba537d764ce49780 Mon Sep 17 00:00:00 2001 From: Jonathan Stokes Date: Mon, 7 Sep 2026 19:12:54 -0500 Subject: [PATCH 5/7] fix: the report's test count names setup and teardown errors, which pytest files apart from failures --- tests/hardware/conftest.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/tests/hardware/conftest.py b/tests/hardware/conftest.py index d255831..6039704 100644 --- a/tests/hardware/conftest.py +++ b/tests/hardware/conftest.py @@ -644,12 +644,18 @@ def pytest_terminal_summary(terminalreporter, exitstatus, config): claimed = set().union(*_VERIFIES.values()) if _VERIFIES else set() lines = _report_lines(cls, _OUTCOMES, claimed) width = max(len(label) for label, _names, _note in lines) - counts = {k: len(tr.stats.get(k, ())) for k in ("passed", "failed", "skipped")} + # pytest files a setup or teardown failure under "error", not "failed" + # (_pytest/runner.py pytest_report_teststatus), and a restore that could + # not finish is exactly that shape - so the count names errors separately + # rather than losing them. The operations lines are unaffected: they read + # report.outcome, which is "failed" for an error too. + counts = {k: len(tr.stats.get(k, ())) + for k in ("passed", "failed", "error", "skipped")} tr.section(f"operations on {cls.__name__} " f"(CorOS {', '.join(cls.MEASURED_ON)}, {cls.EVIDENCE.name})") tr.line(f"tests: {counts['passed']} passed, {counts['failed']} failed, " - f"{counts['skipped']} skipped; operations measured: " - f"{len(_OUTCOMES)} of {len(claimed)} claimed") + f"{counts['error']} errored, {counts['skipped']} skipped; " + f"operations measured: {len(_OUTCOMES)} of {len(claimed)} claimed") for label, names, note in lines: tr.line(f"{label + ':':<{width + 1}} {names} ({len(names)}) {note}".rstrip()) if _measured_nothing(_OUTCOMES, claimed): From 08590eb69962f882ad3f58304fb5e10739ed349d Mon Sep 17 00:00:00 2001 From: Jonathan Stokes Date: Mon, 7 Sep 2026 20:05:20 -0500 Subject: [PATCH 6/7] fix: the report's claims come from the tests that ran, not the tests collected Seen on the unit 2026-09-07 while running the three burst tests with -k: the report said "operations measured: 0 of 16 claimed" and flagged NOTHING PASSED, although the run had deselected every marked test on purpose. pytest's own -k deselection runs AFTER this conftest's collection hook, so recording claims there counted tests that never ran. Claims are now built at report time from the node ids that produced a report (_claimed, pure, held offline); the same narrowed run reports "0 of 0 claimed" with no flag, and a --verifies set_scene_label run reports 1 of 1 with the operation under passed. Verified on Quad Cortex CorOS 4.0.1 / d14e: the shape-based burst assertion passes against real traffic (one announce answer, one full reply, one unit READ), and the three corrected burst tests pass. --- tests/hardware/conftest.py | 22 +++++++++++++++++++++- tests/test_hardware_report.py | 11 +++++++++++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/tests/hardware/conftest.py b/tests/hardware/conftest.py index 6039704..973c869 100644 --- a/tests/hardware/conftest.py +++ b/tests/hardware/conftest.py @@ -36,6 +36,10 @@ #: ``operation -> list of outcomes``, one entry per phase that decided a test. _OUTCOMES = {} +#: Node ids that produced at least one report - the tests that actually RAN, +#: whatever deselected the rest (our --verifies, or pytest's own -k, which runs +#: after this conftest's hook). Claims are read from these at report time. +_RAN = set() def pytest_ignore_collect(collection_path, config): @@ -565,6 +569,7 @@ def _decides(when, outcome): def pytest_runtest_logreport(report): """Record how each operation's tests came out, for the end-of-run report.""" + _RAN.add(report.nodeid) names = _VERIFIES.get(report.nodeid) if not names: return @@ -610,6 +615,21 @@ def _report_lines(cls, outcomes, claimed): ] +def _claimed(verifies, ran): + """The operations claimed by tests that actually ran. + + ``verifies`` maps node id to the names its marker carries; ``ran`` is the + node ids that produced a report. Built at report time rather than at + collection because pytest's own ``-k`` deselection runs AFTER this + conftest's hook, so a `-k` run that kept only unmarked tests still had + every marked operation counted as claimed - and the report then said + "NOTHING PASSED" about a run that measured nothing on purpose (seen on the + unit, 2026-09-07). Pure, held offline. + """ + return set().union(*(names for nodeid, names in verifies.items() + if nodeid in ran)) if verifies else set() + + def _measured_nothing(outcomes, claimed): """True when tests claimed operations and not one of them passed. @@ -641,7 +661,7 @@ def pytest_terminal_summary(terminalreporter, exitstatus, config): tr.line("the hardware session never connected, so no operation was measured; " "see the setup errors above") return - claimed = set().union(*_VERIFIES.values()) if _VERIFIES else set() + claimed = _claimed(_VERIFIES, _RAN) lines = _report_lines(cls, _OUTCOMES, claimed) width = max(len(label) for label, _names, _note in lines) # pytest files a setup or teardown failure under "error", not "failed" diff --git a/tests/test_hardware_report.py b/tests/test_hardware_report.py index 924ca93..9595691 100644 --- a/tests/test_hardware_report.py +++ b/tests/test_hardware_report.py @@ -189,6 +189,17 @@ def test_a_run_where_nothing_passed_is_flagged_not_read_as_clean(conftest): "nothing claimed is a deselected run, not a run that measured nothing") +def test_claims_come_from_tests_that_ran_not_tests_that_were_collected(conftest): + """pytest's -k deselects after our hook records claims; only tests that + produced a report count. Seen on the unit 2026-09-07: a -k run of three + unmarked tests reported 16 claimed operations and NOTHING PASSED.""" + verifies = {"t.py::a": {"set_param"}, "t.py::b": {"set_bypass", "switch_scene"}, + "t.py::c": set()} + assert conftest._claimed(verifies, ran={"t.py::b", "t.py::c"}) == {"set_bypass", "switch_scene"} + assert conftest._claimed(verifies, ran=set()) == set() + assert conftest._claimed({}, ran={"t.py::a"}) == set() + + def test_an_operation_no_test_claims_is_never_a_regression(conftest): """The 89. Nothing names them, so the run says nothing about them.""" lines = dict(_named(conftest._report_lines( From d4c6999c4a6810041c8240e1562ca8eb62762ffd Mon Sep 17 00:00:00 2001 From: Jonathan Stokes Date: Mon, 7 Sep 2026 20:10:21 -0500 Subject: [PATCH 7/7] docs: the report's claimed set is described as built from the tests that ran --- tests/hardware/conftest.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/hardware/conftest.py b/tests/hardware/conftest.py index 973c869..3ab6279 100644 --- a/tests/hardware/conftest.py +++ b/tests/hardware/conftest.py @@ -79,11 +79,11 @@ def _claims(items, operations, wanted, deselect): Narrows ``items`` in place to the tests naming ``wanted`` (when there is one), hands the rest to ``deselect``, and returns the claims of what is - left. The order matters and is the whole point: ``claimed`` in the - end-of-run report is the union of this, and a deselected test measured - nothing - recording it printed every operation the run never touched under - ``VERIFIED and claimed by a test, failed``, which reads as a - regression. Held offline by ``tests/test_hardware_report.py``. + left. Narrowing here is what stops the other tests RUNNING under + ``--verifies``; the report's ``claimed`` set is then built at the end from + the tests that produced a report (:func:`_claimed`), because pytest's own + ``-k`` deselects after this hook and a deselected test measured nothing. + Held offline by ``tests/test_hardware_report.py``. """ marks = {} for item in items: @@ -582,8 +582,8 @@ def _report_lines(cls, outcomes, claimed): """The end-of-run report, as ``(label, names, note)`` rows. Pure, so ``tests/test_hardware_report.py`` holds the arithmetic with no unit - attached. ``claimed`` is every operation some COLLECTED test says it - verifies, and it is what keeps the last line readable: ``QuadCortex`` + attached. ``claimed`` is every operation some test that RAN says it + verifies (:func:`_claimed`), and it is what keeps the last line readable: ``QuadCortex`` verifies EVERYTHING, so the plain difference against ``VERIFIED`` names all ~89 operations no test has ever driven, each tagged as a regression. Only something a test claims can regress - nothing else was measured - and only