Skip to content
17 changes: 17 additions & 0 deletions changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,23 @@ correction.

## Unreleased

### The first hardware run of the profile seam, and what it corrected

Run 2026-09-07 on Quad Cortex, CorOS 4.0.1 / d14e, straight after the seam
merged: 93 passed, 4 skipped, and two tests failed on records the seam itself
made stale. `connect()` now reads `Version` before the handshake, so a listener
registered through `before_handshake` sees three inbound `Version` messages, not
one: the full reply to that read, the unit's own `Version{READ}` a millisecond
behind it, and the answer to our announce. The same reply leaves the identity
entry holding serial and firmware ahead of the burst. Both tests and
`docs/protocol.md` section 4 now say so, with the measurement.

The run also showed the report calling a SKIPPED test a regression: the bypass
echo test declined to run (no stored bypass entry on the loaded preset) and
`set_bypass` landed on the "regressions by name" line. A skip measured nothing.
That line now counts failures only and is labelled "VERIFIED and claimed by a
test, failed"; skipped operations stay on the "failed or skipped" line.

### BREAKING: the generators take `--snapshot`, and constants moved

`scripts/generate_models.py`, `generate_params.py` and `generate_options.py`
Expand Down
6 changes: 3 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,9 +269,9 @@ returns immediately, which is what makes the metronome's tempo stream - a pair
per beat, on every connection, forever - cost nothing.

`entries.py` is the table of what is tracked: per entry, the message types that
carry it, the fields kept from each, and the read that fetches it. Two entries
today, `identity` and `dirty`; the rest of section 9's table arrives with the
surfaces that read it.
carry it, the fields kept from each, and the read that fetches it. Five entries
today - `identity`, `dirty`, `preset`, `scene` and `loaded`; the rest of section 9's
table arrives with the surfaces that read it.

`watch.py` is the write side. A write updates the cache immediately and the
unit's echo confirms it in the background, against one sentence: every field we
Expand Down
25 changes: 18 additions & 7 deletions docs/protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -524,11 +524,20 @@ version request - READ replies carry no `request_id` to disambiguate them, so
the transport hands a `Version` with no id to whichever waiter is first in line.

Skipping it also means the device asks nothing back. Measured 2026-08-27 on
`d14e`: a connect through `_hello()` and its whole burst produce exactly one
inbound `Version`, an `UPDATE` carrying `cortex_control_version_valid` in answer
to the announce, and eight seconds of idling after it produce none. So step 3
above is a consequence of step 2 rather than something the device does on
connecting.
`d14e`: `_hello()` and its whole burst produce exactly one inbound `Version`, an
`UPDATE` carrying `cortex_control_version_valid` in answer to the announce, and
eight seconds of idling after it produce none. So step 3 above is a consequence
of step 2 rather than something the device does on connecting.

Since ADR-0020, `connect()` itself makes one `Version` READ BEFORE calling
`_hello()`, to resolve the profile. Measured 2026-09-07 on CorOS 4.0.1 / d14e: a
`connect()` and its burst now carry exactly THREE inbound `Version` messages -
the full reply to that READ (15 fields, at +0.71 s), the unit's own
`Version{READ}` about 1 ms behind it (section 4.4), and the announce answer above (at
+0.73 s). The state layer listens from before the handshake, so the identity
entry holds `device_serial_number` and `app_fw_version` from that reply; the
reply also carries fields the entry does not keep, so the entry stays marked and
the first `device.firmware` read still asks the unit once.

### 4.3 Keepalive and disconnect

Expand Down Expand Up @@ -2029,8 +2038,10 @@ transport listener registered before the handshake: `connect()` handed back its
in, having seen only the `ResetCommsBuffers` echo and one `Version`. (That session read the
`Version` as the unit's own READ. Re-measured 2026-08-27 on d14e it is the unit's `UPDATE`
answering our version announce, carrying `cortex_control_version_valid`; `_hello()` sends no
host `Version` READ, and with none sent the unit asks nothing back. See section 4.) The
ModelRepo landed at 4.9 s, the 399 `File` listings and most settings at 5.1 s, and the seed
host `Version` READ, and with none sent the unit asks nothing back. Since ADR-0020 `connect()`
reads `Version` once before `_hello()`, so a listener registered before the handshake now sees
three `Version` messages - see section 4.) The ModelRepo landed at 4.9 s, the 399 `File`
listings and most settings at 5.1 s, and the seed
`RecallPreset` at 10.1 s - 474 messages of 24 distinct types by 15 s. So a listener attached
to the client `connect()` returns is about 3 s too late for the ModelRepo and 8 s too late
for the current preset, which is why `connect(before_handshake=...)` exists.
Expand Down
109 changes: 92 additions & 17 deletions tests/hardware/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@

import pytest

from pyquadcortex.protocol.proto import ProductionAutomation_pb2 as pa

#: This directory. Everything under it drives the unit and is gated on the flag.
SUITE = pathlib.Path(__file__).resolve().parent
ROOT = SUITE.parent.parent
Expand All @@ -34,6 +36,10 @@

#: ``operation -> list of outcomes``, one entry per phase that decided a test.
_OUTCOMES = {}
#: Node ids that produced at least one report - the tests that actually RAN,
#: whatever deselected the rest (our --verifies, or pytest's own -k, which runs
#: after this conftest's hook). Claims are read from these at report time.
_RAN = set()


def pytest_ignore_collect(collection_path, config):
Expand Down Expand Up @@ -73,11 +79,11 @@ def _claims(items, operations, wanted, deselect):

Narrows ``items`` in place to the tests naming ``wanted`` (when there is
one), hands the rest to ``deselect``, and returns the claims of what is
left. The order matters and is the whole point: ``claimed`` in the
end-of-run report is the union of this, and a deselected test measured
nothing - recording it printed every operation the run never touched under
``VERIFIED and claimed by a test, not passed``, which reads as a
regression. Held offline by ``tests/test_hardware_report.py``.
left. Narrowing here is what stops the other tests RUNNING under
``--verifies``; the report's ``claimed`` set is then built at the end from
the tests that produced a report (:func:`_claimed`), because pytest's own
``-k`` deselects after this hook and a deselected test measured nothing.
Held offline by ``tests/test_hardware_report.py``.
"""
marks = {}
for item in items:
Expand Down Expand Up @@ -209,6 +215,11 @@ class HandshakeBurst:
def __init__(self):
self._lock = threading.Lock()
self._names = []
#: ``(action, frozenset of field names)`` for every ``Version`` seen,
#: because since ADR-0020 a connect carries three of them and a test
#: that only counts cannot tell a retried identity read from a changed
#: handshake. Shapes only - the values are not this recorder's business.
self._versions = []
self._detach = None
self.closed = False
self.settled_in = None # seconds the burst took, or None if it timed out
Expand All @@ -224,6 +235,14 @@ def __call__(self, message):
# arrive after removal. It must not reopen the recording.
return
self._names.append(type(message).__name__)
if isinstance(message, pa.VersionMessage):
self._versions.append(
(message.action, frozenset(f.name for f, _ in message.ListFields())))

def versions(self):
"""The shape of every ``Version`` recorded: ``(action, field names)``."""
with self._lock:
return list(self._versions)

def record_until(self, sentinel, patience):
"""Record until a ``sentinel``-typed message arrives, then stop.
Expand Down Expand Up @@ -550,6 +569,7 @@ def _decides(when, outcome):

def pytest_runtest_logreport(report):
"""Record how each operation's tests came out, for the end-of-run report."""
_RAN.add(report.nodeid)
names = _VERIFIES.get(report.nodeid)
if not names:
return
Expand All @@ -562,30 +582,66 @@ def _report_lines(cls, outcomes, claimed):
"""The end-of-run report, as ``(label, names, note)`` rows.

Pure, so ``tests/test_hardware_report.py`` holds the arithmetic with no unit
attached. ``claimed`` is every operation some COLLECTED test says it
verifies, and it is what keeps the last line readable: ``QuadCortex``
attached. ``claimed`` is every operation some test that RAN says it
verifies (:func:`_claimed`), and it is what keeps the last line readable: ``QuadCortex``
verifies EVERYTHING, so the plain difference against ``VERIFIED`` names all
~89 operations no test has ever driven, each tagged as a regression. Only
something a test claims can regress - nothing else was measured.
something a test claims can regress - nothing else was measured - and only
a FAILED test regresses it; a skipped one measured nothing.
"""
from pyquadcortex.protocol.support import EVERYTHING

passed = {op for op, seen in outcomes.items()
if seen and all(o == "passed" for o in seen)}
failed = {op for op, seen in outcomes.items()
if any(o != "passed" for o in seen)}
not_passed = {op for op, seen in outcomes.items()
if any(o != "passed" for o in seen)}
# A SKIP is a test that declined to measure - a precondition the loaded
# preset did not meet, an operator-only capture - and says nothing about
# the unit. Only a FAILURE is a regression. Measured on the first post-merge
# run (2026-09-07): the bypass echo test skipped for want of a stored bypass
# entry and the old line called set_bypass a regression.
# pytest reports a setup or teardown ERROR with outcome "failed" too, so an
# errored test lands here as well (checked against pytest 9.1.1).
failed = {op for op, seen in outcomes.items() if "failed" in seen}
verified = (set(cls.operations()) if cls.VERIFIED is EVERYTHING
else set(cls.VERIFIED))
return [
("passed", sorted(passed), ""),
("failed or skipped", sorted(failed), ""),
("failed or skipped", sorted(not_passed), ""),
("passed, not VERIFIED", sorted(passed - verified),
"<- candidates to add"),
("VERIFIED and claimed by a test, not passed",
sorted((verified & claimed) - passed), "<- regressions by name"),
("VERIFIED and claimed by a test, failed",
sorted(verified & claimed & failed), "<- regressions by name"),
]


def _claimed(verifies, ran):
"""The operations claimed by tests that actually ran.

``verifies`` maps node id to the names its marker carries; ``ran`` is the
node ids that produced a report. Built at report time rather than at
collection because pytest's own ``-k`` deselection runs AFTER this
conftest's hook, so a `-k` run that kept only unmarked tests still had
every marked operation counted as claimed - and the report then said
"NOTHING PASSED" about a run that measured nothing on purpose (seen on the
unit, 2026-09-07). Pure, held offline.
"""
return set().union(*(names for nodeid, names in verifies.items()
if nodeid in ran)) if verifies else set()


def _measured_nothing(outcomes, claimed):
"""True when tests claimed operations and not one of them passed.

The regression line is empty on a healthy run AND on a run where every
test skipped its precondition, so an empty line alone cannot be read as
clean. Pure, held offline like :func:`_report_lines`.
"""
passed = {op for op, seen in outcomes.items()
if seen and all(o == "passed" for o in seen)}
return bool(claimed) and not passed


def pytest_terminal_summary(terminalreporter, exitstatus, config):
"""Print which operations this run measured on the connected profile.

Expand All @@ -595,14 +651,33 @@ def pytest_terminal_summary(terminalreporter, exitstatus, config):
"""
if not config.getoption("--hardware"):
return
tr = terminalreporter
cls = getattr(config, "_profile", None)
if cls is None: # the session never connected, so there is nothing to report
if cls is None:
# The session never connected. pytest's own summary carries the setup
# errors; this line exists so the ABSENCE of the operations report is
# never mistaken for "nothing to report".
tr.section("operations: no report")
tr.line("the hardware session never connected, so no operation was measured; "
"see the setup errors above")
return
claimed = set().union(*_VERIFIES.values()) if _VERIFIES else set()
claimed = _claimed(_VERIFIES, _RAN)
lines = _report_lines(cls, _OUTCOMES, claimed)
width = max(len(label) for label, _names, _note in lines)
tr = terminalreporter
# pytest files a setup or teardown failure under "error", not "failed"
# (_pytest/runner.py pytest_report_teststatus), and a restore that could
# not finish is exactly that shape - so the count names errors separately
# rather than losing them. The operations lines are unaffected: they read
# report.outcome, which is "failed" for an error too.
counts = {k: len(tr.stats.get(k, ()))
for k in ("passed", "failed", "error", "skipped")}
tr.section(f"operations on {cls.__name__} "
f"(CorOS {', '.join(cls.MEASURED_ON)}, {cls.EVIDENCE.name})")
tr.line(f"tests: {counts['passed']} passed, {counts['failed']} failed, "
f"{counts['error']} errored, {counts['skipped']} skipped; "
f"operations measured: {len(_OUTCOMES)} of {len(claimed)} claimed")
for label, names, note in lines:
tr.line(f"{label + ':':<{width + 1}} {names} {note}".rstrip())
tr.line(f"{label + ':':<{width + 1}} {names} ({len(names)}) {note}".rstrip())
if _measured_nothing(_OUTCOMES, claimed):
tr.line("NOTHING PASSED: every claimed operation failed or skipped, so the "
"empty regression line above says nothing about the unit")
13 changes: 7 additions & 6 deletions tests/hardware/test_broadcast_listener.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,13 @@ def test_a_listener_registered_before_connecting_sees_the_handshake_burst(
"""The burst is what makes a push-fed cache warm for free.

Note what the wait below says about the hook: ``connect()`` returns about 2 s
in, with only the ResetCommsBuffers echo and one Version recorded - the
unit's UPDATE answering our version announce, not a READ of its own
(``docs/protocol.md`` section 4) - and the state burst starts arriving about
5 s in. So a listener
registered on the client ``connect()`` hands back is already too late, which
is the whole reason ``before_handshake`` exists.
in, with only the ResetCommsBuffers echo and three ``Version`` messages
recorded - the full reply to connect's own identity READ, the unit's
``Version{READ}`` a millisecond behind it, and the UPDATE answering our
version announce (``docs/protocol.md`` section 4; measured 2026-09-07 on
4.0.1 / d14e) - and the state burst starts arriving about 5 s in. So a
listener registered on the client ``connect()`` hands back is already too
late, which is the whole reason ``before_handshake`` exists.

Confirmed on this unit (2026-08-12, CorOS 4.0.1 / d14e): 474 messages of 24
distinct types by 15 s. The floors below sit well under that, because a test
Expand Down
Loading
Loading