diff --git a/changelog.md b/changelog.md index 9ea98c0..d8b2b27 100644 --- a/changelog.md +++ b/changelog.md @@ -20,6 +20,23 @@ correction. ## Unreleased +### The first hardware run of the profile seam, and what it corrected + +Run 2026-09-07 on Quad Cortex, CorOS 4.0.1 / d14e, straight after the seam +merged: 93 passed, 4 skipped, and two tests failed on records the seam itself +made stale. `connect()` now reads `Version` before the handshake, so a listener +registered through `before_handshake` sees three inbound `Version` messages, not +one: the full reply to that read, the unit's own `Version{READ}` a millisecond +behind it, and the answer to our announce. The same reply leaves the identity +entry holding serial and firmware ahead of the burst. Both tests and +`docs/protocol.md` section 4 now say so, with the measurement. + +The run also showed the report calling a SKIPPED test a regression: the bypass +echo test declined to run (no stored bypass entry on the loaded preset) and +`set_bypass` landed on the "regressions by name" line. A skip measured nothing. +That line now counts failures only and is labelled "VERIFIED and claimed by a +test, failed"; skipped operations stay on the "failed or skipped" line. + ### BREAKING: the generators take `--snapshot`, and constants moved `scripts/generate_models.py`, `generate_params.py` and `generate_options.py` diff --git a/docs/architecture.md b/docs/architecture.md index 1fa5a12..642cfbc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -269,9 +269,9 @@ returns immediately, which is what makes the metronome's tempo stream - a pair per beat, on every connection, forever - cost nothing. `entries.py` is the table of what is tracked: per entry, the message types that -carry it, the fields kept from each, and the read that fetches it. Two entries -today, `identity` and `dirty`; the rest of section 9's table arrives with the -surfaces that read it. +carry it, the fields kept from each, and the read that fetches it. Five entries +today - `identity`, `dirty`, `preset`, `scene` and `loaded`; the rest of section 9's +table arrives with the surfaces that read it. `watch.py` is the write side. A write updates the cache immediately and the unit's echo confirms it in the background, against one sentence: every field we diff --git a/docs/protocol.md b/docs/protocol.md index 642773d..cc166d7 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -524,11 +524,20 @@ version request - READ replies carry no `request_id` to disambiguate them, so the transport hands a `Version` with no id to whichever waiter is first in line. Skipping it also means the device asks nothing back. Measured 2026-08-27 on -`d14e`: a connect through `_hello()` and its whole burst produce exactly one -inbound `Version`, an `UPDATE` carrying `cortex_control_version_valid` in answer -to the announce, and eight seconds of idling after it produce none. So step 3 -above is a consequence of step 2 rather than something the device does on -connecting. +`d14e`: `_hello()` and its whole burst produce exactly one inbound `Version`, an +`UPDATE` carrying `cortex_control_version_valid` in answer to the announce, and +eight seconds of idling after it produce none. So step 3 above is a consequence +of step 2 rather than something the device does on connecting. + +Since ADR-0020, `connect()` itself makes one `Version` READ BEFORE calling +`_hello()`, to resolve the profile. Measured 2026-09-07 on CorOS 4.0.1 / d14e: a +`connect()` and its burst now carry exactly THREE inbound `Version` messages - +the full reply to that READ (15 fields, at +0.71 s), the unit's own +`Version{READ}` about 1 ms behind it (section 4.4), and the announce answer above (at ++0.73 s). The state layer listens from before the handshake, so the identity +entry holds `device_serial_number` and `app_fw_version` from that reply; the +reply also carries fields the entry does not keep, so the entry stays marked and +the first `device.firmware` read still asks the unit once. ### 4.3 Keepalive and disconnect @@ -2029,8 +2038,10 @@ transport listener registered before the handshake: `connect()` handed back its in, having seen only the `ResetCommsBuffers` echo and one `Version`. (That session read the `Version` as the unit's own READ. Re-measured 2026-08-27 on d14e it is the unit's `UPDATE` answering our version announce, carrying `cortex_control_version_valid`; `_hello()` sends no -host `Version` READ, and with none sent the unit asks nothing back. See section 4.) The -ModelRepo landed at 4.9 s, the 399 `File` listings and most settings at 5.1 s, and the seed +host `Version` READ, and with none sent the unit asks nothing back. Since ADR-0020 `connect()` +reads `Version` once before `_hello()`, so a listener registered before the handshake now sees +three `Version` messages - see section 4.) The ModelRepo landed at 4.9 s, the 399 `File` +listings and most settings at 5.1 s, and the seed `RecallPreset` at 10.1 s - 474 messages of 24 distinct types by 15 s. So a listener attached to the client `connect()` returns is about 3 s too late for the ModelRepo and 8 s too late for the current preset, which is why `connect(before_handshake=...)` exists. diff --git a/tests/hardware/conftest.py b/tests/hardware/conftest.py index cd6c8e7..3ab6279 100644 --- a/tests/hardware/conftest.py +++ b/tests/hardware/conftest.py @@ -21,6 +21,8 @@ import pytest +from pyquadcortex.protocol.proto import ProductionAutomation_pb2 as pa + #: This directory. Everything under it drives the unit and is gated on the flag. SUITE = pathlib.Path(__file__).resolve().parent ROOT = SUITE.parent.parent @@ -34,6 +36,10 @@ #: ``operation -> list of outcomes``, one entry per phase that decided a test. _OUTCOMES = {} +#: Node ids that produced at least one report - the tests that actually RAN, +#: whatever deselected the rest (our --verifies, or pytest's own -k, which runs +#: after this conftest's hook). Claims are read from these at report time. +_RAN = set() def pytest_ignore_collect(collection_path, config): @@ -73,11 +79,11 @@ def _claims(items, operations, wanted, deselect): Narrows ``items`` in place to the tests naming ``wanted`` (when there is one), hands the rest to ``deselect``, and returns the claims of what is - left. The order matters and is the whole point: ``claimed`` in the - end-of-run report is the union of this, and a deselected test measured - nothing - recording it printed every operation the run never touched under - ``VERIFIED and claimed by a test, not passed``, which reads as a - regression. Held offline by ``tests/test_hardware_report.py``. + left. Narrowing here is what stops the other tests RUNNING under + ``--verifies``; the report's ``claimed`` set is then built at the end from + the tests that produced a report (:func:`_claimed`), because pytest's own + ``-k`` deselects after this hook and a deselected test measured nothing. + Held offline by ``tests/test_hardware_report.py``. """ marks = {} for item in items: @@ -209,6 +215,11 @@ class HandshakeBurst: def __init__(self): self._lock = threading.Lock() self._names = [] + #: ``(action, frozenset of field names)`` for every ``Version`` seen, + #: because since ADR-0020 a connect carries three of them and a test + #: that only counts cannot tell a retried identity read from a changed + #: handshake. Shapes only - the values are not this recorder's business. + self._versions = [] self._detach = None self.closed = False self.settled_in = None # seconds the burst took, or None if it timed out @@ -224,6 +235,14 @@ def __call__(self, message): # arrive after removal. It must not reopen the recording. return self._names.append(type(message).__name__) + if isinstance(message, pa.VersionMessage): + self._versions.append( + (message.action, frozenset(f.name for f, _ in message.ListFields()))) + + def versions(self): + """The shape of every ``Version`` recorded: ``(action, field names)``.""" + with self._lock: + return list(self._versions) def record_until(self, sentinel, patience): """Record until a ``sentinel``-typed message arrives, then stop. @@ -550,6 +569,7 @@ def _decides(when, outcome): def pytest_runtest_logreport(report): """Record how each operation's tests came out, for the end-of-run report.""" + _RAN.add(report.nodeid) names = _VERIFIES.get(report.nodeid) if not names: return @@ -562,30 +582,66 @@ def _report_lines(cls, outcomes, claimed): """The end-of-run report, as ``(label, names, note)`` rows. Pure, so ``tests/test_hardware_report.py`` holds the arithmetic with no unit - attached. ``claimed`` is every operation some COLLECTED test says it - verifies, and it is what keeps the last line readable: ``QuadCortex`` + attached. ``claimed`` is every operation some test that RAN says it + verifies (:func:`_claimed`), and it is what keeps the last line readable: ``QuadCortex`` verifies EVERYTHING, so the plain difference against ``VERIFIED`` names all ~89 operations no test has ever driven, each tagged as a regression. Only - something a test claims can regress - nothing else was measured. + something a test claims can regress - nothing else was measured - and only + a FAILED test regresses it; a skipped one measured nothing. """ from pyquadcortex.protocol.support import EVERYTHING passed = {op for op, seen in outcomes.items() if seen and all(o == "passed" for o in seen)} - failed = {op for op, seen in outcomes.items() - if any(o != "passed" for o in seen)} + not_passed = {op for op, seen in outcomes.items() + if any(o != "passed" for o in seen)} + # A SKIP is a test that declined to measure - a precondition the loaded + # preset did not meet, an operator-only capture - and says nothing about + # the unit. Only a FAILURE is a regression. Measured on the first post-merge + # run (2026-09-07): the bypass echo test skipped for want of a stored bypass + # entry and the old line called set_bypass a regression. + # pytest reports a setup or teardown ERROR with outcome "failed" too, so an + # errored test lands here as well (checked against pytest 9.1.1). + failed = {op for op, seen in outcomes.items() if "failed" in seen} verified = (set(cls.operations()) if cls.VERIFIED is EVERYTHING else set(cls.VERIFIED)) return [ ("passed", sorted(passed), ""), - ("failed or skipped", sorted(failed), ""), + ("failed or skipped", sorted(not_passed), ""), ("passed, not VERIFIED", sorted(passed - verified), "<- candidates to add"), - ("VERIFIED and claimed by a test, not passed", - sorted((verified & claimed) - passed), "<- regressions by name"), + ("VERIFIED and claimed by a test, failed", + sorted(verified & claimed & failed), "<- regressions by name"), ] +def _claimed(verifies, ran): + """The operations claimed by tests that actually ran. + + ``verifies`` maps node id to the names its marker carries; ``ran`` is the + node ids that produced a report. Built at report time rather than at + collection because pytest's own ``-k`` deselection runs AFTER this + conftest's hook, so a `-k` run that kept only unmarked tests still had + every marked operation counted as claimed - and the report then said + "NOTHING PASSED" about a run that measured nothing on purpose (seen on the + unit, 2026-09-07). Pure, held offline. + """ + return set().union(*(names for nodeid, names in verifies.items() + if nodeid in ran)) if verifies else set() + + +def _measured_nothing(outcomes, claimed): + """True when tests claimed operations and not one of them passed. + + The regression line is empty on a healthy run AND on a run where every + test skipped its precondition, so an empty line alone cannot be read as + clean. Pure, held offline like :func:`_report_lines`. + """ + passed = {op for op, seen in outcomes.items() + if seen and all(o == "passed" for o in seen)} + return bool(claimed) and not passed + + def pytest_terminal_summary(terminalreporter, exitstatus, config): """Print which operations this run measured on the connected profile. @@ -595,14 +651,33 @@ def pytest_terminal_summary(terminalreporter, exitstatus, config): """ if not config.getoption("--hardware"): return + tr = terminalreporter cls = getattr(config, "_profile", None) - if cls is None: # the session never connected, so there is nothing to report + if cls is None: + # The session never connected. pytest's own summary carries the setup + # errors; this line exists so the ABSENCE of the operations report is + # never mistaken for "nothing to report". + tr.section("operations: no report") + tr.line("the hardware session never connected, so no operation was measured; " + "see the setup errors above") return - claimed = set().union(*_VERIFIES.values()) if _VERIFIES else set() + claimed = _claimed(_VERIFIES, _RAN) lines = _report_lines(cls, _OUTCOMES, claimed) width = max(len(label) for label, _names, _note in lines) - tr = terminalreporter + # pytest files a setup or teardown failure under "error", not "failed" + # (_pytest/runner.py pytest_report_teststatus), and a restore that could + # not finish is exactly that shape - so the count names errors separately + # rather than losing them. The operations lines are unaffected: they read + # report.outcome, which is "failed" for an error too. + counts = {k: len(tr.stats.get(k, ())) + for k in ("passed", "failed", "error", "skipped")} tr.section(f"operations on {cls.__name__} " f"(CorOS {', '.join(cls.MEASURED_ON)}, {cls.EVIDENCE.name})") + tr.line(f"tests: {counts['passed']} passed, {counts['failed']} failed, " + f"{counts['error']} errored, {counts['skipped']} skipped; " + f"operations measured: {len(_OUTCOMES)} of {len(claimed)} claimed") for label, names, note in lines: - tr.line(f"{label + ':':<{width + 1}} {names} {note}".rstrip()) + tr.line(f"{label + ':':<{width + 1}} {names} ({len(names)}) {note}".rstrip()) + if _measured_nothing(_OUTCOMES, claimed): + tr.line("NOTHING PASSED: every claimed operation failed or skipped, so the " + "empty regression line above says nothing about the unit") diff --git a/tests/hardware/test_broadcast_listener.py b/tests/hardware/test_broadcast_listener.py index 59acb82..c6ec342 100644 --- a/tests/hardware/test_broadcast_listener.py +++ b/tests/hardware/test_broadcast_listener.py @@ -59,12 +59,13 @@ def test_a_listener_registered_before_connecting_sees_the_handshake_burst( """The burst is what makes a push-fed cache warm for free. Note what the wait below says about the hook: ``connect()`` returns about 2 s - in, with only the ResetCommsBuffers echo and one Version recorded - the - unit's UPDATE answering our version announce, not a READ of its own - (``docs/protocol.md`` section 4) - and the state burst starts arriving about - 5 s in. So a listener - registered on the client ``connect()`` hands back is already too late, which - is the whole reason ``before_handshake`` exists. + in, with only the ResetCommsBuffers echo and three ``Version`` messages + recorded - the full reply to connect's own identity READ, the unit's + ``Version{READ}`` a millisecond behind it, and the UPDATE answering our + version announce (``docs/protocol.md`` section 4; measured 2026-09-07 on + 4.0.1 / d14e) - and the state burst starts arriving about 5 s in. So a + listener registered on the client ``connect()`` hands back is already too + late, which is the whole reason ``before_handshake`` exists. Confirmed on this unit (2026-08-12, CorOS 4.0.1 / d14e): 474 messages of 24 distinct types by 15 s. The floors below sit well under that, because a test diff --git a/tests/hardware/test_model_state.py b/tests/hardware/test_model_state.py index 780a702..db4698e 100644 --- a/tests/hardware/test_model_state.py +++ b/tests/hardware/test_model_state.py @@ -174,30 +174,59 @@ def test_nothing_the_burst_delivered_is_read_again_on_first_access( "again, which is the round trip the cache exists to avoid") -def test_the_burst_does_not_warm_what_the_unit_never_announces(burst_warmed, - handshake_burst): - """The other half, and the reason the read path exists. - - The unit does send a ``Version`` during the handshake, but it is the answer - to our version announce - it sets ``cortex_control_version_valid`` and none - of the unit's own fields. So identity is exactly the case section 9's third - column is for: where the unit does not tell us, we ask. - - The count is asserted because the empty cache below is not evidence for that - story: it was equally true of the story this replaced, which had the unit - volunteering a ``Version`` READ of its own during connect. ``_hello`` sends - no host ``Version`` READ, and the unit only asks when asked, so one is the - number - and a second would mean the handshake has changed under us. +def test_the_burst_warms_identity_from_connects_own_version_read(burst_warmed, + handshake_burst, + record_property): + """Identity reaches the cache through the ONE Version READ connect() makes. + + The unit never volunteers its identity, so the model reads it (section 9's + third column). Since ADR-0020, ``connect()`` issues that read itself, before + the handshake, to resolve the profile - and the state layer listens from + before the handshake, so it sees the reply. Measured 2026-09-07 on CorOS + 4.0.1 / d14e: exactly THREE inbound ``Version`` messages through connect and + its burst - the full reply to connect's READ (15 fields, +0.71 s), the + unit's own ``Version{READ}`` about 1 ms behind it (the question it asks Cortex + Control, section 4.4), and the ``UPDATE`` carrying + ``cortex_control_version_valid`` that answers our announce (+0.73 s). + ``_hello`` itself still sends no READ; the 2026-08-27 measurement of one + inbound ``Version`` stands for ``_hello`` alone. + + Asserted by SHAPE rather than by count, because a unit that was still + booting makes ``connect()`` retry its identity READ, and each retry adds a + full reply plus the unit's own READ - a healthy run that a bare ``== 3`` + would call a regression. So: exactly one announce answer, at least one full + reply, and one unit READ per full reply, and nothing else. A message of any + other shape (the handshake changed under us) or no full reply at all + (connect stopped reading identity, and with no READ the unit asks nothing + back) is loud. The number of identity reads is recorded, not asserted. The cache holds the two + fields the entry keeps; the reply also carries fields it does not, so the + entry stays marked and the first read of ``device.firmware`` still goes to + the unit (see ``identity`` in device/entries.py). """ - versions = handshake_burst.names().count("VersionMessage") - assert versions == 1, ( - f"the connect burst carried {versions} Version message(s), not the one " - f"answering our version announce. Section 4 of docs/protocol.md says " - f"the unit asks for our version only when we ask for its.") - assert burst_warmed["identity"] == {}, ( - f"the burst carried the unit's own identity after all, which is worth " - f"knowing - it held {burst_warmed['identity']}. If that is now true, " - f"the entry's docstring in device/entries.py is wrong.") + shapes = handshake_burst.versions() + full = [s for s in shapes if "device_serial_number" in s[1]] + own_reads = [s for s in shapes + if s[0] == pa.MessageAction.READ and s[1] == {"action"}] + announce = [s for s in shapes if "cortex_control_version_valid" in s[1]] + record_property("identity_reads", len(full)) + assert len(announce) == 1, ( + f"{len(announce)} answers to our version announce; one is measured. " + f"Shapes seen: {shapes}") + assert full, ( + f"no full Version reply reached the listener: connect() stopped reading " + f"identity before the handshake, or the unit stopped answering. Shapes " + f"seen: {shapes}") + assert len(own_reads) == len(full), ( + f"{len(full)} full replies but {len(own_reads)} unit READs; the unit asks " + f"its own question once per answer (protocol.md 4.4). Shapes: {shapes}") + assert len(shapes) == len(full) + len(own_reads) + 1, ( + f"a Version of a shape this test does not know arrived - the handshake " + f"changed under us. Shapes: {shapes}") + assert set(burst_warmed["identity"]) == {"device_serial_number", "app_fw_version"}, ( + f"connect's Version read should have left exactly the two kept identity " + f"fields in the cache; it held {burst_warmed['identity']}. The unit's " + f"full reply carries both (protocol.md, version read row); if it stopped, " + f"that is a finding for protocol.md, not a reason to loosen this.") def test_a_version_read_is_answered_and_then_questioned(qc, record_property): diff --git a/tests/hardware/test_preset_surface.py b/tests/hardware/test_preset_surface.py index aa83662..71305db 100644 --- a/tests/hardware/test_preset_surface.py +++ b/tests/hardware/test_preset_surface.py @@ -196,15 +196,18 @@ def test_an_inactive_scene_refuses_writes_and_reads_fine(device): def test_the_burst_delivered_every_entry_the_preset_surface_reads(burst_warmed): """Measured 2026-08-15: the burst carries RecallPreset, SetlistPosition, - PresetDirty and Scene at about 10 s, inside ten milliseconds. Identity is - NOT in it - the unit never announces its own firmware - so that one is - expected to be empty and is asserted, to keep this from passing on a run - where the burst delivered nothing at all.""" + PresetDirty and Scene at about 10 s, inside ten milliseconds. The unit never + announces its own firmware; identity is in the cache anyway since ADR-0020, + because ``connect()`` reads ``Version`` before the handshake and the state + layer is listening by then (measured 2026-09-07 on 4.0.1 / d14e). Asserted + with its two kept fields, to keep this from passing on a run where the burst + delivered nothing at all.""" for name in ("preset", "scene", "dirty", "loaded"): assert burst_warmed[name], f"the burst delivered nothing for {name}" - assert not burst_warmed["identity"], ( - "identity arrived in the burst, which contradicts what the entry's " - "docstring says the unit does") + assert set(burst_warmed["identity"]) == {"device_serial_number", "app_fw_version"}, ( + f"connect's own Version read should have warmed identity with its two " + f"kept fields; the cache held {burst_warmed['identity']}. If the unit's " + f"reply stopped carrying one, that is a finding for protocol.md.") def test_reading_the_preset_surface_costs_no_round_trip(device, model_cache): diff --git a/tests/test_hardware_report.py b/tests/test_hardware_report.py index 69f89b1..9595691 100644 --- a/tests/test_hardware_report.py +++ b/tests/test_hardware_report.py @@ -117,8 +117,8 @@ def test_the_report_names_the_four_things_a_maintainer_asks_for(conftest): assert lines["failed or skipped"] == ["set_bypass"] # set_ir passed and the profile does not claim it: a candidate. assert lines["passed, not VERIFIED"] == ["set_ir"] - # set_bypass is claimed by the profile, a test names it, and it did not pass. - assert lines["VERIFIED and claimed by a test, not passed"] == ["set_bypass"] + # set_bypass is claimed by the profile, a test names it, and it FAILED. + assert lines["VERIFIED and claimed by a test, failed"] == ["set_bypass"] def test_one_failing_run_of_an_operation_sinks_all_of_them(conftest): @@ -145,17 +145,67 @@ def test_a_regression_is_only_an_operation_some_test_claims(conftest): # set_ir and set_block are VERIFIED here too - EVERYTHING says so - and no # test names either, so neither is a regression. Only set_param is. - assert lines["VERIFIED and claimed by a test, not passed"] == ["set_param"] + assert lines["VERIFIED and claimed by a test, failed"] == ["set_param"] assert lines["passed, not VERIFIED"] == [], ( "a profile that verifies everything can have no candidates") +def test_a_skipped_test_is_not_a_regression(conftest): + """Measured on the first post-merge run, 2026-09-07: the bypass echo test + skipped ("no stored bypass entry") and the report called set_bypass a + regression. A skip measured nothing; it belongs on the failed-or-skipped + line and nowhere else.""" + lines = dict(_named(conftest._report_lines( + _Everything, {"set_bypass": ["skipped"], "set_param": ["failed"]}, + claimed={"set_bypass", "set_param"}))) + + assert lines["failed or skipped"] == ["set_bypass", "set_param"] + assert lines["VERIFIED and claimed by a test, failed"] == ["set_param"] + + +def test_a_pass_and_a_skip_on_one_operation_is_neither_passed_nor_a_regression(conftest): + """Two tests name an operation; one skipped its precondition, one passed. + Not all of it passed, so it is not offered as VERIFIED; nothing failed, so + it is not a regression. It sits on the middle line, where a reader sees the + skip.""" + lines = dict(_named(conftest._report_lines( + _Everything, {"set_param": ["passed", "skipped"]}, claimed={"set_param"}))) + + assert lines["passed"] == [] + assert lines["failed or skipped"] == ["set_param"] + assert lines["VERIFIED and claimed by a test, failed"] == [] + + +def test_a_run_where_nothing_passed_is_flagged_not_read_as_clean(conftest): + """An all-skip run has an empty regression line, same as a healthy run. + The flag is what tells them apart.""" + assert conftest._measured_nothing( + {"set_param": ["skipped"], "set_bypass": ["skipped"]}, + claimed={"set_param", "set_bypass"}) is True + assert conftest._measured_nothing( + {"set_param": ["passed"], "set_bypass": ["skipped"]}, + claimed={"set_param", "set_bypass"}) is False + assert conftest._measured_nothing({}, claimed=set()) is False, ( + "nothing claimed is a deselected run, not a run that measured nothing") + + +def test_claims_come_from_tests_that_ran_not_tests_that_were_collected(conftest): + """pytest's -k deselects after our hook records claims; only tests that + produced a report count. Seen on the unit 2026-09-07: a -k run of three + unmarked tests reported 16 claimed operations and NOTHING PASSED.""" + verifies = {"t.py::a": {"set_param"}, "t.py::b": {"set_bypass", "switch_scene"}, + "t.py::c": set()} + assert conftest._claimed(verifies, ran={"t.py::b", "t.py::c"}) == {"set_bypass", "switch_scene"} + assert conftest._claimed(verifies, ran=set()) == set() + assert conftest._claimed({}, ran={"t.py::a"}) == set() + + def test_an_operation_no_test_claims_is_never_a_regression(conftest): """The 89. Nothing names them, so the run says nothing about them.""" lines = dict(_named(conftest._report_lines( _Everything, {"set_param": ["passed"]}, claimed={"set_param"}))) - assert lines["VERIFIED and claimed by a test, not passed"] == [] + assert lines["VERIFIED and claimed by a test, failed"] == [] def test_a_profile_with_nothing_claimed_reports_nothing_as_a_regression(conftest): @@ -219,8 +269,8 @@ def test_a_deselected_test_claims_nothing(conftest, collection): The recording used to happen before the deselection, so `claimed` was built from every COLLECTED test - and an operation whose test never ran - was printed under `VERIFIED and claimed by a test, not passed`, which the - report labels a regression. A run of one test reported ~104 of them. + was printed under the line the report labels a regression (then + `not passed`, now `failed`). A run of one test reported ~104 of them. """ items = [_Item("t.py::a", "set_param"), _Item("t.py::b", "set_bypass")] config = _Config(verifies="set_param") @@ -234,7 +284,7 @@ def test_a_deselected_test_claims_nothing(conftest, collection): claimed = set().union(*collection.values()) lines = dict(_named(conftest._report_lines( _Everything, {"set_param": ["passed"]}, claimed))) - assert lines["VERIFIED and claimed by a test, not passed"] == [], ( + assert lines["VERIFIED and claimed by a test, failed"] == [], ( "set_bypass was deselected, so this run measured nothing about it")