Skip to content

fix(hardware-suite): the burst recorder waits for the whole tail, not its first message - #68

Merged
jonathanstokes merged 6 commits into
mainfrom
fix/connect-burst-recorder-race
Sep 14, 2026
Merged

jonathanstokes merged 6 commits into
mainfrom
fix/connect-burst-recorder-race

Conversation

@jonathanstokes

@jonathanstokes jonathanstokes commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two hardware tests failed together on 2026-09-11 and passed on a re-run of the same commit - test_the_connect_burst_warms_the_cache and test_the_burst_delivered_every_entry_the_preset_surface_reads. This is a genuine race in the suite's burst recorder, not an under-specified wait, and no timeout was widened to fix it.

The connect burst closes with four state messages - RecallPreset, SetlistPosition, PresetDirty, Scene - and they are a group, not a sequence. Re-measured 2026-09-14 on 4.0.1 / d14e over three connections: they land at about 11.1 s, spread over 6.0, 3.6 and 5.8 ms, in that order every time. RecallPreset is the first of them.

record_until stopped on RecallPreset alone, believing it was the tail, and polled for it at 100 ms. Its stop therefore landed at a uniformly random point in the 100 ms after RecallPreset arrived, and a few milliseconds of that window are before the other three have landed. When it fell there the recorder came off the transport and the fixture took its burst_warmed snapshot of a cache that had not yet heard PresetDirty or Scene. The cache stays on the transport all run and got them milliseconds later - which is why only the two tests reading the snapshot failed, and every test reading the live cache passed.

The stop condition is now the postcondition rather than a proxy for it: wait until every message in BURST_TAIL has arrived. Patience is unchanged at 30 s.

Changes

  • record_until(sentinels, patience) waits for every type in the collection. It refuses a single name (a str is a collection of its letters), an empty collection (satisfied by the first poll), and message classes (which match nothing the recorder stores). All refusals detach on the way out.
  • The fixture registers the model cache before the recorder. Listeners run in registration order, so registered the other way round the poll could see the fourth message recorded and snapshot the cache before apply_push had run for it - the same stale snapshot, one __call__ wide instead of six milliseconds, and harder to spot because unfinished() would truthfully return None.
  • unfinished() explains a cut-off recording in words, and prints once in the run's own report as well as in the tests that guard on it - --verifies or a single-file run deselects all of those while the fixture still waits. It distinguishes settled, timed out, interrupted, refused, and never-called, because they want different responses.
  • _seen_types is maintained in __call__, so the poll is a set difference instead of a walk of the whole recording under the lock the RX thread needs.
  • tail_positions() records how far from the end of the recording each closing message landed. The stop condition proves the four arrived; it cannot prove they are the last four, which is what the snapshot actually rests on.
  • Offline tests hold BURST_TAIL against entries.ENTRIES per feed type, so a cache entry added with a new burst message cannot silently re-open this. A new entry comes through BURST_TAIL, OUTSIDE_THE_BURST or NOT_WARMED_BY_THE_BURST, each with its reason.
  • docs/protocol.md, tests/hardware/readme.md, CLAUDE.md and STEERING.md record the measurement and the rule.

Nothing under pyquadcortex/ changed. The bug was in how the suite decided the burst was over, not in the library's handling of it.

Test plan

  • Offline suite: 2738 passed, 3 skipped
  • mypy: no issues in 46 source files
  • Offline reproduction: test_record_until_waits_for_the_whole_tail_not_just_the_first_of_it stages the gap three poll intervals wide, so it does not depend on winning a race. An independent reviewer mutated the recorder to the old behaviour and confirmed it fails.
  • Burst tail re-measured over three connections (order stable, spread 3.6-6.0 ms)
  • pytest tests/hardware --hardware on the final commit

Hardware run

Commit 45db700ca2a3253fcc80c3d66a41b40efd326299 (main merged in), Quad Cortex CorOS 4.0.1, app_fw d14e.

============== operations on QuadCortex (CorOS 4.0.1, MAINTAINER) ==============
tests: 110 passed, 0 failed, 0 errored, 3 skipped; operations measured: 17 of 17 claimed
passed:                                 ['active_scene', 'clear_expression', 'create_local_backup', 'read_current_preset', 'set_bypass', 'set_chain_input', 'set_expression', 'set_global_eq', 'set_hold_timing', 'set_input_port', 'set_param', 'set_scene_color', 'set_scene_label', 'set_tempo_mode', 'switch_scene', 'tempo_mode', 'update_settings'] (17)
failed or skipped:                      [] (0)
passed, not VERIFIED:                   [] (0)   <- candidates to add
VERIFIED and claimed by a test, failed: [] (0)   <- regressions by name

================== 110 passed, 3 skipped in 181.18s (0:03:01) ==================

The new connect burst: cut off section did not print, which is the burst reporting that it settled. Run on the final commit after main was merged in, so it covers create_local_backup and the rest of PR #47 as they stand on this branch. Earlier green runs: 83ae721 at 109 passed, 89aa01c at 100 passed.

On the flake rate

The repo does not quote a percentage for it, deliberately. What is observed is one real failure on 2026-09-11, green on re-run. What is simulated - driving the real recorder with the measured spacing and the poll's phase drawn uniformly from its interval - put it at 6.7% and 10.0% per message here, and 5.7% and 6.7% on an independent reviewer's run that modelled the spacing differently. The order of magnitude is the finding; the digits are not, so every copy in the tree says "a few runs in a hundred".

Deferred findings (from pre-PR self-review and triage)

  • BURST_TAIL is a 4.0.1/d14e fact on a shared class attribute, not on the profile class. ADR-0020 puts what differs by firmware or model on the profile. Deferred: there is one measurement of it, and a --profile run against a Mini or a 4.1 that closes its burst differently now times out and names the message it never saw - which is the evidence that would move it. The constant says so.
  • Nothing asserts the four are the last four. Addressed as observability (tail_positions() in the run report) rather than as an assertion, because the number that would be asserted has been measured on one profile.
  • The 100 ms poll survives; a threading.Event would remove it. The predicate is now a postcondition so there is no correctness gap, and with _seen_types the per-poll cost is O(len(sentinels)). Swapping the wait mechanism means touching __call__, which runs on the RX thread under CLAUDE.md's "the RX thread never dies" rule - not a change to make inside a flake fix.

🤖 Generated with Claude Code

jonathanstokes and others added 4 commits September 14, 2026 15:22
Two hardware tests failed together on 2026-09-11 and passed on a re-run of the
same commit: test_the_connect_burst_warms_the_cache and
test_the_burst_delivered_every_entry_the_preset_surface_reads.

The connect burst closes with four state messages - RecallPreset,
SetlistPosition, PresetDirty, Scene - and they are a GROUP, not a sequence.
Re-measured 2026-09-14 on 4.0.1 / d14e over three connections: they land at
about 11.1 s, spread over 6.0, 3.6 and 5.8 ms, in that order every time.
RecallPreset is the FIRST of them.

The recorder stopped on RecallPreset alone, believing it was the tail, and
polled for it at 100 ms. Its stop therefore landed at a uniformly random point
in the 100 ms after RecallPreset arrived, and about 6 ms of that window is
before the other three have landed. When it fell there the recorder came off
the transport and the fixture took its burst_warmed snapshot of a cache that
had not heard PresetDirty or Scene yet. The cache stays on the transport all
run and got them milliseconds later, which is why only the two tests reading
the snapshot failed and every test reading the live cache passed.

Measured offline against the real recorder with the poll phase randomised the
way the unit randomises it: 300 runs lost PresetDirty 20 times and Scene 30
times, 6.7% and 10.0%. After the change, 0 of 300 for both.

The stop condition is now the postcondition rather than a proxy for it: wait
until every message in BURST_TAIL has arrived. No timeout was widened - the
patience is unchanged at 30 s, and a burst that genuinely does not finish now
reports which messages never came, so the next failure of this kind says
whether the unit stopped sending or the recorder stopped listening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BURST_TAIL and entries.ENTRIES were two hand-written lists about one moment on
the wire, with nothing holding them together: an entry added with a new burst
message would be snapshotted before that message arrived, which is this bug
again with a different name on it. An offline test now makes every entry either
fed by something in BURST_TAIL or named in NOT_WARMED_BY_THE_BURST with the
reason, and there is no third way. The reverse direction is held too - a name
in BURST_TAIL that feeds no entry is a wait paid on every run for nothing.

missing() walked the whole recording under the lock, ten times a second, while
the RX thread wanted that lock to append at ~1490 reports/s - the cost the
_recorded() it replaced said it existed to avoid. The seen type names are now
kept as a set in __call__, so the append stays O(1) and the poll is a set
difference.

record_until now refuses an empty collection as well as a single name: nothing
to wait for is satisfied by the first poll, so the recorder would come off the
transport before the burst began and then report it as finished. Both refusals
close the recorder on the way out - they run with the listener already
attached, and a raise that skipped close() would leave it recording the whole
session with nothing left to stop it.

The unfinished() guard came off the identity test. Identity arrives at 0.7 s
from connect's own Version read, long before the tail, so a cut-off burst would
have failed a test whose evidence was still good.

Also: the whole-tail test compared against BURST_TAIL's declaration order,
which record_until does not use - it builds a frozenset - so correcting that
order would have failed a test for no behavioural reason; the never-recorded
branch of unfinished() had no coverage; the broadcast-listener failure printed
"settled in None s" beside the sentence that already explained it; and the
preset-surface docstring still carried 10 s / 10 ms beside this change's
11.1 s / 3.6-6.0 ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The snapshot race was narrowed, not closed. Listeners run in registration
order and the fixture registered the recorder first, so for the fourth tail
message the recorder counted it and DeviceState.apply_push had not yet run -
the poll could settle, close, and snapshot a cache one message behind. One
__call__ wide instead of six milliseconds, and worse to debug, because
unfinished() truthfully returned None and the test blamed the unit. The cache
is registered first now, which makes "the recorder has all four" mean the cache
has applied them.

record_until tested its deadline before its condition, so a tail landing during
the final sleep was recorded in full and then reported as a burst that never
finished - naming nothing, because by then nothing was missing. It now reads
the condition once per pass and uses that one read for both the settle and the
give-up, so the set it names is the set that was outstanding when it gave up.

unfinished() said the opposite of what it now means. Stopping early is
impossible once the recorder waits for the whole group, which leaves the unit
as the only thing that can make it non-None - and the sentence told the reader
to rule that out.

The drift guard asked whether an entry had ANY feed in BURST_TAIL, so an entry
with one foot in the tail and one in a later burst message passed while
re-opening the race. It is per feed type now, with OUTSIDE_THE_BURST naming the
four feeds that arrive at some other time and why. Both lists are held against
entries.ENTRIES so a stale excuse cannot linger.

Also: record_until refuses message classes, which are neither a str nor empty
and would have burned the whole patience and then named a class as something
the unit never sent; a refused call no longer reports that it was never made;
tail_positions() records how far from the end the group landed, which is the
signal for the one thing the stop condition cannot prove - that these four are
the LAST four; the give-up prints once in the run's own report, since
--verifies or a single-file run deselects all three tests that guard on it; the
settle path asserts detachment like every other exit; the concurrency test
pushes mixed types so it touches _seen_types at all; the PresetDirty assertion
was unreachable behind the guard above it and the surviving one says what is
actually still open.

Records: the measured 6.7% and 10.0% replace a derivation, with the method
written down; the spreads keep the docs' per-session pairing; "fourteen runs in
fifteen" was nine in ten; frozenset("RecallPresetMessage") is eleven distinct
letters; the new paragraph says what its clock is measured from and that 9 to
11.2 s is session variance. tests/hardware/readme.md still described the old
assertions. CLAUDE.md and STEERING.md carry the rule, since tests/ and docs/
are Owned Paths.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fix batch reopened the blank sentence it was written to close, through a
different door. `_called` is set first and the give-up flag last, so anything
that leaves record_until in between - a KeyboardInterrupt during the thirty
second wait is how - left a recorder that had neither settled nor timed out and
fell into the give-up branch, which named nothing and told the reader the unit
was at fault. That state has its own branch now, and pytest prints the string
on an interrupted run, so it is read by the person who just pressed Ctrl-C.

The give-up froze the set read before the deadline test, so the RX thread could
land the last message in between and the report would name something the
recording holds. It re-reads at the deadline, and that one read decides both
branches: empty settles, non-empty is frozen and reported.

The type scan iterated `sentinels` before the frozenset was built, so a
generator argument arrived empty at the next check and was refused for being
empty when it was not.

The rate was quoted three ways and one of them was inverted: "it cost two
hardware tests nine runs in ten" says they failed nine times in ten, where the
measurement says they passed. Worse, the two simulations of it disagree - 6.7%
and 10.0% here, 5.7% and 6.7% from an independent run, differing only in how
the 3.6 to 6.0 ms spacing is modelled. So the digits are not a finding and the
repo no longer prints them: what is observed (one real failure on 2026-09-11,
green on re-run) is now written as observed, what is simulated is written as
simulated, and every copy says "a few runs in a hundred".

Also: the comment block describing NOT_WARMED_BY_THE_BURST had ended up heading
OUTSIDE_THE_BURST, where it stated a rule this branch had already replaced; the
XOR assertion's message lost the half that says which list to edit, which is
the half somebody hits; a test docstring claimed to assert the blank sentence
and asserts the frozen reason instead; the fixture docstring still listed the
recorder before the state layer, one commit after that order was reversed and
called load-bearing; and two places disagreed about whether the burst wait
costs 8 or 9 seconds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jonathanstokes
jonathanstokes marked this pull request as ready for review September 14, 2026 21:55
@jonathanstokes
jonathanstokes merged commit 6b996e7 into main Sep 14, 2026
5 checks passed
@jonathanstokes
jonathanstokes deleted the fix/connect-burst-recorder-race branch September 14, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant