Repository navigation
fix(hardware-suite): the burst recorder waits for the whole tail, not its first message - #68
Merged
Merged
Conversation
Two hardware tests failed together on 2026-09-11 and passed on a re-run of the same commit: test_the_connect_burst_warms_the_cache and test_the_burst_delivered_every_entry_the_preset_surface_reads. The connect burst closes with four state messages - RecallPreset, SetlistPosition, PresetDirty, Scene - and they are a GROUP, not a sequence. Re-measured 2026-09-14 on 4.0.1 / d14e over three connections: they land at about 11.1 s, spread over 6.0, 3.6 and 5.8 ms, in that order every time. RecallPreset is the FIRST of them. The recorder stopped on RecallPreset alone, believing it was the tail, and polled for it at 100 ms. Its stop therefore landed at a uniformly random point in the 100 ms after RecallPreset arrived, and about 6 ms of that window is before the other three have landed. When it fell there the recorder came off the transport and the fixture took its burst_warmed snapshot of a cache that had not heard PresetDirty or Scene yet. The cache stays on the transport all run and got them milliseconds later, which is why only the two tests reading the snapshot failed and every test reading the live cache passed. Measured offline against the real recorder with the poll phase randomised the way the unit randomises it: 300 runs lost PresetDirty 20 times and Scene 30 times, 6.7% and 10.0%. After the change, 0 of 300 for both. The stop condition is now the postcondition rather than a proxy for it: wait until every message in BURST_TAIL has arrived. No timeout was widened - the patience is unchanged at 30 s, and a burst that genuinely does not finish now reports which messages never came, so the next failure of this kind says whether the unit stopped sending or the recorder stopped listening. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BURST_TAIL and entries.ENTRIES were two hand-written lists about one moment on the wire, with nothing holding them together: an entry added with a new burst message would be snapshotted before that message arrived, which is this bug again with a different name on it. An offline test now makes every entry either fed by something in BURST_TAIL or named in NOT_WARMED_BY_THE_BURST with the reason, and there is no third way. The reverse direction is held too - a name in BURST_TAIL that feeds no entry is a wait paid on every run for nothing. missing() walked the whole recording under the lock, ten times a second, while the RX thread wanted that lock to append at ~1490 reports/s - the cost the _recorded() it replaced said it existed to avoid. The seen type names are now kept as a set in __call__, so the append stays O(1) and the poll is a set difference. record_until now refuses an empty collection as well as a single name: nothing to wait for is satisfied by the first poll, so the recorder would come off the transport before the burst began and then report it as finished. Both refusals close the recorder on the way out - they run with the listener already attached, and a raise that skipped close() would leave it recording the whole session with nothing left to stop it. The unfinished() guard came off the identity test. Identity arrives at 0.7 s from connect's own Version read, long before the tail, so a cut-off burst would have failed a test whose evidence was still good. Also: the whole-tail test compared against BURST_TAIL's declaration order, which record_until does not use - it builds a frozenset - so correcting that order would have failed a test for no behavioural reason; the never-recorded branch of unfinished() had no coverage; the broadcast-listener failure printed "settled in None s" beside the sentence that already explained it; and the preset-surface docstring still carried 10 s / 10 ms beside this change's 11.1 s / 3.6-6.0 ms. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The snapshot race was narrowed, not closed. Listeners run in registration
order and the fixture registered the recorder first, so for the fourth tail
message the recorder counted it and DeviceState.apply_push had not yet run -
the poll could settle, close, and snapshot a cache one message behind. One
__call__ wide instead of six milliseconds, and worse to debug, because
unfinished() truthfully returned None and the test blamed the unit. The cache
is registered first now, which makes "the recorder has all four" mean the cache
has applied them.
record_until tested its deadline before its condition, so a tail landing during
the final sleep was recorded in full and then reported as a burst that never
finished - naming nothing, because by then nothing was missing. It now reads
the condition once per pass and uses that one read for both the settle and the
give-up, so the set it names is the set that was outstanding when it gave up.
unfinished() said the opposite of what it now means. Stopping early is
impossible once the recorder waits for the whole group, which leaves the unit
as the only thing that can make it non-None - and the sentence told the reader
to rule that out.
The drift guard asked whether an entry had ANY feed in BURST_TAIL, so an entry
with one foot in the tail and one in a later burst message passed while
re-opening the race. It is per feed type now, with OUTSIDE_THE_BURST naming the
four feeds that arrive at some other time and why. Both lists are held against
entries.ENTRIES so a stale excuse cannot linger.
Also: record_until refuses message classes, which are neither a str nor empty
and would have burned the whole patience and then named a class as something
the unit never sent; a refused call no longer reports that it was never made;
tail_positions() records how far from the end the group landed, which is the
signal for the one thing the stop condition cannot prove - that these four are
the LAST four; the give-up prints once in the run's own report, since
--verifies or a single-file run deselects all three tests that guard on it; the
settle path asserts detachment like every other exit; the concurrency test
pushes mixed types so it touches _seen_types at all; the PresetDirty assertion
was unreachable behind the guard above it and the surviving one says what is
actually still open.
Records: the measured 6.7% and 10.0% replace a derivation, with the method
written down; the spreads keep the docs' per-session pairing; "fourteen runs in
fifteen" was nine in ten; frozenset("RecallPresetMessage") is eleven distinct
letters; the new paragraph says what its clock is measured from and that 9 to
11.2 s is session variance. tests/hardware/readme.md still described the old
assertions. CLAUDE.md and STEERING.md carry the rule, since tests/ and docs/
are Owned Paths.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fix batch reopened the blank sentence it was written to close, through a different door. `_called` is set first and the give-up flag last, so anything that leaves record_until in between - a KeyboardInterrupt during the thirty second wait is how - left a recorder that had neither settled nor timed out and fell into the give-up branch, which named nothing and told the reader the unit was at fault. That state has its own branch now, and pytest prints the string on an interrupted run, so it is read by the person who just pressed Ctrl-C. The give-up froze the set read before the deadline test, so the RX thread could land the last message in between and the report would name something the recording holds. It re-reads at the deadline, and that one read decides both branches: empty settles, non-empty is frozen and reported. The type scan iterated `sentinels` before the frozenset was built, so a generator argument arrived empty at the next check and was refused for being empty when it was not. The rate was quoted three ways and one of them was inverted: "it cost two hardware tests nine runs in ten" says they failed nine times in ten, where the measurement says they passed. Worse, the two simulations of it disagree - 6.7% and 10.0% here, 5.7% and 6.7% from an independent run, differing only in how the 3.6 to 6.0 ms spacing is modelled. So the digits are not a finding and the repo no longer prints them: what is observed (one real failure on 2026-09-11, green on re-run) is now written as observed, what is simulated is written as simulated, and every copy says "a few runs in a hundred". Also: the comment block describing NOT_WARMED_BY_THE_BURST had ended up heading OUTSIDE_THE_BURST, where it stated a rule this branch had already replaced; the XOR assertion's message lost the half that says which list to edit, which is the half somebody hits; a test docstring claimed to assert the blank sentence and asserts the frozen reason instead; the fixture docstring still listed the recorder before the state layer, one commit after that order was reversed and called load-bearing; and two places disagreed about whether the burst wait costs 8 or 9 seconds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jonathanstokes
marked this pull request as ready for review
September 14, 2026 21:55
…order-race # Conflicts: # docs/STEERING.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two hardware tests failed together on 2026-09-11 and passed on a re-run of the same commit -
test_the_connect_burst_warms_the_cacheandtest_the_burst_delivered_every_entry_the_preset_surface_reads. This is a genuine race in the suite's burst recorder, not an under-specified wait, and no timeout was widened to fix it.The connect burst closes with four state messages -
RecallPreset,SetlistPosition,PresetDirty,Scene- and they are a group, not a sequence. Re-measured 2026-09-14 on 4.0.1 / d14e over three connections: they land at about 11.1 s, spread over 6.0, 3.6 and 5.8 ms, in that order every time.RecallPresetis the first of them.record_untilstopped onRecallPresetalone, believing it was the tail, and polled for it at 100 ms. Its stop therefore landed at a uniformly random point in the 100 ms afterRecallPresetarrived, and a few milliseconds of that window are before the other three have landed. When it fell there the recorder came off the transport and the fixture took itsburst_warmedsnapshot of a cache that had not yet heardPresetDirtyorScene. The cache stays on the transport all run and got them milliseconds later - which is why only the two tests reading the snapshot failed, and every test reading the live cache passed.The stop condition is now the postcondition rather than a proxy for it: wait until every message in
BURST_TAILhas arrived. Patience is unchanged at 30 s.Changes
record_until(sentinels, patience)waits for every type in the collection. It refuses a single name (astris a collection of its letters), an empty collection (satisfied by the first poll), and message classes (which match nothing the recorder stores). All refusals detach on the way out.apply_pushhad run for it - the same stale snapshot, one__call__wide instead of six milliseconds, and harder to spot becauseunfinished()would truthfully returnNone.unfinished()explains a cut-off recording in words, and prints once in the run's own report as well as in the tests that guard on it ---verifiesor a single-file run deselects all of those while the fixture still waits. It distinguishes settled, timed out, interrupted, refused, and never-called, because they want different responses._seen_typesis maintained in__call__, so the poll is a set difference instead of a walk of the whole recording under the lock the RX thread needs.tail_positions()records how far from the end of the recording each closing message landed. The stop condition proves the four arrived; it cannot prove they are the last four, which is what the snapshot actually rests on.BURST_TAILagainstentries.ENTRIESper feed type, so a cache entry added with a new burst message cannot silently re-open this. A new entry comes throughBURST_TAIL,OUTSIDE_THE_BURSTorNOT_WARMED_BY_THE_BURST, each with its reason.docs/protocol.md,tests/hardware/readme.md,CLAUDE.mdandSTEERING.mdrecord the measurement and the rule.Nothing under
pyquadcortex/changed. The bug was in how the suite decided the burst was over, not in the library's handling of it.Test plan
test_record_until_waits_for_the_whole_tail_not_just_the_first_of_itstages the gap three poll intervals wide, so it does not depend on winning a race. An independent reviewer mutated the recorder to the old behaviour and confirmed it fails.pytest tests/hardware --hardwareon the final commitHardware run
Commit
45db700ca2a3253fcc80c3d66a41b40efd326299(main merged in), Quad Cortex CorOS 4.0.1, app_fw d14e.The new
connect burst: cut offsection did not print, which is the burst reporting that it settled. Run on the final commit after main was merged in, so it coverscreate_local_backupand the rest of PR #47 as they stand on this branch. Earlier green runs:83ae721at 109 passed,89aa01cat 100 passed.On the flake rate
The repo does not quote a percentage for it, deliberately. What is observed is one real failure on 2026-09-11, green on re-run. What is simulated - driving the real recorder with the measured spacing and the poll's phase drawn uniformly from its interval - put it at 6.7% and 10.0% per message here, and 5.7% and 6.7% on an independent reviewer's run that modelled the spacing differently. The order of magnitude is the finding; the digits are not, so every copy in the tree says "a few runs in a hundred".
Deferred findings (from pre-PR self-review and triage)
BURST_TAILis a 4.0.1/d14e fact on a shared class attribute, not on the profile class. ADR-0020 puts what differs by firmware or model on the profile. Deferred: there is one measurement of it, and a--profilerun against a Mini or a 4.1 that closes its burst differently now times out and names the message it never saw - which is the evidence that would move it. The constant says so.tail_positions()in the run report) rather than as an assertion, because the number that would be asserted has been measured on one profile.threading.Eventwould remove it. The predicate is now a postcondition so there is no correctness gap, and with_seen_typesthe per-poll cost is O(len(sentinels)). Swapping the wait mechanism means touching__call__, which runs on the RX thread under CLAUDE.md's "the RX thread never dies" rule - not a change to make inside a flake fix.🤖 Generated with Claude Code