diff --git a/CLAUDE.md b/CLAUDE.md index 2c5be738..cae9d4f1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -86,6 +86,11 @@ small documentation fixes. ### The model never guesses +- A state-cache read clears its reread mark only when concurrent messages + restate its answer exactly. `FieldPlan.accepts` separates independent + conversations sharing a protobuf type; it never suppresses fields within one + conversation. See ADR-0011. + - The model shows what the unit shows, in the unit's words. A control we understand but cannot drive is modelled and refuses with `protocol.ControlNotDrivable(control, evidence, workaround)`, all three fields diff --git a/changelog.md b/changelog.md index 96a5af23..cef212f3 100644 --- a/changelog.md +++ b/changelog.md @@ -15,6 +15,14 @@ sustained stretch without a correction. ## Unreleased +### Live-preset reads tolerate the device's lazy two-message reply + +`read_current_preset()` and `read_current_preset_push()` now make a second +attempt when the first keyed reply times out. CorOS 4.1.0 sends an uncorrelated +current-preset update before the request-id-correlated answer; the state cache +models both as one normal read and no longer schedules an unnecessary reread. +Version messages that only answer the Cortex Control compatibility announcement +are likewise kept out of the device-identity cache. ### Rename the unit, drive undo/redo, and read inhibited modules `set_device_name()` sends a sparse Version update. `undo()` and `redo()` drive diff --git a/docs/ADR.md b/docs/ADR.md index e6b05a80..47f8c33c 100644 --- a/docs/ADR.md +++ b/docs/ADR.md @@ -186,6 +186,8 @@ Records are append-only once `Decided` and built upon: a shipped decision is nev - `action` and `request_id` are skipped on every entry, and only one of them is really the transport's. `request_id` always is. `action` is not: on `Grid` it is state that matters, because an `UPDATE` carrying `hash: 0` is transmitted and ignored while the same payload with `action: DELETE` removes the block. It is skipped today because the two tracked message types give it no meaning, and a `Grid` entry (issue #12) therefore cannot inherit the skip - two pushes with identical payloads and opposite meanings would apply identically and mark nothing. That entry gives `action` its own decision rather than widening the shared set. - A field the wire gives no presence and the entry does not keep is undetectable rather than merely unkept: proto3 writes such a field only when it differs from its default, so a message leaving one at its default carries no bytes for it and no implementation of this check could see it. `tests/test_state.py` asserts no feeding type has one, which turns a limit of the wire into a question about our own code, checkable and checked. - The arrival count is necessary and not sufficient: the read path counts marks as well. The count answers "did anything arrive for this entry beyond my own reply", which says nothing about a mark set by a path that sends no message for that entry at all - and there are two, both reproduced offline on 2026-08-27. A recall marks `dirty` through `LOADED.resets`, and the unit sends no `PresetDirty` after a recall, so nothing was ever coming to correct it; the write watchdog marks from its own thread when no echo arrives, and the answer to a read already in flight may have been composed before the write reached the unit. A read discarded both, the `dirty` window being the 2-11 ms `preset_dirty()` read. So every path that makes an entry untrusted for a reason other than a message for that entry goes through `mark_for_reread`, which counts, and the read snapshots that counter alongside the arrival count and clears only the marks it can have answered. What does not work is `needs_read or extra > 1`, and a test catches it: the read's own reply re-arms the mark through the listener from the fields the entry does not keep, which is the entry-caches-nothing failure this record's Context already names. + - CorOS 4.1.0 emits an unkeyed live-preset update immediately before the keyed answer. The cache discounts extra arrivals only when their applied fields equal the answer; a genuine recall carries a different preset and preserves the mark. + - `FieldPlan.accepts` is reserved for independent conversations sharing one protobuf type. `Version` uses it to keep host-handshake validity replies out of the device-identity entry; accepted messages still receive the complete unknown-field check. ## ADR-0012: A grid push is noted and re-read, not merged, and the model publishes what it noticed diff --git a/docs/protocol.md b/docs/protocol.md index c75a37eb..9a5a261a 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -585,7 +585,10 @@ not measured. `RecallReason` names them. **The live grid is readable without a recall.** `RecallPreset{READ, request_id}` answers with the preset as it is right now, unsaved edits included, with no side effects: the unsaved edit survives and the active scene does not move. -`read_current_preset()` wraps it. `Scene{READ, request_id}` answers with +CorOS 4.1.0 was measured sending an uncorrelated current-state update before +the keyed reply, and its first request after connecting can be dropped. +`read_current_preset()` therefore makes at most two attempts within the caller's +single timeout and waits for the keyed answer. `Scene{READ, request_id}` answers with `selected_scene`; `active_scene()` wraps it. `SetlistPosition{READ, request_id}` answers with `folder_key`, `position` and `is_factory` in 3 ms (measured 2026-08-15); `loaded_position()` wraps it. A `SetlistPosition` read must never @@ -2122,7 +2125,7 @@ wire, with no independent read-back. | `inhibited_modules` | `CompilerInhibitedModules{READ}` | read-back | explicit false/false reply on CorOS 4.0.1 and 4.1.0; true semantics are schema-derived, not yet observed | | `create_local_backup` | `LocalBackup{CREATE}` then `LocalBackup{UPDATE, backup_json}` pushes, the last with `is_last_chunk` | captured only | section 10.5. `can_apply_backup` never appeared, so the refusal path is unverified | | `recall_preset` / `read_preset` | `SetlistPosition{UPDATE, folder_key, position, is_factory, request_id}` then a `RecallPreset` push | read-back | the push echoes the recall's `request_id` | -| `read_current_preset` / `read_current_preset_push` | `RecallPreset{READ, request_id}` | read-back | the live grid, no side effects. The push variant hands back the whole reply with `reason` | +| `read_current_preset` / `read_current_preset_push` | `RecallPreset{READ, request_id}` | read-back | the live grid, no side effects. Two bounded attempts share the caller's timeout; CorOS 4.1.0 was measured emitting an unkeyed duplicate before the keyed reply. The push variant hands back the whole reply with `reason` | | `loaded_position` | `SetlistPosition{READ, request_id}` | read-back | which slot is loaded; 3 ms measured | | `list_presets` | `File{action: READ}` then `File{folder{files[] = ProductData}}` | read-back | factory listing gzipped; 256 slots; listings lag after a `File` mutation | | `switch_scene` | `Scene{UPDATE, selected_scene}` | on-unit | zero-based | diff --git a/pyquadcortex/device/entries.py b/pyquadcortex/device/entries.py index afa037f0..73a621a7 100644 --- a/pyquadcortex/device/entries.py +++ b/pyquadcortex/device/entries.py @@ -84,11 +84,15 @@ class FieldPlan: either has to decide for itself, and this flag is that decision written down. It does not widen :data:`SCAFFOLDING`, and it is set per entry and per message type rather than globally. + accepts: optional predicate for message types reused by independent + conversations. A rejected message says nothing about this entry; + it is neither applied nor counted as an arrival. """ kept: frozenset = frozenset() no_presence: frozenset = frozenset() invalidates: bool = False + accepts: typing.Callable[[Message], bool] | None = None def voids_the_copy(self) -> bool: """Whether a message of this type makes the entry untrusted on its own. @@ -99,6 +103,10 @@ def voids_the_copy(self) -> bool: """ return self.invalidates + def applies_to(self, message: Message) -> bool: + """Whether this same-type message belongs to the entry's conversation.""" + return self.accepts is None or self.accepts(message) + @dataclasses.dataclass(frozen=True, eq=False) class StateEntry: @@ -110,7 +118,6 @@ class StateEntry: the unit's whole answer for this entry. Runs on the CALLER's thread, never the RX thread. feeds: message class -> :class:`FieldPlan`. - Every entry's :attr:`read` is one request and one ANSWER, which the read path relies on to tell its own answer apart from a push that arrived while it was waiting. One answer is not the same as one message: a ``Version`` @@ -242,6 +249,13 @@ def _carries_unknown_fields(message) -> bool: # -- the entries ------------------------------------------------------------- +def _version_describes_device(message) -> bool: + """Exclude the two Version shapes that are only handshake traffic.""" + named = {field.name for field, _ in message.ListFields()} + host_handshake = SCAFFOLDING | {"cortex_control_version_valid"} + return bool(named - host_handshake) or _carries_unknown_fields(message) + + #: The unit's identity. Firmware and serial cannot change while a connection is #: up: the only thing that changes either is a firmware update, and the firmware #: `Updater` surface is permanently out of scope for this library (repo-root @@ -269,9 +283,13 @@ def _carries_unknown_fields(message) -> bool: #: fields followed 0.5-0.8 ms later by a ``Version{READ}`` carrying ``action`` #: alone. The question is not news about the unit and the cache does not count #: it - see ``_apply_one`` in ``device/state.py``, where counting it cost a -#: second round trip whenever it landed before the reading thread woke. +#: second round trip whenever it landed before the reading thread woke. The +#: device's answer to the host's Cortex Control announcement is also a Version, +#: but it carries only handshake validity and likewise says nothing about the +#: unit's identity. _VERSION_FOR_IDENTITY = FieldPlan( kept=frozenset({"app_fw_version", "device_serial_number"}), + accepts=_version_describes_device, ) @@ -500,13 +518,9 @@ def _read_scene(client) -> dict: #: a plan, not a fact, and every push mentioning a field it did not keep would #: mark it for a read nobody had asked for. #: -#: The Directory's rows are the ones that need something this class does not -#: have. Every read here is one request and one answer, which is how the read path -#: tells its own answer apart from a push that arrived while it was waiting. A -#: setlist listing is a STREAM - one `File` READ makes the unit enumerate its -#: whole tree, several hundred messages over about fifteen seconds - so those -#: entries land with the change to `StateEntry` that lets a read say how many -#: messages it expects. +#: A future entry whose read provokes a stream of answers must declare how that +#: stream completes. No directory entry exists yet; issue #25 still owns that +#: design decision. ENTRIES = (IDENTITY, DIRTY, PRESET, SCENE, LOADED) ENTRY_BY_NAME = {entry.name: entry for entry in ENTRIES} diff --git a/pyquadcortex/device/state.py b/pyquadcortex/device/state.py index 11cae181..a351ff1c 100644 --- a/pyquadcortex/device/state.py +++ b/pyquadcortex/device/state.py @@ -54,8 +54,8 @@ class _Slot: """One entry's copy of the unit's state, and how much we trust it.""" - __slots__ = ("fields", "needs_read", "witnessed", "marks", "_arrivals", - "_marks") + __slots__ = ("fields", "needs_read", "witnessed", "marks", "observed", + "_arrivals", "_marks") def __init__(self): #: field name -> value, holding only what the unit has actually said. @@ -89,6 +89,11 @@ def __init__(self): #: rule. Do not simplify this back. self._arrivals = itertools.count(1) self.witnessed = 0 + #: Fields applied by meaningful messages since the current read began. + #: The read path uses these to distinguish CorOS 4.1's exact duplicate + #: live-preset reply from a genuine recall that happens in the same + #: window. Cleared under the slot lock immediately before each read. + self.observed = [] #: How many times a path OTHER than the listener has said this copy #: cannot be trusted: `mark_for_reread` and everything that calls it - #: a recall's `resets`, the write watchdog giving up, a caller of its @@ -213,6 +218,8 @@ def apply_push(self, message) -> None: from this thread outright (ADR-0009). """ for entry, plan in entries.FEEDS.get(type(message), ()): + if not plan.applies_to(message): + continue try: self._apply_one(entry, plan, message) except Exception: @@ -238,6 +245,7 @@ def _apply_one(self, entry, plan, message) -> None: # unit's own `Version{READ}` is exactly that message - see # :class:`_Slot`. slot.arrived() + slot.observed.append((dict(applied), why)) was_empty = not slot.fields # Worked out BEFORE the update, and against the values we held, # because a caller tracking changes wants the ones that are @@ -393,6 +401,7 @@ def value(self, entry_name: str, field: str): return slot.fields[field] witnessed_before = slot.witnessed marks_before = slot.marks + slot.observed = [] client = self._client if client is None: raise RuntimeError( @@ -420,10 +429,17 @@ def value(self, entry_name: str, field: str): # than one message: the unit answers a `Version` READ and then # asks one of its own, and a question is not news about the # unit's state. `_apply_one` is where that is decided, because - # the read path cannot tell the two apart after the fact. An - # entry whose read provokes a stream of ANSWERS will still have - # to say how many messages that is; see `StateEntry`. + # the read path cannot tell the two apart after the fact. extra = slot.witnessed - witnessed_before + observations = tuple(slot.observed) + # CorOS 4.1.0 was measured returning an unkeyed RecallPreset + # immediately before the keyed answer. It is safe to discount + # that second arrival only when every arrival restates the + # fields in the answer exactly. A real recall in this window + # carries a different preset and therefore keeps the mark. + restated_answer = all(applied == answer and why is None + for applied, why in observations) + conflicting_push = extra > 1 and not restated_answer # And the marks that came from something OTHER than a # message to this entry, which the count above cannot see # however carefully it counts: a recall's `resets`, the @@ -437,16 +453,18 @@ def value(self, entry_name: str, field: str): # snapshot and the request going out is kept although the unit # might have accounted for it. `witnessed_before` has had the # same gap since it was written, and both err by re-reading. The obvious - # `slot.needs_read or extra > 1` does not work: the read's own + # `slot.needs_read or conflicting_push` does not + # work: the read's own # reply re-arms the mark through the listener from the fields # this entry does not keep, so the entry would never cache # anything again. marked_since = slot.marks - marks_before was_marked = slot.needs_read - slot.needs_read = extra > 1 or marked_since > 0 - if extra > 1: + slot.needs_read = conflicting_push or marked_since > 0 + if conflicting_push: log.info("push.forced_reread %s - %d message(s) arrived " - "while it was being read", entry_name, extra) + "while it was being read and did not all restate " + "its answer", entry_name, extra) elif slot.needs_read: # The mirror of the line below, at info rather than its # debug: a surviving mark costs a real round trip, where a diff --git a/pyquadcortex/protocol/client.py b/pyquadcortex/protocol/client.py index 39f72982..777c3ac8 100644 --- a/pyquadcortex/protocol/client.py +++ b/pyquadcortex/protocol/client.py @@ -1611,7 +1611,7 @@ def preset_dirty(self, timeout: float = 5.0) -> bool: pa.PresetDirtyMessage(action=pa.MessageAction.READ), timeout=timeout) return bool(reply.is_dirty) - def read_current_preset(self, timeout: float = 15.0): + def read_current_preset(self, timeout: float = 15.0, attempts: int = 2): """The LIVE grid - the current editing state, unsaved changes included. ``RecallPreset{READ}`` answers with the preset as it exists on the device @@ -1634,10 +1634,17 @@ def read_current_preset(self, timeout: float = 15.0): every time** - even when it recalls the preset already loaded. Interleaving it with scene-targeted writes silently retargets them; use this method for inspection during editing. + + On CorOS 4.1.0, measured 2026-09-11, the first request after a + connection can be dropped. ``attempts`` deliberately defaults to two + on every profile as a bounded resilience measure; ``timeout`` is the + total budget and is divided between attempts. """ - return self.read_current_preset_push(timeout=timeout).preset + return self.read_current_preset_push( + timeout=timeout, attempts=attempts).preset - def read_current_preset_push(self, timeout: float = 15.0): + def read_current_preset_push(self, timeout: float = 15.0, + attempts: int = 2): """The whole ``RecallPreset`` reply, not just the preset inside it. Same request and same match as :meth:`read_current_preset` - this is @@ -1648,15 +1655,32 @@ def read_current_preset_push(self, timeout: float = 15.0): It exists because the reply carries ``reason`` beside the preset, and a caller tracking state needs both from one answer. Confirmed on hardware 2026-08-15: the connect burst's seed push sets ``action``, ``preset`` - and ``reason``, and so does the push a recall produces. + and ``reason``, and so does the push a recall produces. Re-measured on + CorOS 4.1.0 on 2026-09-11: a READ emits an uncorrelated copy before the + request-id-correlated answer, and its first request can be dropped. """ - request_id = self._t.next_request_id() - message = pa.RecallPresetMessage(action=pa.MessageAction.READ, - request_id=request_id) - return self._t.await_broadcast( - pa.RecallPresetMessage, lambda: self._t.send(message), timeout=timeout, - match=lambda m: (m.HasField("request_id") - and m.request_id == request_id)) + last = None + tries = max(1, attempts) + attempt_timeout = timeout / tries + for attempt in range(1, tries + 1): + request_id = self._t.next_request_id() + message = pa.RecallPresetMessage(action=pa.MessageAction.READ, + request_id=request_id) + try: + return self._t.await_broadcast( + pa.RecallPresetMessage, + lambda outgoing=message: self._t.send(outgoing), + timeout=attempt_timeout, + match=lambda m, wanted=request_id: ( + m.HasField("request_id") and m.request_id == wanted)) + except TimeoutError as exc: + last = exc + log.warning("live-preset read attempt %d/%d timed out after " + "%ss", attempt, tries, attempt_timeout) + raise TimeoutError( + f"the device did not answer a live-preset read in {tries} " + f"attempt(s) within {timeout}s total; CorOS 4.1.0 was measured " + f"dropping the first request after connecting") from last def loaded_position(self, timeout: float = 10.0): """Which preset slot is on the grid right now. diff --git a/tests/test_client.py b/tests/test_client.py index 5b35edaa..8ca16c0a 100644 --- a/tests/test_client.py +++ b/tests/test_client.py @@ -1605,6 +1605,65 @@ def test_read_current_preset_push_hands_back_the_whole_reply(): assert asked.HasField("request_id") +def test_read_current_preset_retries_a_dropped_first_request(): + push = pa.RecallPresetMessage(action=pa.MessageAction.UPDATE, request_id=2) + push.preset.name = "live after retry" + + class DropsFirst(StateTransport): + def __init__(self, reply): + super().__init__(reply) + self.calls = 0 + + def await_broadcast(self, expected_class, trigger, timeout=40.0, + match=None): + trigger() + self.calls += 1 + if self.calls == 1: + raise TimeoutError("dropped") + self.matches.append(match) + return self.push + + transport = DropsFirst(push) + got = client.QuadCortex(transport).read_current_preset() + assert got.name == "live after retry" + assert transport.calls == 2 + assert [m.request_id for m in transport.sent] == [1, 2] + assert transport.matches[-1](push) is True + assert transport.matches[-1]( + pa.RecallPresetMessage(request_id=1)) is False + + +def test_read_current_preset_one_attempt_sends_once_and_raises(): + cause = TimeoutError("one miss") + + class NeverAnswers(FakeTransport): + def await_broadcast(self, expected_class, trigger, timeout=40.0, + match=None): + trigger() + raise cause + + qc = client.QuadCortex(NeverAnswers()) + with pytest.raises(TimeoutError, match=r"1 attempt\(s\)") as raised: + qc.read_current_preset(timeout=0.25, attempts=1) + assert raised.value.__cause__ is cause + assert len(qc._t.sent) == 1 + + +def test_read_current_preset_timeout_reports_all_attempts(): + class NeverAnswers(FakeTransport): + def await_broadcast(self, expected_class, trigger, timeout=40.0, + match=None): + trigger() + raise TimeoutError("nothing") + + qc = client.QuadCortex(NeverAnswers()) + with pytest.raises(TimeoutError, match=r"2 attempt\(s\) within 0.01s total") \ + as raised: + qc.read_current_preset(timeout=0.01) + assert [m.request_id for m in qc._t.sent] == [1, 2] + assert isinstance(raised.value.__cause__, TimeoutError) + + def test_loaded_position_reads_the_slot_without_recalling_it(): """`SetlistPosition{READ}`. The same message type as a recall, and the action is the whole difference between asking and loading - so the wire diff --git a/tests/test_state.py b/tests/test_state.py index c08c17b4..fb2bd4a8 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -494,6 +494,20 @@ def answers_and_then_asks_back(): assert cache.needs_read("identity") is False +def test_the_cortex_control_answer_is_not_device_identity(link): + """The Version response to our host announce carries no device fact.""" + transport, cache = link + assert cache.value("identity", "app_fw_version") == "d14e" + + transport.push(pa.VersionMessage( + action=pa.MessageAction.UPDATE, request_id=0, + cortex_control_version_valid=True)) + + assert cache.value("identity", "app_fw_version") == "d14e" + assert cache.needs_read("identity") is False + assert transport.reads["VersionMessage"] == 1 + + def test_a_recall_landing_during_a_read_of_the_dirty_flag_is_not_lost(link): """The same window, reached by a path the arrival count cannot see. @@ -1375,6 +1389,43 @@ def test_the_preset_entry_can_read_back_every_field_it_keeps(link): assert cache.value("preset", "preset").name == "Structural Fixture" +def test_two_normal_preset_pushes_still_settle_one_read(link): + """An exact uncorrelated restatement cannot make the cache stale.""" + transport, cache = link + + def uncorrelated_then_keyed(triggering): + transport.push(recall_push()) + return recall_push(triggering) + + transport.broadcasts["RecallPresetMessage"] = uncorrelated_then_keyed + cache.mark_for_reread("preset", "exercise the measured two-push reply") + + assert cache.value("preset", "preset").name == "Structural Fixture" + assert cache.needs_read("preset") is False + assert transport.reads["RecallPresetMessage"] == 1 + + +def test_a_different_recall_after_the_read_answer_keeps_the_mark(link): + """A real recall must not be mistaken for CorOS 4.1's duplicate answer.""" + transport, cache = link + + answer = recall_push() + + class RecallDuringRead: + def read_current_preset_push(self): + transport.push(answer) + recalled = recall_push() + recalled.preset.name = "newer preset" + transport.push(recalled) + return answer + + cache.bind(RecallDuringRead()) + cache.mark_for_reread("preset", "exercise a concurrent genuine recall") + + assert cache.value("preset", "preset").name == "Structural Fixture" + assert cache.needs_read("preset") is True + + def test_a_recall_push_does_not_invalidate_the_preset_it_delivers(): plan = entries.PRESET.feeds[pa.RecallPresetMessage] assert not plan.voids_the_copy()