Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,11 @@ small documentation fixes.

### The model never guesses

- A state-cache read clears its reread mark only when concurrent messages
restate its answer exactly. `FieldPlan.accepts` separates independent
conversations sharing a protobuf type; it never suppresses fields within one
conversation. See ADR-0011.

- The model shows what the unit shows, in the unit's words. A control we
understand but cannot drive is modelled and refuses with
`protocol.ControlNotDrivable(control, evidence, workaround)`, all three fields
Expand Down
8 changes: 8 additions & 0 deletions changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,14 @@ sustained stretch without a correction.

## Unreleased

### Live-preset reads tolerate the device's lazy two-message reply

`read_current_preset()` and `read_current_preset_push()` now make a second
attempt when the first keyed reply times out. CorOS 4.1.0 sends an uncorrelated
current-preset update before the request-id-correlated answer; the state cache
models both as one normal read and no longer schedules an unnecessary reread.
Version messages that only answer the Cortex Control compatibility announcement
are likewise kept out of the device-identity cache.
### Rename the unit, drive undo/redo, and read inhibited modules

`set_device_name()` sends a sparse Version update. `undo()` and `redo()` drive
Expand Down
2 changes: 2 additions & 0 deletions docs/ADR.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,8 @@ Records are append-only once `Decided` and built upon: a shipped decision is nev
- `action` and `request_id` are skipped on every entry, and only one of them is really the transport's. `request_id` always is. `action` is not: on `Grid` it is state that matters, because an `UPDATE` carrying `hash: 0` is transmitted and ignored while the same payload with `action: DELETE` removes the block. It is skipped today because the two tracked message types give it no meaning, and a `Grid` entry (issue #12) therefore cannot inherit the skip - two pushes with identical payloads and opposite meanings would apply identically and mark nothing. That entry gives `action` its own decision rather than widening the shared set.
- A field the wire gives no presence and the entry does not keep is undetectable rather than merely unkept: proto3 writes such a field only when it differs from its default, so a message leaving one at its default carries no bytes for it and no implementation of this check could see it. `tests/test_state.py` asserts no feeding type has one, which turns a limit of the wire into a question about our own code, checkable and checked.
- The arrival count is necessary and not sufficient: the read path counts marks as well. The count answers "did anything arrive for this entry beyond my own reply", which says nothing about a mark set by a path that sends no message for that entry at all - and there are two, both reproduced offline on 2026-08-27. A recall marks `dirty` through `LOADED.resets`, and the unit sends no `PresetDirty` after a recall, so nothing was ever coming to correct it; the write watchdog marks from its own thread when no echo arrives, and the answer to a read already in flight may have been composed before the write reached the unit. A read discarded both, the `dirty` window being the 2-11 ms `preset_dirty()` read. So every path that makes an entry untrusted for a reason other than a message for that entry goes through `mark_for_reread`, which counts, and the read snapshots that counter alongside the arrival count and clears only the marks it can have answered. What does not work is `needs_read or extra > 1`, and a test catches it: the read's own reply re-arms the mark through the listener from the fields the entry does not keep, which is the entry-caches-nothing failure this record's Context already names.
- CorOS 4.1.0 emits an unkeyed live-preset update immediately before the keyed answer. The cache discounts extra arrivals only when their applied fields equal the answer; a genuine recall carries a different preset and preserves the mark.
- `FieldPlan.accepts` is reserved for independent conversations sharing one protobuf type. `Version` uses it to keep host-handshake validity replies out of the device-identity entry; accepted messages still receive the complete unknown-field check.

## ADR-0012: A grid push is noted and re-read, not merged, and the model publishes what it noticed

Expand Down
7 changes: 5 additions & 2 deletions docs/protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -585,7 +585,10 @@ not measured. `RecallReason` names them.
**The live grid is readable without a recall.** `RecallPreset{READ, request_id}`
answers with the preset as it is right now, unsaved edits included, with no side
effects: the unsaved edit survives and the active scene does not move.
`read_current_preset()` wraps it. `Scene{READ, request_id}` answers with
CorOS 4.1.0 was measured sending an uncorrelated current-state update before
the keyed reply, and its first request after connecting can be dropped.
`read_current_preset()` therefore makes at most two attempts within the caller's
single timeout and waits for the keyed answer. `Scene{READ, request_id}` answers with
`selected_scene`; `active_scene()` wraps it. `SetlistPosition{READ, request_id}`
answers with `folder_key`, `position` and `is_factory` in 3 ms (measured
2026-08-15); `loaded_position()` wraps it. A `SetlistPosition` read must never
Expand Down Expand Up @@ -2122,7 +2125,7 @@ wire, with no independent read-back.
| `inhibited_modules` | `CompilerInhibitedModules{READ}` | read-back | explicit false/false reply on CorOS 4.0.1 and 4.1.0; true semantics are schema-derived, not yet observed |
| `create_local_backup` | `LocalBackup{CREATE}` then `LocalBackup{UPDATE, backup_json}` pushes, the last with `is_last_chunk` | captured only | section 10.5. `can_apply_backup` never appeared, so the refusal path is unverified |
| `recall_preset` / `read_preset` | `SetlistPosition{UPDATE, folder_key, position, is_factory, request_id}` then a `RecallPreset` push | read-back | the push echoes the recall's `request_id` |
| `read_current_preset` / `read_current_preset_push` | `RecallPreset{READ, request_id}` | read-back | the live grid, no side effects. The push variant hands back the whole reply with `reason` |
| `read_current_preset` / `read_current_preset_push` | `RecallPreset{READ, request_id}` | read-back | the live grid, no side effects. Two bounded attempts share the caller's timeout; CorOS 4.1.0 was measured emitting an unkeyed duplicate before the keyed reply. The push variant hands back the whole reply with `reason` |
| `loaded_position` | `SetlistPosition{READ, request_id}` | read-back | which slot is loaded; 3 ms measured |
| `list_presets` | `File{action: READ}` then `File{folder{files[] = ProductData}}` | read-back | factory listing gzipped; 256 slots; listings lag after a `File` mutation |
| `switch_scene` | `Scene{UPDATE, selected_scene}` | on-unit | zero-based |
Expand Down
32 changes: 23 additions & 9 deletions pyquadcortex/device/entries.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,15 @@ class FieldPlan:
either has to decide for itself, and this flag is that decision
written down. It does not widen :data:`SCAFFOLDING`, and it is set
per entry and per message type rather than globally.
accepts: optional predicate for message types reused by independent
conversations. A rejected message says nothing about this entry;
it is neither applied nor counted as an arrival.
"""

kept: frozenset = frozenset()
no_presence: frozenset = frozenset()
invalidates: bool = False
accepts: typing.Callable[[Message], bool] | None = None

def voids_the_copy(self) -> bool:
"""Whether a message of this type makes the entry untrusted on its own.
Expand All @@ -99,6 +103,10 @@ def voids_the_copy(self) -> bool:
"""
return self.invalidates

def applies_to(self, message: Message) -> bool:
"""Whether this same-type message belongs to the entry's conversation."""
return self.accepts is None or self.accepts(message)


@dataclasses.dataclass(frozen=True, eq=False)
class StateEntry:
Expand All @@ -110,7 +118,6 @@ class StateEntry:
the unit's whole answer for this entry. Runs on the CALLER's thread,
never the RX thread.
feeds: message class -> :class:`FieldPlan`.

Every entry's :attr:`read` is one request and one ANSWER, which the read
path relies on to tell its own answer apart from a push that arrived while
it was waiting. One answer is not the same as one message: a ``Version``
Expand Down Expand Up @@ -242,6 +249,13 @@ def _carries_unknown_fields(message) -> bool:
# -- the entries -------------------------------------------------------------


def _version_describes_device(message) -> bool:
"""Exclude the two Version shapes that are only handshake traffic."""
named = {field.name for field, _ in message.ListFields()}
host_handshake = SCAFFOLDING | {"cortex_control_version_valid"}
return bool(named - host_handshake) or _carries_unknown_fields(message)


#: The unit's identity. Firmware and serial cannot change while a connection is
#: up: the only thing that changes either is a firmware update, and the firmware
#: `Updater` surface is permanently out of scope for this library (repo-root
Expand Down Expand Up @@ -269,9 +283,13 @@ def _carries_unknown_fields(message) -> bool:
#: fields followed 0.5-0.8 ms later by a ``Version{READ}`` carrying ``action``
#: alone. The question is not news about the unit and the cache does not count
#: it - see ``_apply_one`` in ``device/state.py``, where counting it cost a
#: second round trip whenever it landed before the reading thread woke.
#: second round trip whenever it landed before the reading thread woke. The
#: device's answer to the host's Cortex Control announcement is also a Version,
#: but it carries only handshake validity and likewise says nothing about the
#: unit's identity.
_VERSION_FOR_IDENTITY = FieldPlan(
kept=frozenset({"app_fw_version", "device_serial_number"}),
accepts=_version_describes_device,
)


Expand Down Expand Up @@ -500,13 +518,9 @@ def _read_scene(client) -> dict:
#: a plan, not a fact, and every push mentioning a field it did not keep would
#: mark it for a read nobody had asked for.
#:
#: The Directory's rows are the ones that need something this class does not
#: have. Every read here is one request and one answer, which is how the read path
#: tells its own answer apart from a push that arrived while it was waiting. A
#: setlist listing is a STREAM - one `File` READ makes the unit enumerate its
#: whole tree, several hundred messages over about fifteen seconds - so those
#: entries land with the change to `StateEntry` that lets a read say how many
#: messages it expects.
#: A future entry whose read provokes a stream of answers must declare how that
#: stream completes. No directory entry exists yet; issue #25 still owns that
#: design decision.
ENTRIES = (IDENTITY, DIRTY, PRESET, SCENE, LOADED)

ENTRY_BY_NAME = {entry.name: entry for entry in ENTRIES}
Expand Down
36 changes: 27 additions & 9 deletions pyquadcortex/device/state.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,8 @@
class _Slot:
"""One entry's copy of the unit's state, and how much we trust it."""

__slots__ = ("fields", "needs_read", "witnessed", "marks", "_arrivals",
"_marks")
__slots__ = ("fields", "needs_read", "witnessed", "marks", "observed",
"_arrivals", "_marks")

def __init__(self):
#: field name -> value, holding only what the unit has actually said.
Expand Down Expand Up @@ -89,6 +89,11 @@ def __init__(self):
#: rule. Do not simplify this back.
self._arrivals = itertools.count(1)
self.witnessed = 0
#: Fields applied by meaningful messages since the current read began.
#: The read path uses these to distinguish CorOS 4.1's exact duplicate
#: live-preset reply from a genuine recall that happens in the same
#: window. Cleared under the slot lock immediately before each read.
self.observed = []
#: How many times a path OTHER than the listener has said this copy
#: cannot be trusted: `mark_for_reread` and everything that calls it -
#: a recall's `resets`, the write watchdog giving up, a caller of its
Expand Down Expand Up @@ -213,6 +218,8 @@ def apply_push(self, message) -> None:
from this thread outright (ADR-0009).
"""
for entry, plan in entries.FEEDS.get(type(message), ()):
if not plan.applies_to(message):
continue
try:
self._apply_one(entry, plan, message)
except Exception:
Expand All @@ -238,6 +245,7 @@ def _apply_one(self, entry, plan, message) -> None:
# unit's own `Version{READ}` is exactly that message - see
# :class:`_Slot`.
slot.arrived()
slot.observed.append((dict(applied), why))
was_empty = not slot.fields
# Worked out BEFORE the update, and against the values we held,
# because a caller tracking changes wants the ones that are
Expand Down Expand Up @@ -393,6 +401,7 @@ def value(self, entry_name: str, field: str):
return slot.fields[field]
witnessed_before = slot.witnessed
marks_before = slot.marks
slot.observed = []
client = self._client
if client is None:
raise RuntimeError(
Expand Down Expand Up @@ -420,10 +429,17 @@ def value(self, entry_name: str, field: str):
# than one message: the unit answers a `Version` READ and then
# asks one of its own, and a question is not news about the
# unit's state. `_apply_one` is where that is decided, because
# the read path cannot tell the two apart after the fact. An
# entry whose read provokes a stream of ANSWERS will still have
# to say how many messages that is; see `StateEntry`.
# the read path cannot tell the two apart after the fact.
extra = slot.witnessed - witnessed_before
observations = tuple(slot.observed)
# CorOS 4.1.0 was measured returning an unkeyed RecallPreset
# immediately before the keyed answer. It is safe to discount
# that second arrival only when every arrival restates the
# fields in the answer exactly. A real recall in this window
# carries a different preset and therefore keeps the mark.
restated_answer = all(applied == answer and why is None
for applied, why in observations)
conflicting_push = extra > 1 and not restated_answer
# And the marks that came from something OTHER than a
# message to this entry, which the count above cannot see
# however carefully it counts: a recall's `resets`, the
Expand All @@ -437,16 +453,18 @@ def value(self, entry_name: str, field: str):
# snapshot and the request going out is kept although the unit
# might have accounted for it. `witnessed_before` has had the
# same gap since it was written, and both err by re-reading. The obvious
# `slot.needs_read or extra > 1` does not work: the read's own
# `slot.needs_read or conflicting_push` does not
# work: the read's own
# reply re-arms the mark through the listener from the fields
# this entry does not keep, so the entry would never cache
# anything again.
marked_since = slot.marks - marks_before
was_marked = slot.needs_read
slot.needs_read = extra > 1 or marked_since > 0
if extra > 1:
slot.needs_read = conflicting_push or marked_since > 0
if conflicting_push:
log.info("push.forced_reread %s - %d message(s) arrived "
"while it was being read", entry_name, extra)
"while it was being read and did not all restate "
"its answer", entry_name, extra)
elif slot.needs_read:
# The mirror of the line below, at info rather than its
# debug: a surviving mark costs a real round trip, where a
Expand Down
46 changes: 35 additions & 11 deletions pyquadcortex/protocol/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -1611,7 +1611,7 @@ def preset_dirty(self, timeout: float = 5.0) -> bool:
pa.PresetDirtyMessage(action=pa.MessageAction.READ), timeout=timeout)
return bool(reply.is_dirty)

def read_current_preset(self, timeout: float = 15.0):
def read_current_preset(self, timeout: float = 15.0, attempts: int = 2):
"""The LIVE grid - the current editing state, unsaved changes included.

``RecallPreset{READ}`` answers with the preset as it exists on the device
Expand All @@ -1634,10 +1634,17 @@ def read_current_preset(self, timeout: float = 15.0):
every time** - even when it recalls the preset already loaded. Interleaving it with
scene-targeted writes silently retargets them; use this method for
inspection during editing.

On CorOS 4.1.0, measured 2026-09-11, the first request after a
connection can be dropped. ``attempts`` deliberately defaults to two
on every profile as a bounded resilience measure; ``timeout`` is the
total budget and is divided between attempts.
"""
return self.read_current_preset_push(timeout=timeout).preset
return self.read_current_preset_push(
timeout=timeout, attempts=attempts).preset

def read_current_preset_push(self, timeout: float = 15.0):
def read_current_preset_push(self, timeout: float = 15.0,
attempts: int = 2):
"""The whole ``RecallPreset`` reply, not just the preset inside it.

Same request and same match as :meth:`read_current_preset` - this is
Expand All @@ -1648,15 +1655,32 @@ def read_current_preset_push(self, timeout: float = 15.0):
It exists because the reply carries ``reason`` beside the preset, and a
caller tracking state needs both from one answer. Confirmed on hardware
2026-08-15: the connect burst's seed push sets ``action``, ``preset``
and ``reason``, and so does the push a recall produces.
and ``reason``, and so does the push a recall produces. Re-measured on
CorOS 4.1.0 on 2026-09-11: a READ emits an uncorrelated copy before the
request-id-correlated answer, and its first request can be dropped.
"""
request_id = self._t.next_request_id()
message = pa.RecallPresetMessage(action=pa.MessageAction.READ,
request_id=request_id)
return self._t.await_broadcast(
pa.RecallPresetMessage, lambda: self._t.send(message), timeout=timeout,
match=lambda m: (m.HasField("request_id")
and m.request_id == request_id))
last = None
tries = max(1, attempts)
attempt_timeout = timeout / tries
for attempt in range(1, tries + 1):
request_id = self._t.next_request_id()
message = pa.RecallPresetMessage(action=pa.MessageAction.READ,
request_id=request_id)
try:
return self._t.await_broadcast(
pa.RecallPresetMessage,
lambda outgoing=message: self._t.send(outgoing),
timeout=attempt_timeout,
match=lambda m, wanted=request_id: (
m.HasField("request_id") and m.request_id == wanted))
except TimeoutError as exc:
last = exc
Comment thread
tony-xmelon marked this conversation as resolved.
Resolved
log.warning("live-preset read attempt %d/%d timed out after "
"%ss", attempt, tries, attempt_timeout)
raise TimeoutError(
f"the device did not answer a live-preset read in {tries} "
f"attempt(s) within {timeout}s total; CorOS 4.1.0 was measured "
f"dropping the first request after connecting") from last

def loaded_position(self, timeout: float = 10.0):
"""Which preset slot is on the grid right now.
Expand Down
Loading
Loading