Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
30af967
Add physical screen control and capture
tony-xmelon Sep 3, 2026
4de8690
Harden remote screen coverage
tony-xmelon Sep 3, 2026
9ec95c6
Merge remote-tracking branch 'upstream/main' into codex/remote-screen…
tony-xmelon Sep 5, 2026
43c9c1b
Merge upstream device profiles into remote screen control
tony-xmelon Sep 7, 2026
644d4d4
Merge remote-tracking branch 'upstream/main' into codex/remote-screen…
tony-xmelon Sep 8, 2026
1762dda
fix: scope remote screen control to CorOS 4.1
tony-xmelon Sep 8, 2026
47056da
Merge remote-tracking branch 'upstream/main' into codex/remote-screen…
tony-xmelon Sep 9, 2026
7c5f1db
fix: address remote screen review
tony-xmelon Sep 9, 2026
4dc6c03
Merge remote-tracking branch 'upstream/main' into codex/remote-screen…
tony-xmelon Sep 12, 2026
923eaef
Add complete CorOS 4.1 remote screen surface
tony-xmelon Sep 14, 2026
ce1c869
Merge remote-tracking branch 'upstream/main' into codex/remote-screen…
tony-xmelon Sep 19, 2026
cae5929
Fix coverage and documentation checks
tony-xmelon Sep 19, 2026
8bd4129
Merge remote-tracking branch 'upstream/main' into codex/remote-screen…
tony-xmelon Sep 22, 2026
b70ae2d
Merge remote-tracking branch 'upstream/main' into codex/remote-screen…
tony-xmelon Sep 22, 2026
e417162
Narrow remote screen PR to capture and tap
tony-xmelon Sep 26, 2026
1cdf2ef
Make CorOS 4.1 hardware verification profile-aware
tony-xmelon Sep 26, 2026
f8387d9
Align remote-screen docs with reviewed API
tony-xmelon Sep 26, 2026
5c4d80d
Merge remote-tracking branch 'upstream/main' into codex/remote-screen…
tony-xmelon Sep 27, 2026
1001b61
Correct RemoteControl message total in protocol notes
tony-xmelon Sep 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,10 @@ small documentation fixes.
`collect` refuse on that thread. To see the connect burst, register through
`protocol.connect(before_handshake=...)`. (ADR-0009)
- New operations follow `docs/architecture.md`, "How to add a new operation".
- A timing-sensitive multi-message operation uses `Transport.send_sequence`,
which frames first and holds the write lock across the sequence. Do not sleep
between writes in `protocol/client.py` or issue those messages as separate
`send` calls.

## Do not

Expand Down
7 changes: 7 additions & 0 deletions changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ sustained stretch without a correction.

## Unreleased

### CorOS 4.1 physical-screen control is exposed

`QuadCortex41` can capture the full 800 x 480 physical screen and tap a pixel.
The touch sequence is atomic and a capture primes the remote-control surface
before input. The 4.0.1 profile keeps these operations visible but refuses them
because they are unmeasured there.

### Rename the unit, drive undo/redo, and read inhibited modules

`set_device_name()` sends a sparse Version update. `undo()` and `redo()` drive
Expand Down
9 changes: 9 additions & 0 deletions docs/ADR.md
Original file line number Diff line number Diff line change
Expand Up @@ -397,6 +397,15 @@ Records are append-only once `Decided` and built upon: a shipped decision is nev
- This is repository governance rather than library architecture, and it lives here rather than in `contributing.md` or `docs/STEERING.md` because the durable part is the reasoning - in particular why `require_last_push_approval` was rejected, which is the decision a future maintainer would otherwise re-litigate from scratch. `contributing.md` gains its note about sync-before-review only once contributors can actually merge.
- Reversal is one API call and costs nothing, which is why this record is short on hedging: the setting is cheap to try and cheap to undo.

## ADR-0024: A timed device gesture is one atomic transport sequence

- **Status:** Decided (2026-09-04)
- **Decision:** A protocol operation made of timing-sensitive messages uses `Transport.send_sequence`. The transport frames every message first and holds the write lock across writes and inter-message intervals; the client selects messages and measured timing but does not sleep or write reports itself.
- **Context:** CorOS 4.1 touchscreen input needs two `RemoteControl` messages 20 ms apart after the initial screenshot read settles. Separate `send` calls allow keepalive traffic to split the gesture and put timing in the wrong layer.
- **Options:** (a) Atomic transport sequence — chosen. (b) Sleep and call `send` from the client — rejected because another write can interleave. (c) Write HID reports directly — rejected because framing belongs to the transport.
- **Open Questions:** None.
- **Rationale:** The transport owns serialization and timing; the protocol client owns semantic message construction.
- **Consequences:** Timing-sensitive multi-message operations use `send_sequence`; tests assert message order and interval/delay arguments. A delayed sequence remains non-transactional at the device and does not imply an acknowledgement.
## ADR-0022: The payload a catalog snapshot is generated from is committed beside it

- **Status:** Decided (2026-09-22)
Expand Down
10 changes: 10 additions & 0 deletions docs/STEERING.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ Decisions are recorded in [`ADR.md`](ADR.md):
| ADR-0020 | Connect resolves a device profile, and nothing else branches on firmware or model |
| ADR-0021 | An approval stops counting once the pull request's code changes |
| ADR-0022 | The payload a catalog snapshot is generated from is committed beside it |
| ADR-0024 | A timed device gesture is one atomic transport sequence |

## 8. Open Questions

Expand Down Expand Up @@ -233,6 +234,15 @@ Entries are short by design ([`writing.md`](writing.md)). The full narrative
behind each one is in the lab repository,
`doc/pyquadcortex/history/steering-change-log.md`, and in the pull requests.

### 2026-09-27 - Remote screen control is narrowed to reviewed operations

- **What changed:** the public PR surface is limited to `capture_screen()` and
`tap_screen()`. The 4.1 hardware test skips unmeasured profiles, and protocol
coverage records the broker-observed 800 x 480 capture and tap behavior.
- **Why:** swipe and graphics-tree were added without review and lacked
operation-specific evidence on this branch.
- **Scope:** protocol client/profile, screen tests and API/protocol/coverage
documentation. See ADR-0024 for atomic gesture timing.
### 2026-09-22 - A snapshot's input is committed beside it (ADR-0022)

- **What changed:** the `ModelRepo` payload the 4.0.1 snapshot is generated from
Expand Down
1 change: 1 addition & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ already read and need no connection; calling them as methods raises
| **Connect** | `protocol.connect(profile=, support=)` resolves the unit's device profile before the handshake (ADR-0020) and returns a connected client. An unknown `(device_type, zenos_git_hash)` raises `UnsupportedDevice`. `Support.VERIFIED` (default) refuses an operation the profile has not verified; `Support.EXPERIMENTAL` runs it with one warning. `qc.models`, `qc.params` and `qc.options` are that connection's own constants |
| **Inspect / device identity** | `version()`, `set_device_name(name)`, `list_presets(setlist)`, `find_preset(name, setlist)`, `read_preset(setlist, slot)` |
| **Navigate** | `recall_preset(setlist, slot)`, `switch_scene(scene)` |
| **Drive and inspect the unit's screen (`QuadCortex41`, CorOS 4.1.0)** | `tap_screen(x, y, timeout=)` taps one raw pixel coordinate; `capture_screen()` returns a PNG of the QC's current display. The 4.0.1 base refuses both operations pending measurement |
| **Edit the grid** | `set_chain_input(row, input)`, `reroute_grid_input(preset, input)`, `set_param(target, param, value)`, `set_bypass(Block(row, column), bypassed)` |
| **Add and remove blocks** | `set_block(Block(row, column, model_id))`, `remove_block(cell)`, `move_block(source, destination)`, `catalog` |
| **Parallel lanes** | `set_split(row, split_column, mix_column)`, `clear_split(row)`, `set_split_mute(row)`, `protocol.splits(preset)` |
Expand Down
16 changes: 9 additions & 7 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ For the wire itself (frame layout, handshake, message shapes) see

- [Layer map](#layer-map)
- [What flows through the layers](#what-flows-through-the-layers)
- [send vs request vs await_broadcast](#send-vs-request-vs-await_broadcast)
- [send vs send_sequence vs request vs await_broadcast](#send-vs-send_sequence-vs-request-vs-await_broadcast)
- [How to add a new operation](#how-to-add-a-new-operation)
- [The generated protobuf bindings](#the-generated-protobuf-bindings)
- [Testing philosophy](#testing-philosophy)
Expand Down Expand Up @@ -160,10 +160,10 @@ classes: `type_for(cls)` and `class_for(message_type)`. A type absent from
### client.py

`QuadCortex` is the message-level API. It builds protobuf messages and calls
`send`, `request`, `await_broadcast` and `next_request_id` on the transport it
`send`, `send_sequence`, `request`, `await_broadcast` and `next_request_id` on the transport it
was given. It never touches a report, a frame or a byte offset.

Because it depends on four transport methods, the whole API is testable with a
Because it depends on five transport methods, the whole API is testable with a
short fake (`tests/test_client.py`) and no unit, no `hid`, no timing. Every wire
concern stays below this line. When you add an operation, the protobuf building
belongs here and nothing else does.
Expand Down Expand Up @@ -282,14 +282,16 @@ device.read() -> one 129-byte input report
else a broadcast waiter, else dropped
```

## send vs request vs await_broadcast
## send vs send_sequence vs request vs await_broadcast

Choosing correctly is most of the work of adding an operation. The first three
rows serve one exchange. The last is how a long-lived caller watches the link.
Choosing correctly is most of the work of adding an operation. The first four
rows send one exchange or wait for its answer. `add_listener` is how a long-lived
caller watches the link.

| Transport method | Use when | Blocking | Correlation |
|---|---|---|---|
| `send(msg)` | The unit acts on the message and you do not need its answer: scene switch, grid edits, recall, keepalive. | No | None |
| `send_sequence(messages, interval=, delay=)` | Messages form one short timing-sensitive gesture that concurrent writes must not split. The transport frames first and holds the write lock across the sequence. | During the requested delay and intervals | None |
| `request(msg, timeout=)` | The unit answers with a message of the **same type**: `Version` read, `ResetCommsBuffers`, the `File` mutations. | Yes | A fresh `request_id` is registered before the write. The reply is the first inbound message of the same type whose `request_id`, if present on both sides, matches. |
| `await_broadcast(cls, trigger, timeout=, match=)` | The answer arrives as a **push of a different type**, or as an unsolicited push the unit emits in response to an action: the `RecallPreset` push that carries a preset, the `File` folder listings. | Yes | By message class, plus your `match` predicate. A message the predicate rejects is left for a later waiter. |
| `add_listener(fn)` | You want every message for the life of the connection: a cache fed by the unit's pushes, or a log of the link. | No, but `fn` runs on the RX thread | None. Every message, every type. Removed with the returned callable or `remove_listener(fn)`. |
Expand All @@ -312,7 +314,7 @@ Worked example: `set_global_tempo(bpm)`.
`protocol/proto/ProductionAutomation.proto` (control messages) and
`protocol/proto/Preset.proto` (the `BinaryPreset` grid model). Start from the
`CortexMessageType.Enum` block at the top of `ProductionAutomation.proto`, which
lists all 71 types with their wire integers. Find `GlobalTempo = 33`, then
lists all 73 types with their wire integers. Find `GlobalTempo = 33`, then
`message GlobalTempoMessage`. Nearly every scalar field sits in a synthetic
`oneof`, so `HasField()` tells "set to zero" from "not set".

Expand Down
5 changes: 3 additions & 2 deletions docs/manual-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ A "candidate" names a message type from the unit's own schema or a field in

## Summary

Of 105 features audited: **68 yes**, **8 partly**, **18 no**, **11 n/a**.
Of 106 features audited: **68 yes**, **9 partly**, **18 no**, **11 n/a**.

Of the 94 features a host could plausibly drive, 68 are fully covered and 8 are
Of the 95 features a host could plausibly drive, 69 are fully covered and 8 are
partly covered. Eighteen remain untouched. Of those, a few are writes confirmed to
do nothing with no route found (preset tags, duplicating a setlist as one
operation), and two whole features need the physical world: creating a Neural
Expand Down Expand Up @@ -112,6 +112,7 @@ settled is in [capture.md](capture.md).
| Create a folder, nested navigation | yes | `create_setlist(name)`. Setlists are siblings under `/media/p4/Presets`, not children of My Presets |
| Favorites and Recents | yes | `recents()` and `favorites()` read the two lists. The request's `is_favorites` flag selects which; the reply never sets it, so correlate on `request_id`. `add_favorite()` and `remove_favorite()` write one entry at a time. Only presets can be favourited |
| Bulk actions | partly | no host-drivable bulk copy; `BulkOperation` only narrates progress. `copy_preset()` and `duplicate_setlist()` do it by recall plus save, a few seconds per preset |
| Physical screen | partly | on `QuadCortex41`, `capture_screen()` and `tap_screen()` use measured CorOS 4.1.0 `RemoteControl` shapes; the 4.0.1 profile refuses them pending measurement |
| Search | no | candidate `RecentSearches` |
| Sort | n/a | client-side once a listing is in hand |
| Neural Captures: list | yes | `captures()` browses the library, over 2000 entries. Not the catalog, which does not grow when a capture is saved |
Expand Down
62 changes: 59 additions & 3 deletions docs/protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -311,8 +311,10 @@ The host's `License` read that provoked it carries `08 03` and `ENCRYPTED = 0`
## 3. Message types and actions

Every frame's trailer carries a `CortexMessageType.Enum` value. The schema
declares **71 types** (`Undefined = 0` through `GenerateTestPreset = 70`, with
`NumberOfMessageTypes = 71` as a sentinel). The ones this library uses most:
declares **73 types** (`Undefined = 0` through `RemoteControl = 72`, with
`NumberOfMessageTypes = 73` as a sentinel). `ModelPreset = 71` is recovered
but unobserved; `RemoteControl = 72` is the live-screen path, distinct from
preset-thumbnail `Screenshot = 25`. The ones this library uses most:

| Value | Type | Role here |
|---|---|---|
Expand All @@ -330,6 +332,7 @@ declares **71 types** (`Undefined = 0` through `GenerateTestPreset = 70`, with
| 49 | `Connection` | connected / disconnected announce |
| 51 | `ModelRepo` | required readiness step in the handshake, and the catalog |
| 52 | `ResetCommsBuffers` | session hello with a session token |
| 72 | `RemoteControl` | physical-screen capture and touch input |

`registry.py` registers 70 of those 72 enum values (the 71 types plus the
sentinel) so the RX path can decode them.
Expand Down Expand Up @@ -424,6 +427,13 @@ inbound `Version` messages, the full reply (15 fields, at +0.71 s), the unit's
own `Version{READ}` about 1 ms behind it, and the answer to the announce
(carrying `cortex_control_version_valid`, at +0.73 s).

A separate contributed capture on CorOS 4.1.0 / app firmware d14e (2026-09-27)
recorded only the full `Version{UPDATE}` reply and the unit's own
`Version{READ}` during `connect()` and its burst. No announce answer carrying
`cortex_control_version_valid` was observed. The 4.0.1 three-message shape above
is therefore not assumed for 4.1.0; the hardware test checks the profile-specific
shape rather than treating the missing announce answer as a regression.

### 4.3 Keepalive and disconnect

Cortex Control 4.0.1 sends `KeepAlive{action: UPDATE}` about once per second:
Expand Down Expand Up @@ -1804,6 +1814,50 @@ every entry for that id.

**Gig View** is `ShowGigView{UPDATE, show}`; `show` has no presence.

### 11.8 CorOS 4.1 physical-screen control

`RemoteControl{READ, screenshot:{}}` produces an uncorrelated `UPDATE` carrying
a complete 800 x 480 `PNG` of the physical screen; it has no `request_id`, so
`capture_screen()` waits for the broadcast and rejects region frames. This is
distinct from preset-image `Screenshot = 25`. The latter takes an address
(`folder_name`, `is_factory`, `index`) and returns the stored preset rendering;
a bare `Screenshot{READ}` without that address did not answer in a contributor's
probe on 2026-09-26. An empty slot has no stored preset rendering; use
`capture_screen()` when the physical screen itself is wanted. That operation
does not recall or navigate to another slot.

The observed tap on CorOS 4.1.0 sent `type: 1`, then 20 ms later omitted the
proto3-default `type: 0`. The recovered labels interpret this as `RELEASE=1`
then `PRESS=0`, though swapped labels fit the bytes equally well. Sending the
labelled `PRESS`/`RELEASE` order left touch held and eventually entered Grid drag
mode. On 2026-09-04, tony-xmelon observed the value-1/value-0 pair at Grid
coordinate `(184, 147)` open the intended block; the following capture showed
its editor. `TAP=3` and `MOVE=2` attempts were unreliable in that retained probe,
which has no defensible trial count. An unprimed pair did not land; one
screenshot read and a conservative 300 ms wait did. That observation does not
establish a minimum settle time. Neither mouse message is acknowledged, so
`tap_screen()` returns after the timed sequence is transmitted.

For `(184, 147)`, the serialized protobuf payloads (not USB/HID reports) are
`08 01 1a 0c 0d 00 00 38 43 15 00 00 13 43 18 01`, then
`08 01 1a 0a 0d 00 00 38 43 15 00 00 13 43`. The first carries numeric
`type: 1`; the second omits the proto3-default `type: 0`. Those bytes are the
measurement; the enum labels are recovered-schema interpretation, not a
separate observation. At `(0, 0)`, the second payload is `08 01 1a 00`; this
default-coordinate boundary is pinned offline, but has not been separately
verified on the unit.

On 2026-09-27, tony-xmelon repeated the direct-HID check from an empty Grid:
`(100, 147)` opened the Plugins chooser; tapping the same point again restored
the original screen byte-for-byte. The loaded preset remained clean and no
block was added. The retained 2026-09-04 probe does not preserve a reliable count
or starting-screen record for its `TAP`/`MOVE` attempts, so those attempts are
not used to claim a success rate.

The capture and tap shapes have contributed CorOS 4.1.0 evidence and are
refused by the unmeasured 4.0.1 profile. Swipe and graphics-tree operations are
not included in this API pending separate review and hardware evidence.

**Undo and redo** are drivable: a sparse `UndoRedo{UPDATE, undo: true}`
(`08 01 28 01`) reverses the last grid edit and `redo: true` (`08 01 30 01`)
reapplies it. Measured 2026-09-03 on CorOS 4.0.1 by preset read-back, and by a
Expand Down Expand Up @@ -2192,6 +2246,8 @@ wire, with no independent read-back.
| `set_block` | `Grid{UPDATE, preset{chains{row, models{column, hash}}}}` | read-back + on-unit | a placement can be refused silently, for DSP capacity or a port conflict; verified against the echo and a read-back |
| `remove_block` | `Grid{action: DELETE, preset{chains{row, models{column, hash: 0}}}}` | read-back + on-unit | the action marks the removal |
| `catalog` | `ModelRepo{READ}` then `ModelRepo{model_repo_payload}` | read-back | gzip(tar(ModelRepo.xml)) |
| `capture_screen` | `RemoteControl{READ, screenshot:{}}` then uncorrelated `UPDATE` with full PNG | contributed direct-HID hardware test | measured 2026-09-27 by tony-xmelon on Quad Cortex, CorOS 4.1.0; full physical-screen PNG is 800 x 480 |
| `tap_screen(x, y)` | screenshot prime, then atomic `RemoteControl{UPDATE, mouse}` pair | contributed direct-HID observation | measured 2026-09-04 and repeated 2026-09-27 by tony-xmelon on CorOS 4.1.0; `(100, 147)` opened Plugins chooser and the second tap restored an identical Grid screen |

## Open questions

Expand Down Expand Up @@ -2221,6 +2277,6 @@ wire, with no independent read-back.
- **Cross-setlist moves**, downloads and plugin folders (`SetlistPosition.is_downloads`,
`is_plugin`), the IR payload format, and bulk operations are present in the
schema and unobserved.
- **About half the schema's 71 message types** have never been seen on the wire
- **About half the schema's 73 message types** have never been seen on the wire
by this project. Their field layouts are known from the schema; their behaviour
is not.
66 changes: 65 additions & 1 deletion protocol/proto/ProductionAutomation.proto
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,9 @@ message CortexMessageType {
SetTestPresetInputOutputPorts = 68;
SetTestPresetSplitMixPoints = 69;
GenerateTestPreset = 70;
NumberOfMessageTypes = 71;
ModelPreset = 71;
RemoteControl = 72;
NumberOfMessageTypes = 73;
}
}
message MessageAction {
Expand Down Expand Up @@ -2074,3 +2076,65 @@ message GenerateTestPresetMessage {
}
}

message ModelPresetID {
optional string value = 1;
optional bool is_factory = 2;
optional uint32 hash = 3;
}

message ModelPreset {
optional cortex_protobuf_v2.ModelPresetID id = 1;
optional string name = 2;
optional bool is_default = 3;
optional Model model = 4;
}

message ModelPresetMessage {
cortex_protobuf_v2.MessageAction.Enum action = 1;
optional uint64 request_id = 2;
repeated cortex_protobuf_v2.ModelPreset presets = 3;
optional int32 create_from_row = 4;
optional int32 create_from_column = 5;
optional int32 create_from_hash = 6;
optional int32 loaded_row = 7;
optional int32 loaded_column = 8;
optional int32 loaded_hash = 9;
optional int32 loaded_capture_version = 10;
optional cortex_protobuf_v2.ModelPresetID loaded_preset_id = 11;
}

message RemoteControlMouse {
float x = 1;
float y = 2;
enum Type {
PRESS = 0;
RELEASE = 1;
MOVE = 2;
TAP = 3;
DRAG = 4;
}
Type type = 3;
float to_x = 4;
float to_y = 5;
}

message RemoteControlScreenshot {
optional bytes payload = 1;
optional uint32 x = 2;
optional uint32 y = 3;
optional uint32 w = 4;
optional uint32 h = 5;
}

message RemoteControlGraphicsTree {
optional string payload = 1;
}

message RemoteControlMessage {
cortex_protobuf_v2.MessageAction.Enum action = 1;
optional uint64 request_id = 2;
optional cortex_protobuf_v2.RemoteControlMouse mouse = 3;
optional cortex_protobuf_v2.RemoteControlScreenshot screenshot = 4;
optional cortex_protobuf_v2.RemoteControlGraphicsTree graphics_tree = 5;
}

Loading
Loading