Repository navigation
Add physical screen control and capture - #50
tony-xmelon wants to merge 19 commits into
Conversation
…-control # Conflicts: # changelog.md # docs/ADR.md # docs/STEERING.md
|
Updated onto #52 in 43c9c1b. capture_screen is now represented by its read-only live test in the new operation inventory; tap_screen remains explicitly excused there because it changes the owner's current screen and has no reliable protocol restore, while its exact sequence and validation stay pinned offline. Focused validation: 409 passed; hardware suite collection 100/100; mypy clean; git diff --check clean. |
|
Synced with upstream �2331cc at head 644d4d4 and moved back to draft pending the required full hardware suite on this exact commit. Focused validation: 415 passed; 100 hardware tests collected; mypy clean. |
|
Thank you for the detailed review. I have addressed the requested changes and accepted every nit in
I also removed the unrelated catalog-fingerprint bullet and updated the PR body with the exact retained #42 full-run command and pytest last line. That run predates the operation reporter, so I called out why no operation block exists rather than reconstructing one. Validation on final head: focused The shared 4.1 unit is currently occupied by the active device-testing task, so I converted this PR back to draft until the complete hardware suite can be run on the final head. I have left the 4.0.1 measurement entirely to your follow-up, as requested. |
…-control # Conflicts: # tests/test_client.py
|
Synced with upstream main at 640a8e9 (including #63–#65). New head: 4dc6c03. Offline suite excluding the three #42-owned Windows portability checks: |
|
Local parity sync is now complete for this head ( Integrated locally at
The PR remains draft because this exact head has not yet had a fresh hardware run. Earlier retail CorOS 4.1 evidence is retained in the PR, but I am not treating it as exact-head validation. |
…-control # Conflicts: # CLAUDE.md # changelog.md # docs/ADR.md # docs/STEERING.md # docs/architecture.md # docs/manual-coverage.md # docs/protocol.md # pyquadcortex/protocol/client.py # pyquadcortex/protocol/profiles.py # tests/test_profiles.py # tests/test_session.py
…-control # Conflicts: # changelog.md
|
#42 has merged, and it brought
Could you merge current main in? A review before that reads code that cannot land. |
…-control # Conflicts: # changelog.md # docs/manual-coverage.md # pyquadcortex/protocol/profiles.py # tests/test_profiles.py # tests/test_session.py
|
Merged current upstream main and resolved the profile and coverage overlaps. The 4.1 profile now retains all four RemoteControl operations plus create_local_backup and inherits main's set_device_name; the manual audit is reconciled at 106 features / 69 yes / 8 partly / 18 no / 11 n/a. Current head: �70ae2d. Exact-head validation: 3013 passed, 7 skipped; mypy clean; build and twine check passed; diff check clean. This remains draft pending the exact-head hardware policy already recorded. |
jonathanstokes
left a comment
There was a problem hiding this comment.
Thanks for keeping this current with main. Two things before the detail.
First: swipe_screen and graphics_tree arrived in 923eaef5 on 2026-09-14 and no
maintainer has reviewed them. Roughly 285 lines of new operation surface. I am reading
them as first review, not as a response to anything I wrote.
Second, and the reason most of my 2026-09-09 comments are back: the upstream merge
ce1c8695 on 2026-09-19 deleted your answers to them. At 7c5f1dbc this branch carried
ADR-0021 "A timed device gesture is one atomic transport sequence", a STEERING change-log
entry and its ADR index row, the CLAUDE.md rule about send_sequence, four corrections to
architecture.md, and protocol.md section 7.0 with the probe record and the two dated
coverage rows. None of it is at b70ae2d0. The threads all read resolved. I do not think
you dropped them on purpose - upstream landed its own ADR-0021 under a different title, and
resolving that heading in upstream's favour took the rest with it. Restoring that content is
most of the work below.
Blocking
1. Both hardware tests in tests/hardware/test_remote_screen.py fail on a CorOS 4.0.1
unit rather than skipping. The guard is if "capture_screen" not in profile.VERIFIED: pytest.skip(...). On the 4.0.1 profile VERIFIED is support.EVERYTHING, and
_Everything.__contains__ returns True for every name, so the skip never fires. The call
then raises ControlNotDrivable, which pytest reports as a failure. Checked offline against
the real classes rather than on a unit: "capture_screen" in QuadCortex.VERIFIED is True,
and QuadCortex(fake).capture_screen() raises. Skip on
profile.VERIFIED is support.EVERYTHING as well, or key the skip off
profile.__name__/MEASURED_ON. This is the one that stops the required hardware run on my
unit from being clean.
2. swipe_screen and graphics_tree are listed in QuadCortex41.VERIFIED with evidence
dated before they existed. profiles.py says they "were subsequently exercised during
physical 4.1.0 screen sweeps on 2026-09-08", and the UNMARKED_OPERATIONS entry for
swipe_screen says the same. Both methods first appear in 923eaef5, 2026-09-14. The
2026-09-08 run pasted in the description reports passed: ['capture_screen'] (1). I take the
underlying observation to be real - from the Rust/Windows/Android work, not from this branch -
but VERIFIED is what the ADR-0020 guard reads, and listing them there is what lets a default
Support.VERIFIED connection drive an unmeasured swipe across the owner's screen with no
warning. I know __init_subclass__ forces the listing once you override
(client.py:388-393), so the choice is to measure them on a 4.1 unit on this head, or to
land them in a follow-up and keep this PR to capture_screen and tap_screen. I would
prefer the second - it also gets the reviewed half of this branch moving.
3. The evidence that makes the touch order safe is no longer in the tree. Every gesture
here ends on a message named PRESS, and the proto this PR adds declares PRESS = 0; RELEASE = 1. At 7c5f1dbc, protocol.md 7.0 carried the finding that settles it: "Sending
the labelled PRESS/RELEASE order left the touch held and eventually entered Grid drag mode."
That sentence is gone. What is left is docstrings asserting "touch-down ... then touch-up"
with no bytes and no measurement behind them. The next reader sees type=RELEASE opening a
tap, reads it as an obvious inversion, swaps the pair, and gets the measured failure - a held
touch that enters Grid drag mode, on an instrument with no protocol restore. Please restore
7.0, keeping the bytes and the interpretation separate the way that version did. It also
matters for swipe_screen, whose third message is not covered by the tap capture at all: the
swipe lifts because the tap lifts, which is an inference, not a measurement.
4. Three documents are contradicted by this PR's own code, and the record for
send_sequence is missing. docs/protocol.md:313 says the schema declares 71 types with
NumberOfMessageTypes = 71 as the sentinel; the proto here says 73 and 73.
docs/architecture.md:163 and :166 name four transport methods and say the client depends
on four; there are five now. The send vs request vs await_broadcast table at :285 has no
send_sequence row, and the string send_sequence appears nowhere under docs/ or in
CLAUDE.md. All four were correct at 7c5f1dbc. Separately, CLAUDE.md asks for STEERING,
CLAUDE.md and ADR.md in the same pull request for changes under a section-4 owned path, and
this touches pyquadcortex/ and protocol/. Restoring your ADR-0021 under the next free
number is the piece I most want back - it is the only record of why timing lives in the
transport.
Worth fixing, not blocking
docs/protocol.md's Operation coverage table gains no rows, and four operations land here.
capture_screenandtap_screenhave honest rows to write (on-unit, 2026-09-08 and
2026-09-04, named). The other two have none yet, which is the point of blocker 2.docs/protocol.md:313also lost the| 72 | RemoteControl |type-table row and the
ModelPreset = 71"recovered, not yet observed" note.ModelPresetis not mentioned in the
description either, and it is the message type that decidesRemoteControlis 72.- The
Screenshot = 25/RemoteControl = 72disambiguation from 7.0 is worth restoring on
its own. Two screenshot paths in one schema is exactly the thing the next reader trips on. tests/hardware/test_remote_screen.py:30asserts"zenUI" in tree. No run has seen a
graphics tree, so that string is a guess in an assertion.changelog.mdis missing a blank line before### Rename the unit, drive undo/redo.docs/manual-coverage.mdmarks the rowyesand rolls it into the headline "69 yes", for a
feature my profile refuses. My earlier note asked for this not to overstate;partlywith
"onQuadCortex41" was closer.- Still six words for one thing: physical screen, physical display, remote screen,
remote-control surface, framebuffer, mouse. One word for the screen, and keep "mouse" to the
field name.
Not blocking, for the record
send_sequence's interruption handling only wrapstime.sleep(interval), and does nothing
whenintervalis 0, while the docstring promises atomicity unconditionally. As I said in
September,_write_reportswallowsexcept Exception, so the sleep is the realistic window
and this is mostly fine - butKeyboardInterruptinsideself._dev.writeis a
BaseExceptionand escapes uncaught. Either narrow the docstring or usetry/finally.tap_screen(0, 0)serialises its touch-up asmouse{}- every field at a proto3 default,
so nothing on the wire. The test pins those bytes; nothing says the unit accepts them.- The bindings are correct: I regenerated from the committed proto with
scripts/compile_protos.sh's toolchain and all four files come back byte-identical. Gencode
stayed 7.35.1, so noprotobufpin orgrpcio-toolsfloor move was owed. - Offline suite and mypy are clean on
b70ae2d0itself: 3013 passed, 7 skipped; mypy clean
across 46 files. The description's numbers namecae59296, two merges back. - The branch is 25 commits behind main. Strict status checks make BEHIND block the merge, so
it needs a sync - but that is the last step, not now.
|
Updated the branch to e417162 in response to the review. The public PR surface is now only capture_screen and tap_screen; swipe_screen and graphics_tree implementations, tests, and profile claims were removed for a follow-up. Restored the 73-type and Screenshot-versus-RemoteControl protocol record, send_sequence architecture and rule, operation coverage, and ADR-0024. The 4.0.1 hardware-test guard now keys off measured profile firmware. Offline validation on this head: pytest -q: 2996 passed, 7 skipped; mypy clean across 46 source files; git diff --check clean. I also exercised the device through the shared Rust broker, not direct pyquadcortex: full 800x480 screen capture succeeded; a tap at the visible plus control (100, 147) opened the Plugins chooser; a second tap restored the Grid. The authoritative preset read stayed dirty=false, with zero blocks before and after. Device diagnostics did not return a firmware identity. The broker owns HID, so I could not run pytest tests/hardware --hardware through Python on this same connection. I have left this PR draft and have not synced main. Please run the required full Python hardware suite on this head, or advise whether broker evidence plus the earlier exact-profile capture/tap evidence is sufficient. |
|
Direct HID evidence on exact head e417162, CorOS 4.1.0 (CONTRIBUTED): |
|
Correction and full-suite result on PR #50 exact head e417162, CorOS 4.1.0 / CONTRIBUTED. The preceding short tap comment was truncated by shell quoting; this is the complete record. The complete command was pytest --hardware -q tests/hardware. Result: 105 passed, 11 failed, 8 skipped in 249.00s. Operation summary: 22 of 22 claimed operations measured; 16 passed; 6 failed or skipped; no VERIFIED operation regressed by name. The remote-screen capture test passed. The direct tap test also passed with 1.5s UI-settle delays: tap at (100,147) opened the Plugins chooser, second tap restored Grid, and restored capture matched the original framebuffer byte-for-byte (SHA-256 081b3b256d6730198a4a610ae043615df0097eef5f349eda1d0b33e7db1dd087). Failures:
Cleanup verified directly afterwards: the original My Presets position 52 is loaded, dirty=false, no preset named pyquadcortex scratch remains, and 182 User slots remain free. The tests made no persistent changes. I have not synced main: this run exposed test-data/profile baselines that need maintainer guidance before I can truthfully call the full-suite gate clean. |
|
@jonathanstokes Thanks for the detailed review. I narrowed this branch to Validation on |
|
Exact-head hardware update for The PR's After the run I reconnected QC Remote and verified the device is ready, synchronized, and back on |
…-control # Conflicts: # docs/ADR.md # docs/STEERING.md
|
Synced current upstream Validation on pushed head
The PR remains draft until the full-suite precondition is resolved and review is refreshed. |
|
Exact-head CorOS 4.1.0 hardware validation is now complete on 5c4d80d: 116 passed, 0 failed, 0 errored, 8 skipped in 290.94s; all 22 claimed operations passed, with no operation-level failures or skips. The full-display capture operation passed. The QC was restored afterward to clean Fender Deluxe 37 at 1A; broker ready/synchronized, dirty false, master volume 97. On the empty-slot point: capture_screen() captures the live display and does not depend on any preset slot being saved. For preset_screenshot() in PR #49, the QuadCortex41 override checks whether the requested address is already loaded; if it is, it calls capture_screen() directly, so an empty currently loaded slot does not wait for a stored-preset PNG that does not exist. I verified that exact case on the device: empty slot 7E returned the live 800x480 screen in 12.49s (1 passed), without changing the loaded address. Requests for a different slot still use the stored preset screenshot operation; they are not silently substituted with the current screen. |
|
Thanks for the detailed review. I’ve addressed the remaining protocol-count drift and pushed exact head For clarity on screenshots: |
|
I reran the required full hardware suite on exact head The single failure is |
|
Follow-up: the sparse-preset run above was a coverage-precondition failure, not a protocol mismatch. I reran the complete suite on the same exact head with the saved high-coverage slot required by this test. Result: CorOS 4.1.0 / |
Summary
RemoteControlmessagescapture_screen,graphics_tree,tap_screen, andswipe_screenon the shared operation surface as structured CorOS 4.0.1 refusalsQuadCortex41and list them in that profile'sVERIFIEDsetRemoteControlupdatesThis follows the maintainer's ADR-0020 guidance from reviews on #47 and #49. The maintainer-owned CorOS 4.0.1 measurement remains intentionally out of this branch.
Testing
Final head:
cae5929679a2edbb49fc01417b55c26c6933ad99694 passed, 1 skipped in 20.40s2723 passed, 3 skipped, 3 failed in 48.98s; the three failures are the existing Windows path-separator/CRLF fixture checkspython -m mypy: passed all 46 source filesgit diff --check: passedOffline tests pin the exact protobuf bytes, asynchronous full-frame screenshot matching, rejected-update diagnostics, capture-to-tap settle remainder, timeout context, atomic touch completion after interruption, accepted coordinate boundaries, invalid coordinates, strict-profile behavior, and verification inventory.
Hardware
Historical complete 4.1 hardware run from #42 (head
acf3ffe, 2026-09-03, standard Quad Cortex, CorOS 4.1.0):That run predates the per-profile operation reporter added on 2026-09-05, so there is no operation-summary block in its retained output to paste without inventing one.
Remote-screen capture was subsequently run on exact head
1762ddaon 2026-09-08:The complete 800 x 480 PNG, IHDR, and terminal IEND were validated.
tap_screen(184, 147)was manually verified by tony-xmelon on 2026-09-04: it opened the intended top-row Grid block and a following capture showed its editor. This is recorded explicitly because an automated tap changes the owner's screen and the protocol has no reliable restore operation.The PR remains draft until a complete final-head hardware run can be performed without colliding with the active device-testing work.
Checklist
Upstream sync — 2026-09-12
Merged current upstream main at 640a8e9 (including merged PRs #63–#65) into this branch. Current head:
4dc6c03Offline suite excluding the three #42-owned Windows portability checks:
2621 passed, 3 skipped; full run:2748 passed, 3 skipped, 3 known portability failures; mypy: clean across 46 source files;git diff --check: clean.The three excluded Windows checks are unrelated path-separator and CRLF-fixture assertions fixed within #42; they remain isolated there rather than being copied into this PR.
Local parity sync — 2026-09-14
Local Rust/Windows/Android protocol work exposed the remaining schema-backed
RemoteControlsurface, so commit923eaefadds it here without changing the firmware boundary:swipe_screen(x, y, to_x, to_y, timeout=)performs a fresh framebuffer preflight and then atomically sends the exact touch-down / DRAG-with-both-endpoints / touch-up sequence, 20 ms apart; integer bounds and distinct endpoints match the shared host contractgraphics_tree(timeout=)installs its waiter before sendingRemoteControl{READ, graphics_tree:{}}and accepts only a nonblank, NUL-free, at-most-1-MiB UTF-8UPDATEQuadCortex41Validation in an isolated environment:
2765 passed, 3 skipped, 3 deselected; the deselections are the same three documented Windows path/CRLF baseline checks owned by #42.python -m mypy: clean across 46 source files. Wheel and sdist build plustwine check: passed.git diff --check: passed.The PR remains draft. The read-only
graphics_treehardware test is present; a complete hardware run on exact head923eaefand any screen-changing swipe confirmation should be recorded before requesting review.Upstream sync - 2026-09-19
Merged upstream main at
cfe39cc6592148d4196fa3028c2776f6618c41b6(including maintainer PRs #70-#75) and resolved overlaps in favor of the current upstream API and evidence model.Current head:
cae5929679a2edbb49fc01417b55c26c6933ad99Focused validation: 45 remote-control tests passed; focused coverage and writing-policy checks passed.
GitHub CI on this exact head is green: build + twine check, mypy, and tests on Python 3.11, 3.12, and 3.13 all pass.
No new maintainer comments were posted during this sync. This PR remains draft under its stated hardware/evidence policy; the displayed CHANGES_REQUESTED decision is the earlier review and has not yet been refreshed.
Current scope and status — 2026-09-27
This section supersedes the earlier summary and validation text above. Current head:
f8387d9d0c544272186666ded3e2d6c86f08dd5a.The public surface is narrowed to
capture_screen()andtap_screen();swipe_screen()andgraphics_tree()were removed in response to review and need separate review/evidence before returning. The hardware test now skips profiles that do not identify as CorOS 4.1.0. The docs restore the message-type 72RemoteControlrow, preserve the bytes separately from enum-label interpretation, use “physical screen” consistently, and describe only the shipped two-operation scope. ADR-0024 records the atomic gesture transport decision; the 2026-09-27 STEERING entry records the scope change.Validation at this content: full offline suite 2,996 passed, 7 skipped; focused client/registry/marker/writing suite 458 passed, 3 skipped; mypy clean (46 source files); diff check clean. The direct-HID capture and manual tap observations are recorded above. The complete hardware suite has not been run on this exact head. QC Remote currently reports the device disconnected, so no reconnect was attempted. PR remains draft pending exact-head hardware validation.
Current synced status — 2026-09-27 (head
5c4d80d)This supersedes the earlier head and validation above. Merged upstream
main(d8d40f3) in5c4d80d, preserving both the screen-control record and upstream catalog-payload record.3,002 passed, 7 skipped; exact-head CI run36289727069is green (build/twine, mypy, Python 3.11–3.13).capture_screenhardware test:1 passed.3,116 passed, 1 failed, 16 skippedin 316.16s. The sole failure is the unrelated option-position coverage floor: the cleanFender Deluxe 37preset exposes 59 positions versus a 100-position minimum; no mismatch was found. The pending Add profile-owned CorOS 4.1 catalog and observations #44 change classifies insufficient coverage as a skip, but is not yet onmain, so this is not a clean full-suite pass.dirty: false, master volume unchanged at 97.The PR remains draft until the full-suite precondition is resolved and review is refreshed.
Exact-head hardware recheck — 2026-09-27
This supersedes the earlier complete-suite failure recorded above. On exact head
5c4d80d15ec11be7a7f34399916653ff79cc6505, the complete CorOS 4.1.0 hardware suite passed using a saved high-coverage preset:capture_screenpassed. The earlier sole failure was the 100-position coverage floor onFender Deluxe 37(59 positions), not a behavioral mismatch; this rerun usedShine On Crazy Synthat 12E and met the coverage precondition. The shared broker was reconnected and QC restored to cleanFender Deluxe 37at 1A, synchronized, master volume unchanged at 97.Empty-slot behavior is separate from the stored-preset screenshot request:
capture_screen()captures the live display independent of slot occupancy. PR #49'sQuadCortex41.preset_screenshot()checks whether the requested slot is already loaded and routes that case directly tocapture_screen(), so an empty current slot does not wait for a nonexistent stored image. This was verified against an empty loaded slot (7E): live 800 x 480 PNG returned, test passed in 12.49s, loaded address unchanged. A request for another slot still asks for that slot's stored screenshot.The PR remains draft pending maintainer re-review; the earlier review's requested scope/evidence/document follow-ups still need to be formally cleared.
Exact final-head validation — 2026-09-27 (1001b61)
This supersedes the earlier hardware-status entries above. After correcting the remaining stale
71message-type count in the open-questions section, validation was rerun on exact head1001b61516ad1548b911ac2428593c15f942a745:3,002 passed, 7 skipped in 43.24s.Shine On Crazy Synthat 12E loaded to satisfy the option-position coverage floor:116 passed, 0 failed, 0 errored, 8 skipped in 290.72s.capture_screenpassed.36292984335: build/twine, mypy, and Python 3.11/3.12/3.13 all pass.Fender Deluxe 37at 1A, synchronized, master volume 97.All 18 inline review threads are resolved. The requested operation scope is capture and tap only; swipe and graphics-tree remain out of this PR. The 4.0.1 measurement remains with the maintainer, as agreed in the review. The PR is ready for the requested maintainer re-review.
Exact-head full-suite rerun — 2026-09-28
Reran
pytest tests/hardware --hardware -qon unchanged head1001b61516ad1548b911ac2428593c15f942a745against CorOS 4.1.0 /QuadCortex41, using the saved high-coverage preset at 12E required by the option-position test:116 passed, 0 failed, 0 errored, 8 skipped in 290.55s. All 22/22 claimed hardware operations passed; no operation-level failures or skips. The original active preset was restored cleanly; the shared broker is ready/synchronized and master volume remained unchanged at 100.