Skip to content

Commit f550eba

Browse files
committed
Record what shipped, and the one thing deliberately left
WHATS_NEW and both translations describe the six phases; CHANGELOG records the compatibility-visible parts — the window backends, the AT-SPI backend, platform_id, the new exception class and the D-Bus client's move. Progress.md gains the macOS recorder: OSXRecorder is a complete implementation that the platform wrapper never selects, and that is not an oversight. osx_listener builds an NSApplication at import time and stopping a recording needs a blocking run loop, so wiring it up as it stands would move both onto the import path of the whole package. The capability matrix already says 'unavailable', which matches.
1 parent 3afc453 commit f550eba

5 files changed

Lines changed: 282 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,25 @@ only when documented here with a migration path.
1010

1111
### Added
1212

13+
- Cross-platform window management. The 23 `AC_*` window commands and their
14+
MCP tools now work on macOS and Linux/X11 as well as Windows, through a
15+
backend seam (`je_auto_control.wrapper.window_backends`). Wayland remains
16+
unsupported: the protocol does not let a client enumerate or move another
17+
application's windows.
18+
- Linux accessibility backend over AT-SPI2
19+
(`je_auto_control.utils.accessibility.backends.linux_backend`), with no new
20+
dependency. Serves both X11 and Wayland sessions.
21+
- `je_auto_control.utils.platform_id` — one place that classifies the
22+
operating system family, and the BSDs are now one of them. FreeBSD, OpenBSD,
23+
NetBSD and DragonFly route to the X11 backend instead of raising "unknown
24+
operating system".
25+
- `AutoControlUnsupportedOperationException`, raised when a platform backend
26+
cannot perform an operation. It subclasses both `AutoControlException` and
27+
`NotImplementedError`, so existing `except NotImplementedError` handlers are
28+
unaffected while the executor's containment boundaries now catch it.
29+
- `je_auto_control.utils.dbus_client` — the D-Bus client, moved out of
30+
`linux_wayland/` so `utils/` can use it. The old path re-exports it.
31+
1332
- Stable, headless `je_auto_control.api` façade.
1433
- Portable `autocontrol.failure-bundle/v1` diagnostic archives and CLI command.
1534
- Public API lifecycle, capability matrix, security policy, coverage and type
@@ -247,6 +266,19 @@ only when documented here with a migration path.
247266

248267
### Fixed
249268

269+
- macOS: `write()` typed a space instead of a backspace, because `"\b"` had
270+
no route in the macOS key table and fell through to the space fallback.
271+
- macOS: USB enumeration returned `apple_vendor_id` in `vendor_id`, a field
272+
documented as a four-hex-digit string. A value that is not a hex id is now
273+
`None`; the device is still listed and `manufacturer` still names the vendor.
274+
- Linux/X11: `window_rect` returned the client area rather than the frame,
275+
disagreeing with Win32's `GetWindowRect` by the window decorations.
276+
- Linux/X11: `move_window_by_title` configured the client window directly,
277+
which under a reparenting window manager positions it in the wrong
278+
coordinate space. It now goes through `_NET_MOVERESIZE_WINDOW`.
279+
- The D-Bus client could not marshal or demarshal signed integers, so any
280+
protocol using them (AT-SPI extents among them) failed to decode.
281+
250282
- **Wayland: an absolute mouse move through the ydotool fallback counted from
251283
the wrong origin.** `ydotool mousemove --absolute` emits no absolute event —
252284
it drives the cursor into the corner the compositor clamps to and then moves

‎Progress.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,21 @@
5050

5151
---
5252

53+
## macOS 的錄製器寫了,但接不上去
54+
55+
`TODO` — `osx/record/osx_record.py`、`osx/listener/osx_listener.py`
56+
57+
`OSXRecorder` 是完整實作,但 `wrapper/_platform_osx.py` 裡寫的是
58+
`recorder = None`,所以它永遠不會被選到。這不是疏失:
59+
`osx_listener.py` 在 **import 時**就呼叫 `NSApplication.sharedApplication()`,
60+
而停止錄製要靠 `AppHelper.runEventLoop()`,那是一個會卡住呼叫緒的
61+
事件迴圈。直接接上去會把這兩件事都搬進 `import je_auto_control`
62+
的路徑上,那是回歸而不是修好。
63+
64+
要接上去得先把 listener 改成:不在 import 時建立 NSApplication,
65+
且把 run loop 放到自己的執行緒。`docs/CAPABILITY_MATRIX.md` 的 Recorder
66+
macOS 格寫的是 `unavailable`,跟現狀一致。
67+
5368
## `mouse_scroll` 的方向在三個平台上不是同一回事
5469

5570
`DECIDE` — `wrapper/auto_control_mouse.py::mouse_scroll`

‎README/WHATS_NEW_zh-CN.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,69 @@
11
# 本次更新 — AutoControl
22

3+
## 本次更新 (2026-08-20) — 声称支援的平台,這回真的量過了
4+
5+
整套测试一直只在 `windows-2022` 跑,另加容器裡一次 Linux
6+
執行。macOS 只跑兩行指令。Wayland 有五個 job 對真的對等體
7+
讀回輸入;X11——兩條 Linux 路徑中更老、部署更廣的那一
8+
條——一個都沒有,而且套件裡每一條 X11 斷言都是對著
9+
`python-Xlib` 的 mock 做的。
10+
11+
**测试套件現在真的在它宣稱支援的平台上跑。**
12+
`pytest-headless` 改成 OS 矩陣;Linux 跑在真的 Xvfb 上而不是 Qt 的
13+
offscreen,因為 X11 後端在 import 時就連線,offscreen 會將正好要找的
14+
毛病蓋掉。第一輪就抓到兩個真的 macOS 缺陷:
15+
16+
- `write("\b")` 在 macOS 沒有任何按鍵路徑,會落到空白鍵
17+
fallback——要求退格,打出來的是空白。
18+
- `system_profiler` 對 Apple 自家裝置回的是 **符號式** vendor
19+
id(`apple_vendor_id`),而那個欄位文件上寫的是四位十六進位。
20+
21+
**X11 的輸入現在從真的客戶端讀回。** 新的 `x11-verification`
22+
job 跑在真的 Xvfb + 真的視窗管理員上,對照組來自受測對象以外的
23+
程式碼:`xev`、ImageMagick 的 `import`、`xdotool` 與 `xdpyinfo`。
24+
最值得點名的一項是 `synthetic NO`——`XSendEvent` 的事件帶的是
25+
`YES`,大多數 toolkit 會直接丟掉,所以一個患患停止驅動真實
26+
輸入的後端,在只數事件的檢查下仍然會全綠。
27+
28+
**macOS 在 CI 裡其實完全驗得了,跟一般假設相反。**
29+
`macos-14` runner **兩個 TCC 權限都給**:擷取回來的是真像素而不是
30+
被拒時的全黑矩形,`CGEventPost` 真的移得動游標且讀回完全相符,
31+
AX 樹也走得出真的元素。這是先量再斷言的,而且探針在期望表
32+
還是空的時候會拒絕通過。
33+
34+
### 視窗管理不再是 Windows 專屬
35+
36+
以前是:門面在 `sys.platform` 上分支,其他平台一律丟例外,
37+
23 個 `AC_*` 指令跟對應的 MCP 工具在 macOS 與 Linux 上都是死的。
38+
現在走平台縫:Win32、X11 的 EWMH、macOS 的 Quartz + 無障礙 API。
39+
40+
兩件只有真的視窗管理員才能披露的錯,第一版都錯了:
41+
42+
- **矩形是外框,不是客戶區。** Win32 的 `GetWindowRect`
43+
回的是外框,所有呼叫端都是照那個寫的。
44+
- **移動必須走 `_NET_MOVERESIZE_WINDOW`。** 在 reparenting
45+
視窗管理員下,客戶端自己的 x/y 是相對於外框的;對 openbox
46+
要 (300, 220),直接 `ConfigureWindow` 的結果是落在 (302, 260)。
47+
48+
### Linux 有無障礙後端了
49+
50+
之前完全沒有。新後端走 **AT-SPI2**——它是 D-Bus 協定而不是
51+
函式庫,這就是它不用加新相依的原因:`pyatspi` 與
52+
`gi.repository.Atspi` 是發行版套件,裝不進 venv。
53+
54+
拿真的 bus 跟真的 GTK 程式一驗,當場抓到 D-Bus 客戶端的一個缺口:
55+
**它不會解有號整數**。portal 從來不需要,而 AT-SPI 的 extents 是四個
56+
**有號**值——因為主螢幕左邊(或上方)的螢幕上,視窗坐標是負的。
57+
58+
### BSD 與 arm64
59+
60+
`platform_wrapper` 對非 win/darwin/linux 一律丟「unknown operating
61+
system」,七個 X11 後端模組又各自帶一份同樣的 Linux 專屬守衛。
62+
新的 `utils/platform_id` 是唯一的判定點,`freebsd` job 在 runner 裡開
63+
真的 FreeBSD 14 VM,在真的 X server 上 import X11 模組並把游標移完讀回。
64+
`ubuntu-22.04-arm` 與 `windows-11-arm` 加進 smoke 矩陣。
65+
66+
367
## 本次更新 (2026-08-19) — Wayland 两个等人拍板的取舍,拍板了
468

569
`Progress.md` 上挂着两个 `DECIDE`:缺的不是活,是决定。两件事其实是同一个问题犯两次

‎README/WHATS_NEW_zh-TW.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,69 @@
11
# 本次更新 — AutoControl
22

3+
## 本次更新 (2026-08-20) — 嬣稱支援的平台,這回真的量過了
4+
5+
整套測試一直只在 `windows-2022` 跑,另加容器裡一次 Linux
6+
執行。macOS 只跑兩行指令。Wayland 有五個 job 對真的對等體
7+
讀回輸入;X11——兩條 Linux 路徑中更老、部署更廣的那一
8+
條——一個都沒有,而且套件裡每一條 X11 斷言都是對著
9+
`python-Xlib` 的 mock 做的。
10+
11+
**測試套件現在真的在它宣稱支援的平台上跑。**
12+
`pytest-headless` 改成 OS 矩陣;Linux 跑在真的 Xvfb 上而不是 Qt 的
13+
offscreen,因為 X11 後端在 import 時就連線,offscreen 會將正好要找的
14+
毛病蓋掉。第一輪就抓到兩個真的 macOS 缺陷:
15+
16+
- `write("\b")` 在 macOS 沒有任何按鍵路徑,會落到空白鍵
17+
fallback——要求退格,打出來的是空白。
18+
- `system_profiler` 對 Apple 自家裝置回的是 **符號式** vendor
19+
id(`apple_vendor_id`),而那個欄位文件上寫的是四位十六進位。
20+
21+
**X11 的輸入現在從真的客戶端讀回。** 新的 `x11-verification`
22+
job 跑在真的 Xvfb + 真的視窗管理員上,對照組來自受測對象以外的
23+
程式碼:`xev`、ImageMagick 的 `import`、`xdotool` 與 `xdpyinfo`。
24+
最值得點名的一項是 `synthetic NO`——`XSendEvent` 的事件帶的是
25+
`YES`,大多數 toolkit 會直接丟掉,所以一個患患停止驅動真實
26+
輸入的後端,在只數事件的檢查下仍然會全綠。
27+
28+
**macOS 在 CI 裡其實完全驗得了,跟一般假設相反。**
29+
`macos-14` runner **兩個 TCC 權限都給**:擷取回來的是真像素而不是
30+
被拒時的全黑矩形,`CGEventPost` 真的移得動游標且讀回完全相符,
31+
AX 樹也走得出真的元素。這是先量再斷言的,而且探針在期望表
32+
還是空的時候會拒絕通過。
33+
34+
### 視窗管理不再是 Windows 專屬
35+
36+
以前是:門面在 `sys.platform` 上分支,其他平台一律丟例外,
37+
23 個 `AC_*` 指令跟對應的 MCP 工具在 macOS 與 Linux 上都是死的。
38+
現在走平台縫:Win32、X11 的 EWMH、macOS 的 Quartz + 無障礙 API。
39+
40+
兩件只有真的視窗管理員才能披露的錯,第一版都錯了:
41+
42+
- **矩形是外框,不是客戶區。** Win32 的 `GetWindowRect`
43+
回的是外框,所有呼叫端都是照那個寫的。
44+
- **移動必須走 `_NET_MOVERESIZE_WINDOW`。** 在 reparenting
45+
視窗管理員下,客戶端自己的 x/y 是相對於外框的;對 openbox
46+
要 (300, 220),直接 `ConfigureWindow` 的結果是落在 (302, 260)。
47+
48+
### Linux 有無障礙後端了
49+
50+
之前完全沒有。新後端走 **AT-SPI2**——它是 D-Bus 協定而不是
51+
函式庫,這就是它不用加新相依的原因:`pyatspi` 與
52+
`gi.repository.Atspi` 是發行版套件,裝不進 venv。
53+
54+
拿真的 bus 跟真的 GTK 程式一驗,當場抓到 D-Bus 客戶端的一個缺口:
55+
**它不會解有號整數**。portal 從來不需要,而 AT-SPI 的 extents 是四個
56+
**有號**值——因為主螢幕左邊(或上方)的螢幕上,視窗坐標是負的。
57+
58+
### BSD 與 arm64
59+
60+
`platform_wrapper` 對非 win/darwin/linux 一律丟「unknown operating
61+
system」,七個 X11 後端模組又各自帶一份同樣的 Linux 專屬守衛。
62+
新的 `utils/platform_id` 是唯一的判定點,`freebsd` job 在 runner 裡開
63+
真的 FreeBSD 14 VM,在真的 X server 上 import X11 模組並把游標移完讀回。
64+
`ubuntu-22.04-arm` 與 `windows-11-arm` 加進 smoke 矩陣。
65+
66+
367
## 本次更新 (2026-08-19) — Wayland 兩個等人拍板的取捨,拍板了
468

569
`Progress.md` 上掛著兩個 `DECIDE`:缺的不是工,是決定。兩件事其實是同一個問題犯兩次

‎WHATS_NEW.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,112 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-08-20)
4+
5+
### The Platforms This Project Claims, Now Measured
6+
7+
The suite ran on `windows-2022` alone for its whole life, plus one Linux
8+
container run. macOS got two commands and nothing else. Wayland had five jobs
9+
reading input back off a real peer; X11 — the older and more widely deployed
10+
of the two Linux paths — had none, and every X11 assertion in the suite was
11+
made against a mock of `python-Xlib`.
12+
13+
**The suite now runs where the project says it runs.** `pytest-headless`
14+
became an OS matrix: Windows keeps all five Pythons, Linux and macOS carry the
15+
two ends of the range. Linux runs under a real Xvfb rather than Qt's offscreen
16+
platform, because the X11 backend opens a display at import time and offscreen
17+
would hide exactly the breakage this exists to find. It found two real macOS
18+
defects on the first run:
19+
20+
- `write("\b")` had no key route on macOS, so it fell through to the space
21+
fallback and typed a space where a backspace was asked for. X11 and Wayland
22+
both carry the raw character; macOS was the one that did not.
23+
- `system_profiler` reports a *symbolic* vendor id for Apple's own devices —
24+
`apple_vendor_id`, not a number — and that went straight into a field
25+
documented as four hex digits. Its leading `a` is a valid hex digit, so a
26+
lenient parse turns it into `000a`.
27+
28+
**X11 input is read back out of a real client.** A new `x11-verification` job
29+
runs against a real Xvfb server with a real window manager, taking ground
30+
truth from other codebases than the subject: `xev`, a real X client that
31+
prints every event delivered to its window; ImageMagick's `import` against a
32+
root painted two asymmetric colours; `xdotool` and `xdpyinfo`. The assertion
33+
worth naming is `synthetic NO` — `XSendEvent` traffic arrives with `YES` and
34+
is discarded by most toolkits, so a backend that quietly stopped driving real
35+
input would still pass any check that only counted events.
36+
37+
**macOS turned out to be fully testable in CI, contrary to the usual
38+
assumption.** A `macos-14` runner grants *both* Screen Recording and
39+
Accessibility: capture returns real pixels rather than the black rectangle a
40+
refusal produces, `CGEventPost` moves the cursor and the move reads back
41+
exactly, and the AX walk returns real elements. That was measured first and
42+
asserted second, and the probe still refuses to pass while its expectations
43+
table is empty.
44+
45+
### Window Management Is No Longer Windows-Only
46+
47+
It was: the facade branched on `sys.platform` and raised everywhere else,
48+
leaving 23 `AC_*` commands and their MCP tools dead on macOS and Linux. It now
49+
goes through a backend seam — Win32, EWMH over `python-Xlib` on X11, Quartz
50+
plus the accessibility API on macOS, and a null fallback that lists nothing
51+
and refuses actions with a reason.
52+
53+
Two things only a real window manager could show up were wrong first time:
54+
55+
- **The rectangle is the frame, not the client.** Win32's `GetWindowRect`
56+
returns the frame, and every caller is written against that, so reporting
57+
the client area was off by the decorations on X11 alone — silently, and by
58+
a different amount per window manager.
59+
- **A move has to go through `_NET_MOVERESIZE_WINDOW`.** Under a reparenting
60+
window manager a client's own x/y are relative to its frame, so a direct
61+
`ConfigureWindow` asks in the wrong coordinate space. Asking openbox for
62+
(300, 220) that way landed the window at (302, 260).
63+
64+
Refusals now raise a class that is both an `AutoControlException` and a
65+
`NotImplementedError`. The GUI tabs and the REST handler already catch the
66+
latter to say "not on this platform"; the executor catches the former, and a
67+
bare `NotImplementedError` slipped past every containment boundary — aborting
68+
a whole script where one action should have been reported as failed.
69+
70+
### Linux Has an Accessibility Backend
71+
72+
It had none — the selector fell through to the null one while the capability
73+
matrix claimed "backend tests" for Linux X11. The new backend speaks
74+
**AT-SPI2**, which is a D-Bus protocol rather than a library, and that is what
75+
makes it reachable without a new dependency: `pyatspi` and
76+
`gi.repository.Atspi` are distribution packages built against the system
77+
introspection data and cannot be installed into a virtual environment.
78+
79+
The D-Bus client written for the portal handshake moved from `linux_wayland/`
80+
to `utils/dbus_client/` to make that possible, and verifying the backend
81+
against a real bus and a real GTK application immediately found a gap in it:
82+
**it could not demarshal signed integers.** The portal never needed one, and
83+
AT-SPI reports a component's extents as four *signed* values, because a window
84+
on a monitor left of or above the primary one is at a negative coordinate — so
85+
the backend could read a tree but not where anything in it was.
86+
87+
Because AT-SPI is a bus rather than a display protocol, this is the one
88+
capability where Wayland is not the restricted case: the same bus serves both
89+
Linux sessions.
90+
91+
### The BSDs, and arm64
92+
93+
`platform_wrapper` refused to start on anything that was not
94+
win32/cygwin/msys, darwin or linux/linux2, and each of the seven X11 backend
95+
modules carried its own copy of the same Linux-only guard — so a FreeBSD,
96+
OpenBSD or NetBSD desktop, which runs the same X server and the same
97+
`python-Xlib`, could not import the package at all. `sys.platform` was being
98+
compared against literal lists in over a hundred places, so the fix is one
99+
place that decides: `utils/platform_id`, whose `is_x11_unix()` asks the
100+
question those guards were always trying to ask.
101+
102+
A `freebsd` job boots a real FreeBSD 14 VM inside the runner, imports the X11
103+
modules under a real X server, and moves the pointer and reads it back. It
104+
covers the platform layer rather than the whole package, because opencv has no
105+
FreeBSD wheel — a limit stated in the job rather than left to be discovered.
106+
`ubuntu-22.04-arm` and `windows-11-arm` join the smoke matrix; `macos-14` was
107+
already arm64.
108+
109+
3110
## What's new (2026-08-19)
4111

5112
### Two Wayland Judgement Calls, Settled

0 commit comments

Comments
 (0)