Skip to content

Commit 688dd2b

Browse files
committed
Give Linux an accessibility backend, over AT-SPI2
Linux had none: the selector fell through to the null backend while the capability matrix claimed "backend tests" for Linux X11. AT-SPI2 is a D-Bus protocol rather than a library, which is what makes it reachable without a new dependency — pyatspi and gi.repository.Atspi are distribution packages built against the system introspection data and cannot be installed into a virtual environment, so depending on them would be depending on something most users cannot get. The D-Bus client written for the portal handshake moves from linux_wayland/ to utils/dbus_client/ to make that possible: utils/ sits above the per-OS packages, so an accessibility backend reaching down into a platform backend to borrow its D-Bus code would invert the layering. The old path re-exports it, so the portal code is untouched; the client's own tests follow it, since a shim forwarding private names too would be a second copy of the surface to keep in step. Verifying it against a real bus and a real GTK application immediately found a gap in that client: it could not demarshal signed integers. The portal never needed one, and AT-SPI reports a component's extents as four signed values, because a window on a monitor left of or above the primary one is at a negative coordinate — so the backend could read a tree but not where anything in it was. The whole fixed-width numeric set marshals now, except UNIX_FD, which stays an error on purpose: it is an index into a descriptor array this client does not receive, so returning it would hand a caller a number that addresses nothing. The x11-verification job grew a third script for it, against zenity on a D-Bus-activated accessibility bus. Neither half can be mocked usefully — an application only appears on the bus if its toolkit bridge loaded, and the tree's shape is the toolkit's business. Because AT-SPI is a bus rather than a display protocol, this is the one capability where Wayland is not the restricted case: the same bus serves both Linux sessions.
1 parent c0ac50a commit 688dd2b

17 files changed

Lines changed: 1683 additions & 641 deletions

‎CLAUDE.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
AutoControl (`je_auto_control`) is a cross-platform GUI automation framework: mouse and keyboard control, image recognition, OCR, accessibility-tree and VLM element location, action scripting, and report generation behind one API. Backends: Windows (Win32 ctypes), macOS (pyobjc/Quartz), Linux X11 (python-Xlib), Linux Wayland (libei / ydotool), Android (adb), iOS (WebDriverAgent).
66

77
- **Package**: `je_auto_control` · **Python** ≥ 3.10 · **License**: MIT · **Author**: JE-Chen
8-
- **[architecture_explore.md](architecture_explore.md)** is the per-module map — read it before changing structure; it lists all 308 `utils/` subpackages, every GUI tab, and file-level tables for the large subsystems.
8+
- **[architecture_explore.md](architecture_explore.md)** is the per-module map — read it before changing structure; it lists all 309 `utils/` subpackages, every GUI tab, and file-level tables for the large subsystems.
99

1010
## Architecture
1111

@@ -18,7 +18,7 @@ AutoControl (`je_auto_control`) is a cross-platform GUI automation framework: mo
1818
| Template Method | `utils/generate_report/` | HTML / JSON / XML share collect → format → write. |
1919
| Backend seam | `backends/` under `accessibility`, `ocr`, `vision`, `llm`, `agent`, `hotkey`, `usb`, `usbip` | Abstract base + concrete impls + null fallback, so dependency-free environments still import. |
2020

21-
Layering: entry points (`cli.py`, `gui/`, socket / REST / MCP servers) → executor → `utils/` (308 headless subpackages) → `wrapper/` → per-OS backend.
21+
Layering: entry points (`cli.py`, `gui/`, socket / REST / MCP servers) → executor → `utils/` (309 headless subpackages) → `wrapper/` → per-OS backend.
2222

2323
## Development Commands
2424

@@ -68,7 +68,7 @@ The map is only useful while it matches the tree, so **update it in the same cha
6868

6969
Never adjust one by hand: the counts are `len(text.splitlines())` (what `wc -l` reports), and hand-editing is how the document ended up quoting the same subsystem at two sizes at once — most tables had been counting a phantom trailing line per file while §1 and §8 counted correctly.
7070

71-
- A new `utils/` subpackage needs a row in **exactly one** §5.4 theme table — the tables partition all 308 subpackages; appearing twice or not at all is a defect.
71+
- A new `utils/` subpackage needs a row in **exactly one** §5.4 theme table — the tables partition all 309 subpackages; appearing twice or not at all is a defect.
7272
- A new subsystem over ~1,000 lines also needs a file-level table in §5.4.17.
7373
- Keep the header's scan date, version, and branch current.
7474
- `README.md` and both translations under `README/` cite the same figures (command / subpackage / tab / MCP-tool / example counts) — update all three alongside the map.

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ desktop app; tab commands live in the window's **Actions** menu.
153153
| Diagnostics | `run_diagnostics` | `AC_diagnose` | Diagnostics |
154154
| Test-code generation | `generate_code` | — | — |
155155

156-
Beyond this table, `utils/` holds 308 headless packages covering assertions, resilience,
156+
Beyond this table, `utils/` holds 309 headless packages covering assertions, resilience,
157157
data quality, i18n auditing, redaction, governance, observability, and more. The full
158158
per-module map is in **[architecture_explore.md](architecture_explore.md)**.
159159

‎README/README_zh-CN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ python -m je_auto_control # 或:je_auto_control.start_autocontrol_gui
147147
| 系统诊断 | `run_diagnostics` | `AC_diagnose` | Diagnostics |
148148
| 测试代码生成 | `generate_code` | — | — |
149149

150-
除了这张表,`utils/` 下还有 308 个无头包,覆盖断言、韧性、数据质量、i18n 审计、脱敏、
150+
除了这张表,`utils/` 下还有 309 个无头包,覆盖断言、韧性、数据质量、i18n 审计、脱敏、
151151
治理、可观测性等等。完整的逐模块地图在 **[architecture_explore.md](../architecture_explore.md)**。
152152

153153
---

‎README/README_zh-TW.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ python -m je_auto_control # 或:je_auto_control.start_autocontrol_gui
147147
| 系統診斷 | `run_diagnostics` | `AC_diagnose` | Diagnostics |
148148
| 測試碼產生 | `generate_code` | — | — |
149149

150-
除了這張表,`utils/` 底下還有 308 個無頭套件,涵蓋斷言、韌性、資料品質、i18n 稽核、遮蔽、
150+
除了這張表,`utils/` 底下還有 309 個無頭套件,涵蓋斷言、韌性、資料品質、i18n 稽核、遮蔽、
151151
治理、可觀測性等等。完整的逐模組地圖在 **[architecture_explore.md](../architecture_explore.md)**。
152152

153153
---

‎architecture_explore.md‎

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,9 @@ iOS(WebDriverAgent)。核心能力是滑鼠/鍵盤控制、影像辨識、
1919

2020
| 指標 | 數值 |
2121
| --- | ---: |
22-
| Python 模組總數(含周邊子專案) | 1,022 |
23-
| 程式碼總行數 | 138,510 |
24-
| `je_auto_control/utils/` 子套件數 | 308 |
22+
| Python 模組總數(含周邊子專案) | 1,025 |
23+
| 程式碼總行數 | 139,017 |
24+
| `je_auto_control/utils/` 子套件數 | 309 |
2525
| `AC_*` 動作指令數(`known_commands()` 實測) | 773 |
2626
| 套件門面 `__all__` 公開名稱數 | 1,238 |
2727
| GUI 分頁數(`main_widget` 註冊) | 48 |
@@ -55,7 +55,7 @@ USB/IP 協定、Prometheus 指標),以維持這條輕相依基線。
5555
└───────────────────────────────┬──────────────────────────────────────────┘
5656
│
5757
┌───────────────────────────────▼──────────────────────────────────────────┐
58-
│ 能力層 utils/(308 個子套件,全部無 Qt 相依) │
58+
│ 能力層 utils/(309 個子套件,全部無 Qt 相依) │
5959
│ 影像辨識 │ OCR │ 無障礙樹 │ 定位自癒 │ AI/Agent │ 遠端桌面 │ USB │
6060
│ 報表觀測 │ 資料 │ 安全 │ 韌性 │ 系統整合 │ 排程觸發 │ 網路協定 │
6161
└───────────────────────────────┬──────────────────────────────────────────┘
@@ -226,14 +226,14 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。
226226
| `uinput/keyboard.py` | 33 | uinput 鍵盤後端,介面與 X11 版一致。 |
227227
| `uinput/mouse.py` | 116 | uinput 滑鼠後端。 |
228228

229-
#### Linux Wayland(`linux_wayland/`,17 檔/3,431 行)
229+
#### Linux Wayland(`linux_wayland/`,17 檔/2,830 行)
230230

231231
| 模組 | 行數 | 職責 |
232232
| --- | ---: | --- |
233233
| `_detect.py` | 77 | Wayland session 偵測與 CLI 工具探測。 |
234234
| `_ydotool_cli.py` | 134 | 判定安裝的是哪一代 ydotool 命令列,擋掉會靜默失效的 0.1.x(對本專案送的 argv 回傳 0 卻不送任何事件)。 |
235235
| `_ctypes_bind.py` | 75 | libei/liboeffis 共用的 ctypes 載入與 prototype 綁定。 |
236-
| `_dbus_client.py` | 625 | 只用標準函式庫的 D-Bus session bus 客戶端(連線/認證/`Hello`/`AddMatch`/一次方法呼叫/等訊號)。portal 的回應是**指名送給發出呼叫的那條連線**,所以訂閱與呼叫必須同一條連線——這是 `gdbus monitor` + `gdbus call` 兩個行程做不到的事。 |
236+
| `_dbus_client.py` | 24 | 只用標準函式庫的 D-Bus session bus 客戶端(連線/認證/`Hello`/`AddMatch`/一次方法呼叫/等訊號)。portal 的回應是**指名送給發出呼叫的那條連線**,所以訂閱與呼叫必須同一條連線——這是 `gdbus monitor` + `gdbus call` 兩個行程做不到的事。 |
237237
| `_select_input.py` | 85 | 決定使用原生 libei 或 CLI shim;`active_backend()` 是 keyboard/mouse 的唯一入口,`emitted()` 讓被拒絕的單次發送退回 CLI。 |
238238
| `_layout.py` | 83 | 版面原點的共用查詢。擷取與輸入不是同一個座標空間,差的就是這個原點:libei 的 region offset 是 `uint32`(描述不了負原點),`ydotool mousemove --absolute` 的原點是合成器夾取的那個角落——兩條路都要減掉它,所以放在這裡而不是各自複製。讀數快取一秒——擷取那一側刻意不快取,但 ydotool 每次絕對移動都會問,不快取等於每次移動多開一個 `wlr-randr` 行程。 |
239239
| `oeffis.py` | 196 | liboeffis 綁定:跑完 RemoteDesktop portal 交握,交出 EIS fd。 |
@@ -258,7 +258,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。
258258
| `ios/input.py` | 46 | iOS 觸控與按鍵原語。 |
259259
| `ios/screen.py` | 32 | iOS 裝置螢幕擷取與尺寸。 |
260260

261-
### 5.4 能力層 `utils/`(308 個子套件)
261+
### 5.4 能力層 `utils/`(309 個子套件)
262262

263263
以下依主題分組。每個子套件都是獨立可匯入的無頭模組,不含任何 Qt 相依。
264264

@@ -296,14 +296,15 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。
296296

297297
### 5.4.2 框架基礎設施
298298

299-
> 12 個套件、約 1,895 行。
299+
> 13 個套件、約 2,575 行。
300300
301301
| 模組 | 行數 | 職責 |
302302
| --- | ---: | --- |
303303
| `utils/callback/` | 200 | Observer 模式:`callback_executor` 以字串名觸發功能,執行後呼叫回呼 |
304304
| `utils/config_bundle/` | 399 | 使用者設定的單檔匯出/匯入 |
305305
| `utils/critical_exit/` | 97 | 監看緊急停止鍵的守護執行緒,用於中止失控腳本 |
306306
| `utils/diagnostics/` | 312 | 跨子系統的「一切正常嗎」健檢,附 `python -m` 進入點 |
307+
| `utils/dbus_client/` | 680 | 只用標準函式庫的 D-Bus session bus 客戶端。原本在 `linux_wayland/` 為 portal 交握而寫,AT-SPI 無障礙後端成為第二個使用者後搬到這裡(`utils/` 在分層上在各 OS 套件之上) |
307308
| `utils/exception/` | 208 | **例外階層根**。所有錯誤繼承 `AutoControlException`,加上集中式錯誤訊息字串(`exception_tags`) |
308309
| `utils/failure_bundle/` | 187 | 可攜、已遮蔽的失敗診斷 ZIP(截圖 + 診斷 + log 尾段) |
309310
| `utils/file_process/` | 26 | 目錄檔案列舉(`execute_dir` 的後端) |
@@ -432,12 +433,12 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。
432433

433434
### 5.4.7 無障礙樹與原生控制項
434435

435-
> 16 個套件、約 3,851 行。
436+
> 16 個套件、約 4,279 行。
436437
437438
| 模組 | 行數 | 職責 |
438439
| --- | ---: | --- |
439440
| `utils/a11y_audit/` | 355 | 以無障礙樹 + OCR 進行無障礙與 i18n 稽核 |
440-
| `utils/accessibility/` | 2,390 | 跨平台無障礙樹定位與錄製;Windows UIA/macOS AX/null 三後端。支援限定視窗(換搜尋起點,不是過濾)、逐節點可中斷走訪、`IUIAutomation2` 連線逾時、名稱子字串比對與排序、`control_get_state` 一次讀完值/勾選/選取/數值(密碼欄位不回內容) |
441+
| `utils/accessibility/` | 2,818 | 跨平台無障礙樹定位與錄製;Windows UIA/macOS AX/null 三後端。支援限定視窗(換搜尋起點,不是過濾)、逐節點可中斷走訪、`IUIAutomation2` 連線逾時、名稱子字串比對與排序、`control_get_state` 一次讀完值/勾選/選取/數值(密碼欄位不回內容) |
441442
| `utils/ax_events/` | 29 | 反應式 UIA 事件等待(focus-changed) |
442443
| `utils/ax_props/` | 44 | 讀取豐富 UIA 屬性(enabled/offscreen/help/status/快捷鍵) |
443444
| `utils/ax_text/` | 102 | 透過 UIA TextPattern 取得原生文字(讀取/尋找/選取/屬性) |
@@ -1021,20 +1022,20 @@ socket 預設綁 `127.0.0.1`;資源一律用 `with`。
10211022
| `utils/executor/` | 6 | 9,075 |
10221023
| `utils/usb/` | 17 | 4,247 |
10231024
| `je_auto_control/`(頂層 3 檔) | 3 | 2,366 |
1024-
| `utils/accessibility/` | 12 | 2,390 |
1025+
| `utils/accessibility/` | 13 | 2,818 |
10251026
| `wrapper/` | 3,026 | 3,013 新增 `window_backends/`:視窗管理的平台縫(`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`)。放在 `wrapper/` 而不是 `utils/`,因為它必須 import `windows/`、`linux_with_x11/`、`osx/`,而 `utils/` 在分層上在那三者之上。 |
10261027
| `windows/` | 23 | 1,995 |
10271028
| `utils/rest_api/` | 8 | 1,738 |
10281029
| `utils/agent/` | 8 | 1,250 |
10291030
| `linux_with_x11/` | 19 | 1,175 |
1030-
| `linux_wayland/` | 17 | 3,431 |
1031+
| `linux_wayland/` | 17 | 2,830 |
10311032
| `utils/triggers/` | 4 | 1,146 |
10321033
| `utils/ocr/` | 9 | 1,112 |
10331034
| `utils/usbip/` | 5 | 920 |
10341035
| `utils/assertion/` | 3 | 863 |
10351036
| `osx/` | 17 | 761 |
10361037
| `autocontrol-lsp/` | 8 | 744 |
10371038
| `utils/hotkey/` | 7 | 727 |
1038-
| 其餘模組(約 286 個 `utils/` 子套件 + `android/`/`ios/`/周邊小工具) | 685 | 49,278 |
1039-
| **總計** | **1,016** | **138,445** |
1039+
| 其餘模組(約 286 個 `utils/` 子套件 + `android/`/`ios/`/周邊小工具) | 687 | 49,958 |
1040+
| **總計** | **1,019** | **138,952** |
10401041

‎docker/Dockerfile.x11‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,12 +62,18 @@ ARG DEBIAN_FRONTEND=noninteractive
6262
# - imagemagick: `import -window root`, an independent grabber.
6363
# - openbox: a real EWMH window manager, without a desktop over the root.
6464
# - xterm: a real client to own a real window.
65+
# - at-spi2-core + dbus-x11: the accessibility bus. AT-SPI is a D-Bus
66+
# protocol, so the backend needs a session bus and the two services
67+
# D-Bus activates on it (org.a11y.Bus and the registry).
68+
# - zenity: a real GTK application, so there is an actual accessible
69+
# tree to walk. xterm exposes none — it is not a toolkit application.
6570
# - libgl1 + libglib2.0-0: opencv-python hard-requires libGL.so.1 and
6671
# libgthread-2.0.so.0 at import, so the package cannot even load without them.
6772
RUN apt-get update \
6873
&& apt-get install -y --no-install-recommends \
6974
xvfb xauth x11-utils x11-xserver-utils xdotool \
7075
imagemagick openbox xterm \
76+
at-spi2-core dbus-x11 zenity \
7177
libgl1 libglib2.0-0 \
7278
ca-certificates \
7379
&& rm -rf /var/lib/apt/lists/*
@@ -82,6 +88,7 @@ RUN pip install --no-cache-dir --only-binary :all: --upgrade "pip==26.0.1" \
8288

8389
COPY docker/x11_verify.py /opt/verify/x11_verify.py
8490
COPY docker/x11_window_verify.py /opt/verify/x11_window_verify.py
91+
COPY docker/x11_atspi_verify.py /opt/verify/x11_atspi_verify.py
8592
COPY docker/entrypoint-x11.sh /usr/local/bin/autocontrol-x11-verify
8693
RUN chmod +x /usr/local/bin/autocontrol-x11-verify
8794

@@ -97,7 +104,12 @@ RUN useradd --create-home --shell /bin/sh verify \
97104
&& chown -R verify:verify /app
98105
USER verify
99106

107+
# GTK3 only loads the accessibility bridge when it is asked to, and
108+
# what normally asks is a desktop setting this container has no
109+
# store for. Naming the module directly is what bridges the app.
100110
ENV HOME=/home/verify \
111+
GTK_MODULES=gail:atk-bridge \
112+
NO_AT_BRIDGE=0 \
101113
PYTHONUNBUFFERED=1 \
102114
DISPLAY=:99 \
103115
XDG_SESSION_TYPE=x11 \

‎docker/entrypoint-x11.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,14 @@
1919
# or the monitor layout cannot be brought up, this says so and fails.
2020
set -eu
2121

22+
# AT-SPI is a D-Bus protocol: without a session bus there is no accessibility
23+
# bus to activate, and the backend would correctly report itself unavailable
24+
# for a reason that is this container's fault rather than the code's. Re-exec
25+
# under one, once.
26+
if [ -z "${DBUS_SESSION_BUS_ADDRESS:-}" ]; then
27+
exec dbus-run-session -- "$0" "$@"
28+
fi
29+
2230
GEOMETRY="${SCREEN_GEOMETRY:-1280x800x24}"
2331
DISPLAY_NUM="${DISPLAY:-:99}"
2432

@@ -118,6 +126,12 @@ echo "window management against a real window manager"
118126
echo "========================================================================"
119127
python3 /opt/verify/x11_window_verify.py || total=$((total + $?))
120128

129+
echo
130+
echo "========================================================================"
131+
echo "accessibility against a real AT-SPI bus"
132+
echo "========================================================================"
133+
python3 /opt/verify/x11_atspi_verify.py || total=$((total + $?))
134+
121135
echo
122136
echo "========================================================================"
123137
echo "total failed checks: ${total}"

0 commit comments

Comments
 (0)