Skip to content

Commit 45aa9f9

Browse files
committed
Add ax_tree_walk: readable roles + addressable paths for a11y dumps
dump_accessibility_tree emits the platform's raw role (on Windows the bare UIA ControlType id, e.g. ControlType_50000) and a serialised dump carries no stable per-node identity. Add the pure post-processing it lacks: a ControlType-id to friendly-name table, whole-tree role humanization, a stable positional path per node (a pure stand-in for RuntimeId), and path resolution. AC_walk_tree is the readable counterpart to AC_a11y_dump.
1 parent 930388e commit 45aa9f9

11 files changed

Lines changed: 391 additions & 0 deletions

File tree

‎WHATS_NEW.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-06-24) — Readable, Addressable Accessibility Tree (role names + node paths)
4+
5+
Turn a raw `ControlType_50000` tree dump into readable roles with a stable path per node. Full reference: [`docs/source/Eng/doc/new_features/v183_features_doc.rst`](docs/source/Eng/doc/new_features/v183_features_doc.rst).
6+
7+
- **`control_type_name` / `humanize_role` / `humanize_tree` / `assign_node_paths` / `find_by_path`** (`AC_walk_tree`, `AC_humanize_role`): `dump_accessibility_tree` emits the platform's raw role (on Windows the bare UIA ControlType id, e.g. `ControlType_50000` for a button) and carries no stable per-node identity once serialised. This adds the pure post-processing it lacks: translate ControlType ids to friendly names, deep-copy a tree with every role humanised, stamp each node with a stable positional `path` (`"0.2.1"` — a pure stand-in for RuntimeId), and resolve a node back by path. `AC_walk_tree` is the readable counterpart to `AC_a11y_dump`. Pure-stdlib over `AXTreeNode`; unknown / non-UIA roles pass through unchanged. No `PySide6`.
8+
39
## What's new (2026-06-24) — Native Text Reading via the UIA TextPattern (document / selection / visible)
410

511
Read the text in multiline editors and document controls where ValuePattern returns nothing. Full reference: [`docs/source/Eng/doc/new_features/v182_features_doc.rst`](docs/source/Eng/doc/new_features/v182_features_doc.rst).
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
Readable, Addressable Accessibility Tree (role names + node paths)
2+
==================================================================
3+
4+
``dump_accessibility_tree`` emits nodes with the platform's *raw* role — on
5+
Windows that is the bare UI Automation ControlType id, e.g. ``"ControlType_50000"``
6+
for a button. That is unreadable, and a serialised dump carries no stable
7+
per-node identity (UIA RuntimeId needs the live element, which the dump has
8+
thrown away). ``ax_tree_walk`` adds the pure, platform-agnostic post-processing
9+
the dump lacks, composable on top of any ``dump_accessibility_tree`` output:
10+
11+
* :func:`control_type_name` / :func:`humanize_role` — translate a ControlType id
12+
(or ``"ControlType_NNNNN"`` / ``"NNNNN"`` string) to a friendly name,
13+
* :func:`humanize_tree` — a deep copy of the tree with every role humanised,
14+
* :func:`assign_node_paths` — a deep copy stamping each node with a stable
15+
positional ``path`` (``"0.2.1"``) — a pure stand-in for RuntimeId identity,
16+
* :func:`find_by_path` — resolve a node back from its path.
17+
18+
Pure-stdlib over ``AXTreeNode`` values; no device or backend access. Imports no
19+
``PySide6``.
20+
21+
Headless API
22+
------------
23+
24+
.. code-block:: python
25+
26+
from je_auto_control import (dump_accessibility_tree, humanize_tree,
27+
assign_node_paths, find_by_path, humanize_role)
28+
29+
humanize_role("ControlType_50000") # "Button"
30+
humanize_role(50004) # "Edit"
31+
32+
tree = assign_node_paths(humanize_tree(dump_accessibility_tree()))
33+
# every node now has a readable role and tree["attributes"]["path"]
34+
node = find_by_path(tree, "0.0.1") # re-resolve a node by its path
35+
36+
Unknown ids and non-UIA roles (``"AXApplication"``) pass through unchanged, so
37+
nothing is lost. The path is stable for a given tree shape, giving scripts /
38+
agents a deterministic handle to a node across a dump → act round-trip.
39+
40+
Executor commands
41+
-----------------
42+
43+
``AC_walk_tree`` (``app_name`` / ``max_results``) returns the humanised,
44+
path-stamped tree as a nested dict — the readable counterpart to
45+
``AC_a11y_dump``. ``AC_humanize_role`` (``role``) returns ``{"role": ...}``.
46+
Both are exposed as read-only ``ac_*`` MCP tools and as Script Builder commands
47+
under **Native UI**.
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
可讀且可定址的無障礙樹(角色名稱 + 節點路徑)
2+
=============================================
3+
4+
``dump_accessibility_tree`` 輸出的節點帶有平台的*原始*角色——在 Windows 上就是裸的
5+
UI Automation ControlType id,例如按鈕是 ``"ControlType_50000"``。這既難以閱讀,且序列化後的
6+
dump 不帶任何穩定的逐節點身分(UIA RuntimeId 需要存活的元素,而 dump 已將其丟棄)。
7+
``ax_tree_walk`` 補上 dump 所缺、純粹且跨平台的後處理,可疊加在任何
8+
``dump_accessibility_tree`` 輸出之上:
9+
10+
* :func:`control_type_name` / :func:`humanize_role` ——把 ControlType id(或
11+
``"ControlType_NNNNN"`` / ``"NNNNN"`` 字串)轉成友善名稱,
12+
* :func:`humanize_tree` ——回傳一份每個角色都已人性化的樹深拷貝,
13+
* :func:`assign_node_paths` ——回傳一份深拷貝,為每個節點蓋上穩定的位置 ``path``
14+
(``"0.2.1"``)——作為 RuntimeId 身分的純粹替代,
15+
* :func:`find_by_path` ——由 path 反解回節點。
16+
17+
純標準庫,針對 ``AXTreeNode`` 值運算;不存取裝置或後端。不匯入 ``PySide6``。
18+
19+
無頭 API
20+
--------
21+
22+
.. code-block:: python
23+
24+
from je_auto_control import (dump_accessibility_tree, humanize_tree,
25+
assign_node_paths, find_by_path, humanize_role)
26+
27+
humanize_role("ControlType_50000") # "Button"
28+
humanize_role(50004) # "Edit"
29+
30+
tree = assign_node_paths(humanize_tree(dump_accessibility_tree()))
31+
# 每個節點現在都有可讀角色與 tree["attributes"]["path"]
32+
node = find_by_path(tree, "0.0.1") # 由 path 重新解析節點
33+
34+
未知 id 與非 UIA 角色(``"AXApplication"``)原樣通過,故不會遺失任何資訊。path 對於給定的
35+
樹形狀是穩定的,讓腳本 / agent 在 dump → 操作的往返中對某節點有確定性的把手。
36+
37+
執行器指令
38+
----------
39+
40+
``AC_walk_tree``(``app_name`` / ``max_results``)以巢狀 dict 回傳已人性化、已蓋上 path 的樹
41+
——即 ``AC_a11y_dump`` 的可讀對應版本。``AC_humanize_role``(``role``)回傳
42+
``{"role": ...}``。兩者皆以唯讀 ``ac_*`` MCP 工具及 Script Builder 指令(位於 **Native UI**
43+
分類下)形式提供。

‎je_auto_control/__init__.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,11 @@
5757
from je_auto_control.utils.ax_text import (
5858
get_control_text, get_selected_text, get_visible_text,
5959
)
60+
# Readable / addressable a11y-tree post-processing (role names + node paths)
61+
from je_auto_control.utils.ax_tree_walk import (
62+
assign_node_paths, control_type_name, find_by_path, humanize_role,
63+
humanize_tree,
64+
)
6065
# VLM element locator (headless)
6166
from je_auto_control.utils.vision import (
6267
VLMNotAvailableError, click_by_description, locate_by_description,
@@ -1622,6 +1627,8 @@ def start_autocontrol_gui(*args, **kwargs):
16221627
"select_control_item", "control_range", "set_control_range",
16231628
"scroll_control_into_view",
16241629
"get_control_text", "get_selected_text", "get_visible_text",
1630+
"control_type_name", "humanize_role", "humanize_tree",
1631+
"assign_node_paths", "find_by_path",
16251632
# VLM locator
16261633
"VLMNotAvailableError", "locate_by_description", "click_by_description",
16271634
"verify_description",

‎je_auto_control/gui/script_builder/command_schema.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1538,6 +1538,18 @@ def _add_native_control_specs(specs: List[CommandSpec]) -> None:
15381538
fields=fields,
15391539
description="Read only the on-screen text via TextPattern.GetVisibleRanges.",
15401540
))
1541+
specs.append(CommandSpec(
1542+
"AC_walk_tree", "Native UI", "Walk Accessibility Tree",
1543+
fields=(FieldSpec("app_name", FieldType.STRING, optional=True),
1544+
FieldSpec("max_results", FieldType.INT, optional=True,
1545+
default=500)),
1546+
description="Dump the a11y tree with friendly roles + a path per node.",
1547+
))
1548+
specs.append(CommandSpec(
1549+
"AC_humanize_role", "Native UI", "Humanize UIA Role",
1550+
fields=(FieldSpec("role", FieldType.STRING),),
1551+
description="Translate a raw UIA role (ControlType_50000) to a name.",
1552+
))
15411553

15421554

15431555
def _add_misc_specs(specs: List[CommandSpec]) -> None:
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
"""Readable, addressable accessibility-tree post-processing (role names + node paths)."""
2+
from je_auto_control.utils.ax_tree_walk.ax_tree_walk import (
3+
assign_node_paths, control_type_name, find_by_path, humanize_role,
4+
humanize_tree,
5+
)
6+
7+
__all__ = [
8+
"control_type_name", "humanize_role", "humanize_tree",
9+
"assign_node_paths", "find_by_path",
10+
]
Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
"""Make an accessibility-tree dump readable and addressable.
2+
3+
``dump_accessibility_tree`` emits nodes with the platform's *raw* role —
4+
on Windows that is the bare UI Automation ControlType id, e.g.
5+
``"ControlType_50000"`` for a button. That is unreadable, and the dump
6+
carries no stable per-node identity (UIA RuntimeId needs the live element,
7+
which a serialised dump has thrown away). ``ax_tree_walk`` adds the pure,
8+
platform-agnostic post-processing the dump lacks:
9+
10+
* :func:`control_type_name` / :func:`humanize_role` — translate a UIA
11+
ControlType id (or ``"ControlType_NNNNN"`` string) to a friendly name,
12+
* :func:`humanize_tree` — a deep copy of the tree with every role humanised,
13+
* :func:`assign_node_paths` — a deep copy stamping each node with a stable
14+
positional ``path`` (``"0.2.1"``) — a pure stand-in for RuntimeId identity,
15+
* :func:`find_by_path` — resolve a node back from its path.
16+
17+
Pure-stdlib over :class:`AXTreeNode` values; no device or backend access, no
18+
``PySide6``. Compose it on top of any ``dump_accessibility_tree`` output.
19+
"""
20+
from typing import Optional, Union
21+
22+
from je_auto_control.utils.accessibility.tree import AXTreeNode
23+
24+
# UIA ControlType ids → friendly names (UIAutomationClient ControlTypeId range).
25+
_CONTROL_TYPE_NAMES = {
26+
50000: "Button", 50001: "Calendar", 50002: "CheckBox", 50003: "ComboBox",
27+
50004: "Edit", 50005: "Hyperlink", 50006: "Image", 50007: "ListItem",
28+
50008: "List", 50009: "Menu", 50010: "MenuBar", 50011: "MenuItem",
29+
50012: "ProgressBar", 50013: "RadioButton", 50014: "ScrollBar",
30+
50015: "Slider", 50016: "Spinner", 50017: "StatusBar", 50018: "Tab",
31+
50019: "TabItem", 50020: "Text", 50021: "ToolBar", 50022: "ToolTip",
32+
50023: "Tree", 50024: "TreeItem", 50025: "Custom", 50026: "Group",
33+
50027: "Thumb", 50028: "DataGrid", 50029: "DataItem", 50030: "Document",
34+
50031: "SplitButton", 50032: "Window", 50033: "Pane", 50034: "Header",
35+
50035: "HeaderItem", 50036: "Table", 50037: "TitleBar", 50038: "Separator",
36+
50039: "SemanticZoom", 50040: "AppBar",
37+
}
38+
_ROLE_PREFIX = "ControlType_"
39+
40+
41+
def control_type_name(control_type: int) -> str:
42+
"""Return the friendly name for a UIA ControlType id (e.g. ``50000`` → ``Button``).
43+
44+
Unknown ids fall back to ``"ControlType_<id>"`` so nothing is lost.
45+
"""
46+
cid = int(control_type)
47+
return _CONTROL_TYPE_NAMES.get(cid, f"{_ROLE_PREFIX}{cid}")
48+
49+
50+
def humanize_role(role: Union[str, int]) -> str:
51+
"""Map a raw UIA role to a friendly name.
52+
53+
Accepts an int id (``50000``), a ``"ControlType_50000"`` string, or a bare
54+
``"50000"`` string. Any role that is not a recognised ControlType — already
55+
friendly (``"Button"``) or a non-UIA role (``"AXApplication"``) — is returned
56+
unchanged.
57+
"""
58+
if isinstance(role, int):
59+
return control_type_name(role)
60+
text = str(role)
61+
digits = text[len(_ROLE_PREFIX):] if text.startswith(_ROLE_PREFIX) else text
62+
if digits.isdigit():
63+
return control_type_name(int(digits))
64+
return text
65+
66+
67+
def humanize_tree(node: AXTreeNode) -> AXTreeNode:
68+
"""Return a deep copy of ``node`` with every role run through :func:`humanize_role`."""
69+
return AXTreeNode(
70+
name=node.name, role=humanize_role(node.role), bounds=node.bounds,
71+
app_name=node.app_name, process_id=node.process_id,
72+
attributes=dict(node.attributes),
73+
children=[humanize_tree(child) for child in node.children],
74+
)
75+
76+
77+
def assign_node_paths(node: AXTreeNode, prefix: str = "0") -> AXTreeNode:
78+
"""Return a deep copy stamping each node with a stable positional ``path``.
79+
80+
The root is ``"0"``; its third child is ``"0.2"``, and so on. The path is a
81+
pure stand-in for a RuntimeId: stable for a given tree shape and re-resolvable
82+
with :func:`find_by_path`. Stored under ``attributes["path"]``.
83+
"""
84+
attributes = dict(node.attributes)
85+
attributes["path"] = prefix
86+
children = [assign_node_paths(child, f"{prefix}.{index}")
87+
for index, child in enumerate(node.children)]
88+
return AXTreeNode(
89+
name=node.name, role=node.role, bounds=node.bounds,
90+
app_name=node.app_name, process_id=node.process_id,
91+
attributes=attributes, children=children,
92+
)
93+
94+
95+
def find_by_path(root: AXTreeNode, path: str) -> Optional[AXTreeNode]:
96+
"""Resolve the node addressed by ``path`` (e.g. ``"0.2.1"``); ``None`` if absent."""
97+
parts = str(path).split(".")
98+
if not parts or parts[0] != "0":
99+
return None
100+
node = root
101+
for part in parts[1:]:
102+
if not part.isdigit():
103+
return None
104+
index = int(part)
105+
if index >= len(node.children):
106+
return None
107+
node = node.children[index]
108+
return node

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,23 @@ def _a11y_dump(app_name: Optional[str] = None,
182182
).to_dict()
183183

184184

185+
def _walk_tree(app_name: Optional[str] = None,
186+
max_results: int = 500) -> Dict[str, Any]:
187+
"""Executor adapter: dump the a11y tree with friendly roles + node paths."""
188+
from je_auto_control.utils.accessibility import dump_accessibility_tree
189+
from je_auto_control.utils.ax_tree_walk import (
190+
assign_node_paths, humanize_tree)
191+
root = dump_accessibility_tree(app_name=app_name,
192+
max_results=int(max_results))
193+
return assign_node_paths(humanize_tree(root)).to_dict()
194+
195+
196+
def _humanize_role(role: str) -> Dict[str, Any]:
197+
"""Executor adapter: translate a raw UIA role to a friendly name."""
198+
from je_auto_control.utils.ax_tree_walk import humanize_role
199+
return {"role": humanize_role(role)}
200+
201+
185202
def _a11y_record_start(app_name: Optional[str] = None,
186203
poll_interval_s: float = 0.25,
187204
min_movement_px: int = 8) -> Dict[str, Any]:
@@ -6108,6 +6125,8 @@ def __init__(self):
61086125
"AC_a11y_find": _a11y_find_as_dict,
61096126
"AC_a11y_click": click_accessibility_element,
61106127
"AC_a11y_dump": _a11y_dump,
6128+
"AC_walk_tree": _walk_tree,
6129+
"AC_humanize_role": _humanize_role,
61116130
"AC_control_get_value": _control_get_value,
61126131
"AC_control_set_value": _control_set_value,
61136132
"AC_control_invoke": _control_invoke,

‎je_auto_control/utils/mcp_server/tools/_factories.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1200,6 +1200,29 @@ def a11y_tree_tools() -> List[MCPTool]:
12001200
handler=h.a11y_dump,
12011201
annotations=READ_ONLY,
12021202
),
1203+
MCPTool(
1204+
name="ac_walk_tree",
1205+
description=("Dump the accessibility tree like ac_a11y_dump but with "
1206+
"friendly role names (UIA 'ControlType_50000' → "
1207+
"'Button') and a stable positional 'path' per node "
1208+
"(addressable via the path attribute)."),
1209+
input_schema=schema({
1210+
"app_name": {"type": "string"},
1211+
"max_results": {"type": "integer"},
1212+
}),
1213+
handler=h.walk_tree,
1214+
annotations=READ_ONLY,
1215+
),
1216+
MCPTool(
1217+
name="ac_humanize_role",
1218+
description=("Translate a raw UIA role ('ControlType_50000' / "
1219+
"'50000') to a friendly name: {role}. Unknown / "
1220+
"already-friendly roles pass through unchanged."),
1221+
input_schema=schema({"role": {"type": "string"}},
1222+
required=["role"]),
1223+
handler=h.humanize_role,
1224+
annotations=READ_ONLY,
1225+
),
12031226
MCPTool(
12041227
name="ac_a11y_record_start",
12051228
description=("Start the polling accessibility recorder. "

‎je_auto_control/utils/mcp_server/tools/_handlers.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2887,6 +2887,16 @@ def a11y_dump(app_name: Optional[str] = None,
28872887
).to_dict()
28882888

28892889

2890+
def walk_tree(app_name=None, max_results: int = 500):
2891+
from je_auto_control.utils.executor.action_executor import _walk_tree
2892+
return _walk_tree(app_name, max_results)
2893+
2894+
2895+
def humanize_role(role):
2896+
from je_auto_control.utils.executor.action_executor import _humanize_role
2897+
return _humanize_role(role)
2898+
2899+
28902900
def a11y_record_start(app_name: Optional[str] = None,
28912901
poll_interval_s: float = 0.25,
28922902
min_movement_px: int = 8) -> Dict[str, Any]:

0 commit comments

Comments
 (0)