Add canonical computer-use action schema (Anthropic/OpenAI -> AC_*)

Author: JE-Chen

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 标准化 Computer-Use 动作结构
+
+把 Anthropic / OpenAI agent 动作桥接到 AutoControl 命令。完整参考:[`docs/source/Zh/doc/new_features/v151_features_doc.rst`](../docs/source/Zh/doc/new_features/v151_features_doc.rst)。
+
+- **`from_anthropic` / `from_openai_cua` / `to_ac_command` / `canonical_action`**(`AC_cua_command`):`tool_use_schema` 导出 AC_* 签章、`coordinate_space` 缩放——两者都不*正规化进来的动作载荷*。Anthropic 发出 `{action:"left_click", coordinate:[x,y]}`、OpenAI CUA 发出 `{type:"click", x, y, button}`;这些转接器把两者对应为标准动作再对应为可执行的 `[AC_*, params]`(含可选坐标空间 `scale`)。纯标准库、可无头测试;执行器命令对任一来源返回 `{canonical, command}`。
+
 ## 本次更新 (2026-06-23) — 窗口客户区几何
 
 不论标题栏 / 边框,点击窗口*内部*。完整参考:[`docs/source/Zh/doc/new_features/v150_features_doc.rst`](../docs/source/Zh/doc/new_features/v150_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 標準化 Computer-Use 動作結構
+
+把 Anthropic / OpenAI agent 動作橋接到 AutoControl 命令。完整參考:[`docs/source/Zh/doc/new_features/v151_features_doc.rst`](../docs/source/Zh/doc/new_features/v151_features_doc.rst)。
+
+- **`from_anthropic` / `from_openai_cua` / `to_ac_command` / `canonical_action`**(`AC_cua_command`):`tool_use_schema` 匯出 AC_* 簽章、`coordinate_space` 縮放——兩者都不*正規化進來的動作酬載*。Anthropic 發出 `{action:"left_click", coordinate:[x,y]}`、OpenAI CUA 發出 `{type:"click", x, y, button}`;這些轉接器把兩者對應為標準動作再對應為可執行的 `[AC_*, params]`(含選用座標空間 `scale`)。純標準函式庫、可無頭測試;執行器命令對任一來源回傳 `{canonical, command}`。
+
 ## 本次更新 (2026-06-23) — 視窗客戶區幾何
 
 不論標題列 / 邊框,點擊視窗*內部*。完整參考:[`docs/source/Zh/doc/new_features/v150_features_doc.rst`](../docs/source/Zh/doc/new_features/v150_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Canonical Computer-Use Action Schema
+
+Bridge Anthropic / OpenAI agent actions to AutoControl commands. Full reference: [`docs/source/Eng/doc/new_features/v151_features_doc.rst`](docs/source/Eng/doc/new_features/v151_features_doc.rst).
+
+- **`from_anthropic` / `from_openai_cua` / `to_ac_command` / `canonical_action`** (`AC_cua_command`): `tool_use_schema` exports AC_* signatures and `coordinate_space` rescales — neither *normalizes an inbound action payload*. Anthropic emits `{action:"left_click", coordinate:[x,y]}`, OpenAI CUA emits `{type:"click", x, y, button}`; these adapters map both to a canonical action and then to a runnable `[AC_*, params]` (with optional coordinate-space `scale`). Pure-stdlib, headless-testable; the executor command returns `{canonical, command}` for any source.
+
 ## What's new (2026-06-23) — Window Client-Area Geometry
 
 Click *inside* a window regardless of its title bar / borders. Full reference: [`docs/source/Eng/doc/new_features/v150_features_doc.rst`](docs/source/Eng/doc/new_features/v150_features_doc.rst).

docs/source/Eng/doc/new_features/v151_features_doc.rst

@@ -0,0 +1,45 @@
+Canonical Computer-Use Action Schema
+====================================
+
+``tool_use_schema`` exports the AC_* command *signatures* as tool definitions and
+``coordinate_space`` rescales a model grid — but neither *normalizes an inbound action
+payload*. Anthropic's computer-use tool emits ``{action:"left_click",
+coordinate:[x,y]}``, OpenAI's CUA emits ``{type:"click", x, y, button}`` — there was no
+adapter mapping these heterogeneous shapes onto a canonical action and then onto a
+runnable AC_* command, so integrators hand-wrote the glue.
+
+Pure-stdlib dict mapping (an optional ``scale`` callable applies coordinate-space
+rescaling), fully headless-testable. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (from_anthropic, from_openai_cua, to_ac_command,
+                                 canonical_action)
+
+    # Anthropic agent output -> canonical -> runnable AC action.
+    canonical = from_anthropic({"action": "left_click", "coordinate": [120, 80]})
+    command = to_ac_command(canonical)
+    # -> ["AC_click_mouse", {"mouse_keycode": "mouse_left", "x": 120, "y": 80}]
+
+    # OpenAI CUA, with model->physical coordinate rescaling.
+    cmd = to_ac_command(from_openai_cua({"type": "scroll", "x": 5, "y": 6,
+                                         "scroll_y": 120}),
+                        scale=lambda x, y: (x * 2, y * 2))
+
+``from_anthropic`` / ``from_openai_cua`` map each provider's payload to a canonical
+``{type, x, y, text, …}`` (clicks, double/right/middle click, move, type, key, scroll,
+screenshot). ``to_ac_command`` maps a canonical action to a ``[command_name, params]``
+AC action (``AC_click_mouse`` / ``AC_set_mouse_position`` / ``AC_write`` / ``AC_hotkey``
+/ ``AC_mouse_scroll`` / ``AC_screenshot``), applying ``scale`` to coordinates; an
+unmapped type raises ``AutoControlActionException``. ``canonical_action`` builds a
+canonical dict directly.
+
+Executor command
+----------------
+
+``AC_cua_command`` normalizes a ``payload`` from ``source`` (``anthropic`` / ``openai``
+/ ``canonical``) and returns ``{canonical, command}``. It is exposed as the MCP tool
+``ac_cua_command`` and as a Script Builder command under **Native UI**.

docs/source/Eng/eng_index.rst

@@ -173,6 +173,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v148_features_doc
    doc/new_features/v149_features_doc
    doc/new_features/v150_features_doc
+   doc/new_features/v151_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v151_features_doc.rst

@@ -0,0 +1,39 @@
+標準化 Computer-Use 動作結構
+============================
+
+``tool_use_schema`` 把 AC_* 命令*簽章*匯出為工具定義,``coordinate_space`` 縮放模型網格——但兩者都不*正規化進來的
+動作酬載*。Anthropic 的 computer-use 工具發出 ``{action:"left_click", coordinate:[x,y]}``,OpenAI 的 CUA 發出
+``{type:"click", x, y, button}``——先前沒有把這些異質形狀對應到標準動作、再對應到可執行 AC_* 命令的轉接器,
+整合者只能手寫膠水程式。
+
+純標準函式庫的字典對應(選用 ``scale`` callable 套用座標空間縮放),完全可無頭測試。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (from_anthropic, from_openai_cua, to_ac_command,
+                                 canonical_action)
+
+    # Anthropic agent 輸出 -> 標準 -> 可執行 AC 動作。
+    canonical = from_anthropic({"action": "left_click", "coordinate": [120, 80]})
+    command = to_ac_command(canonical)
+    # -> ["AC_click_mouse", {"mouse_keycode": "mouse_left", "x": 120, "y": 80}]
+
+    # OpenAI CUA,含 模型->實體 座標縮放。
+    cmd = to_ac_command(from_openai_cua({"type": "scroll", "x": 5, "y": 6,
+                                         "scroll_y": 120}),
+                        scale=lambda x, y: (x * 2, y * 2))
+
+``from_anthropic`` / ``from_openai_cua`` 把各供應商酬載對應為標準 ``{type, x, y, text, …}``(click、double/right/
+middle click、move、type、key、scroll、screenshot)。``to_ac_command`` 把標準動作對應為 ``[command_name, params]``
+AC 動作(``AC_click_mouse`` / ``AC_set_mouse_position`` / ``AC_write`` / ``AC_hotkey`` / ``AC_mouse_scroll`` /
+``AC_screenshot``),並對座標套用 ``scale``;無法對應的類型會丟出 ``AutoControlActionException``。``canonical_action``
+直接建立標準字典。
+
+執行器命令
+----------
+
+``AC_cua_command`` 從 ``source``(``anthropic`` / ``openai`` / ``canonical``)正規化 ``payload`` 並回傳
+``{canonical, command}``。它以 MCP 工具 ``ac_cua_command`` 以及 Script Builder 中 **Native UI** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -173,6 +173,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v148_features_doc
    doc/new_features/v149_features_doc
    doc/new_features/v150_features_doc
+   doc/new_features/v151_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -361,6 +361,10 @@
 from je_auto_control.utils.window_geometry import (
     client_point, client_to_screen, frame_insets, get_client_rect,
 )
+# Canonical computer-use action schema (normalize Anthropic / OpenAI -> AC_*)
+from je_auto_control.utils.cua_action import (
+    canonical_action, from_anthropic, from_openai_cua, to_ac_command,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1235,6 +1239,10 @@ def start_autocontrol_gui(*args, **kwargs):
     "client_to_screen",
     "get_client_rect",
     "client_point",
+    "canonical_action",
+    "from_anthropic",
+    "from_openai_cua",
+    "to_ac_command",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -2910,6 +2910,16 @@ def _add_screen_state_specs(specs: List[CommandSpec]) -> None:
 
 
 def _add_set_of_marks_specs(specs: List[CommandSpec]) -> None:
+    specs.append(CommandSpec(
+        "AC_cua_command", "Native UI", "Computer-Use: Map Action",
+        fields=(
+            FieldSpec("payload", FieldType.STRING,
+                      placeholder='{"action":"left_click","coordinate":[x,y]}'),
+            FieldSpec("source", FieldType.ENUM, optional=True, default="canonical",
+                      choices=("canonical", "anthropic", "openai")),
+        ),
+        description="Map an Anthropic / OpenAI computer-use action to an AC command.",
+    ))
     specs.append(CommandSpec(
         "AC_mark_screen", "Native UI", "Set-of-Marks: Number Elements",
         fields=(

je_auto_control/utils/cua_action/__init__.py

@@ -0,0 +1,7 @@
+"""Canonical computer-use action schema (normalize Anthropic / OpenAI -> AC_*)."""
+from je_auto_control.utils.cua_action.cua_action import (
+    canonical_action, from_anthropic, from_openai_cua, to_ac_command,
+)
+
+__all__ = ["canonical_action", "from_anthropic", "from_openai_cua",
+           "to_ac_command"]