Add grid/table cell addressing by row/col

Author: JE-Chen

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 表格 / 网格单元格定位
+
+依(行、列)从单元格边界框定位表格单元格。完整参考:[`docs/source/Zh/doc/new_features/v125_features_doc.rst`](../docs/source/Zh/doc/new_features/v125_features_doc.rst)。
+
+- **`cluster_grid` / `locate_cell`**(`AC_grid_cell`):`anchor_locator` 处理成对关系,但无法定位二维网格。给定单元格边界框(来自 `locate_all_image` / `find_text_matches`),本功能将其分群为行(依中心 y 在 `row_tolerance` 内)与列(依中心 x),并返回 0 起算 `(row, col)` 单元格的中心——可直接点击。纯分群、完全无头可测。
+
 ## 本次更新 (2026-06-23) — 锚点序数与全部定位
 
 挑选第 N 个锚点相对匹配,或列举全部。完整参考:[`docs/source/Zh/doc/new_features/v124_features_doc.rst`](../docs/source/Zh/doc/new_features/v124_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 表格 / 格線儲存格定位
+
+依(列、欄)從儲存格邊界框定位表格儲存格。完整參考:[`docs/source/Zh/doc/new_features/v125_features_doc.rst`](../docs/source/Zh/doc/new_features/v125_features_doc.rst)。
+
+- **`cluster_grid` / `locate_cell`**(`AC_grid_cell`):`anchor_locator` 處理成對關係,但無法定位二維格線。給定儲存格邊界框(來自 `locate_all_image` / `find_text_matches`),本功能將其分群為列(依中心 y 在 `row_tolerance` 內)與欄(依中心 x),並回傳 0 起算 `(row, col)` 儲存格的中心——可直接點擊。純分群、完全無頭可測。
+
 ## 本次更新 (2026-06-23) — 錨點序數與全部定位
 
 挑選第 N 個錨點相對比對,或列舉全部。完整參考:[`docs/source/Zh/doc/new_features/v124_features_doc.rst`](../docs/source/Zh/doc/new_features/v124_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Grid / Table Cell Addressing
+
+Address a table cell by (row, column) from cell bounding boxes. Full reference: [`docs/source/Eng/doc/new_features/v125_features_doc.rst`](docs/source/Eng/doc/new_features/v125_features_doc.rst).
+
+- **`cluster_grid` / `locate_cell`** (`AC_grid_cell`): `anchor_locator` does pairwise relations but nothing addresses a 2-D grid. Given the cell bounding boxes (from `locate_all_image` / `find_text_matches`), this clusters them into rows (by centre-y within `row_tolerance`) and columns (by centre-x) and returns the centre of the 0-based `(row, col)` cell — ready to click. Pure clustering, fully headless-testable.
+
 ## What's new (2026-06-23) — Anchor Ordinal & Locate-All
 
 Pick the Nth anchor-relative match, or enumerate them all. Full reference: [`docs/source/Eng/doc/new_features/v124_features_doc.rst`](docs/source/Eng/doc/new_features/v124_features_doc.rst).

docs/source/Eng/doc/new_features/v125_features_doc.rst

@@ -0,0 +1,41 @@
+Grid / Table Cell Addressing
+============================
+
+``anchor_locator`` does pairwise spatial relations (target *near* / *below* an
+anchor) but nothing addresses a 2-D grid — "the cell at row 3, column 2" of a
+table. Given the bounding boxes of the cells (from an image or OCR enumeration,
+e.g. ``locate_all_image`` / ``find_text_matches``), this clusters them into rows
+and columns and returns the requested cell's centre.
+
+The clustering and lookup are pure (boxes in, grid / cell out) and fully
+unit-testable; the box enumeration stays the caller's job, so nothing here needs a
+real screen. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import cluster_grid, locate_cell
+
+    boxes = [(10, 100, 20, 10), (110, 100, 20, 10), (210, 100, 20, 10),
+             (10, 200, 20, 10), (110, 200, 20, 10), (210, 200, 20, 10)]
+
+    locate_cell(boxes, row=1, col=2)
+    # {'found': True, 'center': [220, 205], 'box': [210, 200, 20, 10],
+    #  'row': 1, 'col': 2, 'rows': 2, 'cols': 3}
+
+    cluster_grid(boxes)   # rows top-to-bottom, cells left-to-right
+
+``cluster_grid`` sorts the boxes by centre-y, starts a new row when the gap
+exceeds ``row_tolerance``, and orders each row's cells by centre-x.
+``locate_cell`` returns the centre (ready to click) of the 0-based ``(row, col)``
+cell, or ``{found: False, reason}`` when the index is out of range.
+
+Executor commands
+-----------------
+
+``AC_grid_cell`` takes ``boxes`` (a JSON ``[[x, y, w, h], ...]`` list, e.g. from a
+prior ``AC_locate_all_image`` step) plus ``row`` / ``col`` / ``row_tolerance`` and
+returns the cell dict. It is exposed as the MCP tool ``ac_grid_cell`` and as a
+Script Builder command under **Mouse**.

docs/source/Eng/eng_index.rst

@@ -147,6 +147,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v122_features_doc
    doc/new_features/v123_features_doc
    doc/new_features/v124_features_doc
+   doc/new_features/v125_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v125_features_doc.rst

@@ -0,0 +1,36 @@
+表格 / 格線儲存格定位
+====================
+
+``anchor_locator`` 處理成對的空間關係(目標在錨點*附近* / *下方*),但無法定位二維格線——表格中「第 3 列、
+第 2 欄的儲存格」。給定各儲存格的邊界框(來自影像或 OCR 列舉,例如 ``locate_all_image`` / ``find_text_matches``),
+本功能將其分群為列與欄,並回傳所求儲存格的中心。
+
+分群與查詢皆為純函式(框進、格線 / 儲存格出),完全可單元測試;框的列舉仍由呼叫端負責,因此此處不需真實螢幕。
+不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import cluster_grid, locate_cell
+
+    boxes = [(10, 100, 20, 10), (110, 100, 20, 10), (210, 100, 20, 10),
+             (10, 200, 20, 10), (110, 200, 20, 10), (210, 200, 20, 10)]
+
+    locate_cell(boxes, row=1, col=2)
+    # {'found': True, 'center': [220, 205], 'box': [210, 200, 20, 10],
+    #  'row': 1, 'col': 2, 'rows': 2, 'cols': 3}
+
+    cluster_grid(boxes)   # 列由上到下、儲存格由左到右
+
+``cluster_grid`` 依中心 y 排序框,當間距超過 ``row_tolerance`` 時開始新的一列,並將每列的儲存格依中心 x 排序。
+``locate_cell`` 回傳 0 起算 ``(row, col)`` 儲存格的中心(可直接點擊),索引超出範圍時回傳
+``{found: False, reason}``。
+
+執行器命令
+----------
+
+``AC_grid_cell`` 接受 ``boxes``(JSON ``[[x, y, w, h], ...]`` 清單,例如來自前一個 ``AC_locate_all_image`` 步驟)
+以及 ``row`` / ``col`` / ``row_tolerance``,並回傳儲存格 dict。它以 MCP 工具 ``ac_grid_cell`` 以及 Script Builder
+中 **Mouse** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -147,6 +147,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v122_features_doc
    doc/new_features/v123_features_doc
    doc/new_features/v124_features_doc
+   doc/new_features/v125_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -271,6 +271,8 @@
 from je_auto_control.utils.modifier_state import (
     hold_modifiers, plan_with_modifiers,
 )
+# Address a table / grid cell by (row, column) from bounding boxes
+from je_auto_control.utils.grid_locator import cluster_grid, locate_cell
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1058,6 +1060,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "unicode_code_units",
     "hold_modifiers",
     "plan_with_modifiers",
+    "cluster_grid",
+    "locate_cell",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -800,6 +800,17 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Move the pointer by (dx, dy) from its current position.",
     ))
+    specs.append(CommandSpec(
+        "AC_grid_cell", "Mouse", "Grid Cell (row / col)",
+        fields=(
+            FieldSpec("boxes", FieldType.STRING,
+                      placeholder="[[10,100,20,10],[110,100,20,10], ...]"),
+            FieldSpec("row", FieldType.INT, placeholder="0"),
+            FieldSpec("col", FieldType.INT, placeholder="0"),
+            FieldSpec("row_tolerance", FieldType.INT, optional=True, default=10),
+        ),
+        description="Resolve a table cell centre by row/col from cell boxes.",
+    ))
     specs.append(CommandSpec(
         "AC_list_plugins", "Tools", "List Plugin Commands",
         fields=(FieldSpec("group", FieldType.STRING, optional=True,

je_auto_control/utils/executor/action_executor.py

@@ -3175,6 +3175,17 @@ def _type_unicode(text: str, modifier: str = "ctrl") -> Dict[str, Any]:
     return type_unicode(text, modifier=modifier)
 
 
+def _grid_cell(boxes: Any, row: Any, col: Any,
+               row_tolerance: Any = 10) -> Dict[str, Any]:
+    """Adapter: address a grid cell by (row, col) from a JSON list of boxes."""
+    import json
+    from je_auto_control.utils.grid_locator import locate_cell
+    if isinstance(boxes, str):
+        boxes = json.loads(boxes)
+    return locate_cell(list(boxes), int(row), int(col),
+                       row_tolerance=int(row_tolerance))
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -4891,6 +4902,7 @@ def __init__(self):
             "AC_move_mouse_relative": _move_mouse_relative,
             "AC_type_unicode": _type_unicode,
             "AC_with_modifiers": _with_modifiers,
+            "AC_grid_cell": _grid_cell,
             "AC_detect_drift": _detect_drift,
             "AC_categorical_drift": _categorical_drift,
             "AC_diff_rows": _diff_rows,