Skip to content

Commit 8d7296a

Browse files
JE-ChenJE-Chen
committed
Hardcode AC_mouse_* aliases and sync executor command surface
The AC_mouse_left/right/middle aliases all delegated to click_mouse and required mouse_keycode anyway, so the alias name was misleading. Worse, stop_record() recorded them as {"mouse_keycode": "AC_mouse_left", ...}, which is not a valid keycode in the mouse table — recorded JSON could not be replayed. Bind each alias to its hardcoded button and drop the keycode from the recorder output so record/replay round-trips cleanly. Other cleanups: - Use default_shell_manager singleton in both action and callback executors instead of constructing a fresh ShellManager per import. - Remove the duplicate screenshot() in auto_control_image.py — it was never imported and shadowed the canonical one in auto_control_screen. - Add the ~25 commands that were registered but missing from the README command tables (window, flow control, history, MCP, etc.).
1 parent 6c7d79a commit 8d7296a

14 files changed

Lines changed: 107 additions & 81 deletions

File tree

‎README.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -488,20 +488,24 @@ je_auto_control.execute_action([
488488

489489
| Category | Commands |
490490
|---|---|
491-
| Mouse | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
492-
| Keyboard | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press` |
491+
| Mouse | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_get_mouse_table`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
492+
| Keyboard | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press`, `AC_get_keyboard_keys_table` |
493493
| Image | `AC_locate_all_image`, `AC_locate_image_center`, `AC_locate_and_click` |
494494
| Screen | `AC_screen_size`, `AC_screenshot` |
495495
| Accessibility | `AC_a11y_list`, `AC_a11y_find`, `AC_a11y_click` |
496496
| VLM (AI Locator) | `AC_vlm_locate`, `AC_vlm_click` |
497497
| OCR | `AC_locate_text`, `AC_click_text`, `AC_wait_text` |
498498
| Clipboard | `AC_clipboard_get`, `AC_clipboard_set` |
499-
| Record | `AC_record`, `AC_stop_record` |
499+
| Window | `AC_list_windows`, `AC_focus_window`, `AC_wait_window`, `AC_close_window` |
500+
| Flow control | `AC_loop`, `AC_break`, `AC_continue`, `AC_if_image_found`, `AC_if_pixel`, `AC_while_image`, `AC_wait_image`, `AC_wait_pixel`, `AC_sleep`, `AC_retry` |
501+
| Record | `AC_record`, `AC_stop_record`, `AC_set_record_enable` |
500502
| Report | `AC_generate_html`, `AC_generate_json`, `AC_generate_xml`, `AC_generate_html_report`, `AC_generate_json_report`, `AC_generate_xml_report` |
503+
| Run history | `AC_history_list`, `AC_history_clear` |
501504
| Project | `AC_create_project` |
502505
| Shell | `AC_shell_command` |
503506
| Process | `AC_execute_process` |
504-
| Executor | `AC_execute_action`, `AC_execute_files` |
507+
| Executor | `AC_execute_action`, `AC_execute_files`, `AC_add_package_to_executor`, `AC_add_package_to_callback_executor` |
508+
| MCP server | `AC_start_mcp_server`, `AC_start_mcp_http_server` |
505509

506510
### MCP Server (Use AutoControl from Claude)
507511

‎README/README_zh-CN.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -482,20 +482,24 @@ je_auto_control.execute_action([
482482

483483
| 类别 | 命令 |
484484
|---|---|
485-
| 鼠标 | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
486-
| 键盘 | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press` |
485+
| 鼠标 | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_get_mouse_table`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
486+
| 键盘 | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press`, `AC_get_keyboard_keys_table` |
487487
| 图像 | `AC_locate_all_image`, `AC_locate_image_center`, `AC_locate_and_click` |
488488
| 屏幕 | `AC_screen_size`, `AC_screenshot` |
489489
| Accessibility | `AC_a11y_list`, `AC_a11y_find`, `AC_a11y_click` |
490490
| VLM（AI 定位） | `AC_vlm_locate`, `AC_vlm_click` |
491491
| OCR | `AC_locate_text`, `AC_click_text`, `AC_wait_text` |
492492
| 剪贴板 | `AC_clipboard_get`, `AC_clipboard_set` |
493-
| 录制 | `AC_record`, `AC_stop_record` |
493+
| 窗口 | `AC_list_windows`, `AC_focus_window`, `AC_wait_window`, `AC_close_window` |
494+
| 流程控制 | `AC_loop`, `AC_break`, `AC_continue`, `AC_if_image_found`, `AC_if_pixel`, `AC_while_image`, `AC_wait_image`, `AC_wait_pixel`, `AC_sleep`, `AC_retry` |
495+
| 录制 | `AC_record`, `AC_stop_record`, `AC_set_record_enable` |
494496
| 报告 | `AC_generate_html`, `AC_generate_json`, `AC_generate_xml`, `AC_generate_html_report`, `AC_generate_json_report`, `AC_generate_xml_report` |
497+
| 执行记录 | `AC_history_list`, `AC_history_clear` |
495498
| 项目 | `AC_create_project` |
496499
| Shell | `AC_shell_command` |
497500
| 进程 | `AC_execute_process` |
498-
| 执行器 | `AC_execute_action`, `AC_execute_files` |
501+
| 执行器 | `AC_execute_action`, `AC_execute_files`, `AC_add_package_to_executor`, `AC_add_package_to_callback_executor` |
502+
| MCP 服务器 | `AC_start_mcp_server`, `AC_start_mcp_http_server` |
499503

500504
### MCP 服务器（让 Claude 使用 AutoControl）
501505

‎README/README_zh-TW.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -482,20 +482,24 @@ je_auto_control.execute_action([
482482

483483
| 類別 | 命令 |
484484
|---|---|
485-
| 滑鼠 | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
486-
| 鍵盤 | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press` |
485+
| 滑鼠 | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_get_mouse_table`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
486+
| 鍵盤 | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press`, `AC_get_keyboard_keys_table` |
487487
| 圖像 | `AC_locate_all_image`, `AC_locate_image_center`, `AC_locate_and_click` |
488488
| 螢幕 | `AC_screen_size`, `AC_screenshot` |
489489
| Accessibility | `AC_a11y_list`, `AC_a11y_find`, `AC_a11y_click` |
490490
| VLM（AI 定位） | `AC_vlm_locate`, `AC_vlm_click` |
491491
| OCR | `AC_locate_text`, `AC_click_text`, `AC_wait_text` |
492492
| 剪貼簿 | `AC_clipboard_get`, `AC_clipboard_set` |
493-
| 錄製 | `AC_record`, `AC_stop_record` |
493+
| 視窗 | `AC_list_windows`, `AC_focus_window`, `AC_wait_window`, `AC_close_window` |
494+
| 流程控制 | `AC_loop`, `AC_break`, `AC_continue`, `AC_if_image_found`, `AC_if_pixel`, `AC_while_image`, `AC_wait_image`, `AC_wait_pixel`, `AC_sleep`, `AC_retry` |
495+
| 錄製 | `AC_record`, `AC_stop_record`, `AC_set_record_enable` |
494496
| 報告 | `AC_generate_html`, `AC_generate_json`, `AC_generate_xml`, `AC_generate_html_report`, `AC_generate_json_report`, `AC_generate_xml_report` |
497+
| 執行紀錄 | `AC_history_list`, `AC_history_clear` |
495498
| 專案 | `AC_create_project` |
496499
| Shell | `AC_shell_command` |
497500
| 程序 | `AC_execute_process` |
498-
| 執行器 | `AC_execute_action`, `AC_execute_files` |
501+
| 執行器 | `AC_execute_action`, `AC_execute_files`, `AC_add_package_to_executor`, `AC_add_package_to_callback_executor` |
502+
| MCP 伺服器 | `AC_start_mcp_server`, `AC_start_mcp_http_server` |
499503

500504
### MCP 伺服器（讓 Claude 使用 AutoControl）
501505

‎je_auto_control/utils/callback/callback_function_executor.py‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
from typing import Callable, Any
1+
from typing import Any, Callable, Optional, Tuple
22

33
# utils cv2_utils
44
from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot
@@ -22,7 +22,7 @@
2222
# project
2323
from je_auto_control.utils.project.create_project_structure import create_project_dir
2424
# shell
25-
from je_auto_control.utils.shell_process.shell_exec import ShellManager
25+
from je_auto_control.utils.shell_process.shell_exec import default_shell_manager
2626
# socket server
2727
from je_auto_control.utils.socket_server.auto_control_socket_server import start_autocontrol_socket_server
2828
# process
@@ -48,6 +48,24 @@
4848
from je_auto_control.wrapper.auto_control_screen import screen_size, screenshot
4949

5050

51+
def _click_mouse_left(x: Optional[int] = None,
52+
y: Optional[int] = None) -> Tuple[int, int, int]:
53+
"""Callback adapter: click left mouse button (button is hardcoded)."""
54+
return click_mouse("mouse_left", x, y)
55+
56+
57+
def _click_mouse_right(x: Optional[int] = None,
58+
y: Optional[int] = None) -> Tuple[int, int, int]:
59+
"""Callback adapter: click right mouse button (button is hardcoded)."""
60+
return click_mouse("mouse_right", x, y)
61+
62+
63+
def _click_mouse_middle(x: Optional[int] = None,
64+
y: Optional[int] = None) -> Tuple[int, int, int]:
65+
"""Callback adapter: click middle mouse button (button is hardcoded)."""
66+
return click_mouse("mouse_middle", x, y)
67+
68+
5169
class CallbackFunctionExecutor:
5270
"""
5371
CallbackFunctionExecutor
@@ -61,9 +79,9 @@ def __init__(self):
6179
# 事件字典，對應字串名稱到實際函式
6280
self.event_dict: dict = {
6381
# mouse 滑鼠相關
64-
"AC_mouse_left": click_mouse,
65-
"AC_mouse_right": click_mouse,
66-
"AC_mouse_middle": click_mouse,
82+
"AC_mouse_left": _click_mouse_left,
83+
"AC_mouse_right": _click_mouse_right,
84+
"AC_mouse_middle": _click_mouse_middle,
6785
"AC_click_mouse": click_mouse,
6886
"AC_get_mouse_table": get_mouse_table,
6987
"AC_get_mouse_position": get_mouse_position,
@@ -129,7 +147,7 @@ def __init__(self):
129147
"AC_add_package_to_callback_executor": package_manager.add_package_to_callback_executor,
130148

131149
# shell command
132-
"AC_shell_command": ShellManager().exec_shell,
150+
"AC_shell_command": default_shell_manager.exec_shell,
133151

134152
# process
135153
"AC_execute_process": start_exe,

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import types
2-
from typing import Any, Callable, Dict, List, Optional, Union
2+
from typing import Any, Callable, Dict, List, Optional, Tuple, Union
33

44
from je_auto_control.utils.exception.exception_tags import (
55
action_is_null_error_message, add_command_exception_error_message,
@@ -38,7 +38,7 @@
3838
from je_auto_control.utils.mcp_server.server import start_mcp_stdio_server
3939
from je_auto_control.utils.package_manager.package_manager_class import package_manager
4040
from je_auto_control.utils.project.create_project_structure import create_project_dir
41-
from je_auto_control.utils.shell_process.shell_exec import ShellManager
41+
from je_auto_control.utils.shell_process.shell_exec import default_shell_manager
4242
from je_auto_control.utils.start_exe.start_another_process import start_exe
4343
from je_auto_control.utils.test_record.record_test_class import record_action_to_list, test_record_instance
4444
from je_auto_control.wrapper.auto_control_image import locate_all_image, locate_and_click, locate_image_center
@@ -91,6 +91,24 @@ def _vlm_locate_as_list(description: str,
9191
return None if coords is None else [coords[0], coords[1]]
9292

9393

94+
def _click_mouse_left(x: Optional[int] = None,
95+
y: Optional[int] = None) -> Tuple[int, int, int]:
96+
"""Executor adapter: click left mouse button (button is hardcoded)."""
97+
return click_mouse("mouse_left", x, y)
98+
99+
100+
def _click_mouse_right(x: Optional[int] = None,
101+
y: Optional[int] = None) -> Tuple[int, int, int]:
102+
"""Executor adapter: click right mouse button (button is hardcoded)."""
103+
return click_mouse("mouse_right", x, y)
104+
105+
106+
def _click_mouse_middle(x: Optional[int] = None,
107+
y: Optional[int] = None) -> Tuple[int, int, int]:
108+
"""Executor adapter: click middle mouse button (button is hardcoded)."""
109+
return click_mouse("mouse_middle", x, y)
110+
111+
94112
def _history_list_as_dicts(limit: int = 100,
95113
source_type: Optional[str] = None) -> List[dict]:
96114
"""Executor adapter: list run history as plain dicts (JSON-friendly)."""
@@ -124,9 +142,9 @@ def __init__(self):
124142
# 事件字典，對應字串名稱到函式
125143
self.event_dict: dict = {
126144
# Mouse 滑鼠相關
127-
"AC_mouse_left": click_mouse,
128-
"AC_mouse_right": click_mouse,
129-
"AC_mouse_middle": click_mouse,
145+
"AC_mouse_left": _click_mouse_left,
146+
"AC_mouse_right": _click_mouse_right,
147+
"AC_mouse_middle": _click_mouse_middle,
130148
"AC_click_mouse": click_mouse,
131149
"AC_get_mouse_table": get_mouse_table,
132150
"AC_get_mouse_position": get_mouse_position,
@@ -178,7 +196,7 @@ def __init__(self):
178196
"AC_create_project": create_project_dir,
179197

180198
# Shell
181-
"AC_shell_command": ShellManager().exec_shell,
199+
"AC_shell_command": default_shell_manager.exec_shell,
182200

183201
# Process
184202
"AC_execute_process": start_exe,

‎je_auto_control/wrapper/auto_control_image.py‎

Lines changed: 1 addition & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
1-
from typing import List, Tuple, Optional, Union
1+
from typing import List, Tuple, Union
22

33
from je_auto_control.utils.cv2_utils import template_detection
4-
from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot
54
from je_auto_control.utils.exception.exception_tags import cant_find_image_error_message
65
from je_auto_control.utils.exception.exceptions import ImageNotFoundException
76
from je_auto_control.utils.logging.logging_instance import autocontrol_logger
@@ -81,23 +80,4 @@ def locate_and_click(image, mouse_keycode: Union[int, str],
8180
except (OSError, RuntimeError, AttributeError, TypeError, ValueError, ImageNotFoundException) as error:
8281
record_action_to_list("locate_and_click", {"image": image}, repr(error))
8382
autocontrol_logger.error(f"locate_and_click failed: {repr(error)}")
84-
raise
85-
86-
87-
def screenshot(file_path: Optional[str] = None, region: Optional[List[int]] = None):
88-
"""
89-
擷取螢幕畫面
90-
Take a screenshot
91-
92-
:param file_path: 儲存路徑 (None = 不儲存)
93-
:param region: 擷取區域 [x1, y1, x2, y2]
94-
:return: PIL Image
95-
"""
96-
autocontrol_logger.info(f"screenshot, file={file_path}, region={region}")
97-
try:
98-
record_action_to_list("screenshot", {"file_path": file_path, "region": region})
99-
return pil_screenshot(file_path, region)
100-
except (OSError, RuntimeError, AttributeError, TypeError, ValueError, ImageNotFoundException) as error:
101-
record_action_to_list("screenshot", {"file_path": file_path, "region": region}, repr(error))
102-
autocontrol_logger.error(f"screenshot failed: {repr(error)}")
10383
raise

‎je_auto_control/wrapper/auto_control_record.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ def stop_record() -> list:
4040
if action[0] == "AC_type_keyboard":
4141
new_list.append([action[0], {"keycode": action[1]}])
4242
else:
43-
new_list.append([action[0], {"mouse_keycode": action[0], "x": action[1], "y": action[2]}])
43+
new_list.append([action[0], {"x": action[1], "y": action[2]}])
4444
record_action_to_list("stop_record", None)
4545
return new_list
4646
except (OSError, RuntimeError, AttributeError, TypeError, ValueError, AutoControlException, AutoControlJsonActionException) as error:

‎test/unit_test/argparse/test1.json‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@
88
[
99
"AC_mouse_left",
1010
{
11-
"mouse_keycode": "mouse_left",
1211
"x": 500,
1312
"y": 500
1413
}

‎test/unit_test/argparse/test2.json‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@
88
[
99
"AC_mouse_left",
1010
{
11-
"mouse_keycode": "mouse_left",
1211
"x": 500,
1312
"y": 500
1413
}

‎test/unit_test/execute_action/execute_action_test.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
if sys.platform in ["win32", "cygwin", "msys"]:
99
test_list = [
1010
["AC_type_keyboard", {"keycode": 65}],
11-
["AC_mouse_left", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],
11+
["AC_mouse_left", {"x": 500, "y": 500}],
1212
["AC_get_mouse_position"],
1313
["AC_press_mouse", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],
1414
["AC_release_mouse", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],
@@ -18,7 +18,7 @@
1818
elif sys.platform in ["linux", "linux2"]:
1919
test_list = [
2020
["AC_type_keyboard", {"keycode": 38}],
21-
["AC_mouse_left", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],
21+
["AC_mouse_left", {"x": 500, "y": 500}],
2222
["AC_get_mouse_position"],
2323
["AC_press_mouse", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],
2424
["AC_release_mouse", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],
@@ -27,7 +27,7 @@
2727
elif sys.platform in ["darwin"]:
2828
test_list = [
2929
["AC_type_keyboard", {"keycode": 0x00}],
30-
["AC_mouse_left", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],
30+
["AC_mouse_left", {"x": 500, "y": 500}],
3131
["AC_get_mouse_position"],
3232
["AC_press_mouse", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],
3333
["AC_release_mouse", {"mouse_keycode": "mouse_left", "x": 500, "y": 500}],

0 commit comments

Comments
 (0)