Skip to content

Commit 71b751e

Browse files
JE-ChenJE-Chen
committed
Merge dev into main
2 parents 6961cd9 + d13b115 commit 71b751e

16 files changed

Lines changed: 122 additions & 179 deletions

File tree

‎README.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -488,20 +488,24 @@ je_auto_control.execute_action([
488488

489489
| Category | Commands |
490490
|---|---|
491-
| Mouse | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
492-
| Keyboard | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press` |
491+
| Mouse | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_get_mouse_table`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
492+
| Keyboard | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press`, `AC_get_keyboard_keys_table` |
493493
| Image | `AC_locate_all_image`, `AC_locate_image_center`, `AC_locate_and_click` |
494494
| Screen | `AC_screen_size`, `AC_screenshot` |
495495
| Accessibility | `AC_a11y_list`, `AC_a11y_find`, `AC_a11y_click` |
496496
| VLM (AI Locator) | `AC_vlm_locate`, `AC_vlm_click` |
497497
| OCR | `AC_locate_text`, `AC_click_text`, `AC_wait_text` |
498498
| Clipboard | `AC_clipboard_get`, `AC_clipboard_set` |
499-
| Record | `AC_record`, `AC_stop_record` |
499+
| Window | `AC_list_windows`, `AC_focus_window`, `AC_wait_window`, `AC_close_window` |
500+
| Flow control | `AC_loop`, `AC_break`, `AC_continue`, `AC_if_image_found`, `AC_if_pixel`, `AC_while_image`, `AC_wait_image`, `AC_wait_pixel`, `AC_sleep`, `AC_retry` |
501+
| Record | `AC_record`, `AC_stop_record`, `AC_set_record_enable` |
500502
| Report | `AC_generate_html`, `AC_generate_json`, `AC_generate_xml`, `AC_generate_html_report`, `AC_generate_json_report`, `AC_generate_xml_report` |
503+
| Run history | `AC_history_list`, `AC_history_clear` |
501504
| Project | `AC_create_project` |
502505
| Shell | `AC_shell_command` |
503506
| Process | `AC_execute_process` |
504-
| Executor | `AC_execute_action`, `AC_execute_files` |
507+
| Executor | `AC_execute_action`, `AC_execute_files`, `AC_add_package_to_executor`, `AC_add_package_to_callback_executor` |
508+
| MCP server | `AC_start_mcp_server`, `AC_start_mcp_http_server` |
505509

506510
### MCP Server (Use AutoControl from Claude)
507511

‎README/README_zh-CN.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -482,20 +482,24 @@ je_auto_control.execute_action([
482482

483483
| 类别 | 命令 |
484484
|---|---|
485-
| 鼠标 | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
486-
| 键盘 | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press` |
485+
| 鼠标 | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_get_mouse_table`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
486+
| 键盘 | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press`, `AC_get_keyboard_keys_table` |
487487
| 图像 | `AC_locate_all_image`, `AC_locate_image_center`, `AC_locate_and_click` |
488488
| 屏幕 | `AC_screen_size`, `AC_screenshot` |
489489
| Accessibility | `AC_a11y_list`, `AC_a11y_find`, `AC_a11y_click` |
490490
| VLM（AI 定位） | `AC_vlm_locate`, `AC_vlm_click` |
491491
| OCR | `AC_locate_text`, `AC_click_text`, `AC_wait_text` |
492492
| 剪贴板 | `AC_clipboard_get`, `AC_clipboard_set` |
493-
| 录制 | `AC_record`, `AC_stop_record` |
493+
| 窗口 | `AC_list_windows`, `AC_focus_window`, `AC_wait_window`, `AC_close_window` |
494+
| 流程控制 | `AC_loop`, `AC_break`, `AC_continue`, `AC_if_image_found`, `AC_if_pixel`, `AC_while_image`, `AC_wait_image`, `AC_wait_pixel`, `AC_sleep`, `AC_retry` |
495+
| 录制 | `AC_record`, `AC_stop_record`, `AC_set_record_enable` |
494496
| 报告 | `AC_generate_html`, `AC_generate_json`, `AC_generate_xml`, `AC_generate_html_report`, `AC_generate_json_report`, `AC_generate_xml_report` |
497+
| 执行记录 | `AC_history_list`, `AC_history_clear` |
495498
| 项目 | `AC_create_project` |
496499
| Shell | `AC_shell_command` |
497500
| 进程 | `AC_execute_process` |
498-
| 执行器 | `AC_execute_action`, `AC_execute_files` |
501+
| 执行器 | `AC_execute_action`, `AC_execute_files`, `AC_add_package_to_executor`, `AC_add_package_to_callback_executor` |
502+
| MCP 服务器 | `AC_start_mcp_server`, `AC_start_mcp_http_server` |
499503

500504
### MCP 服务器（让 Claude 使用 AutoControl）
501505

‎README/README_zh-TW.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -482,20 +482,24 @@ je_auto_control.execute_action([
482482

483483
| 類別 | 命令 |
484484
|---|---|
485-
| 滑鼠 | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
486-
| 鍵盤 | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press` |
485+
| 滑鼠 | `AC_click_mouse`, `AC_set_mouse_position`, `AC_get_mouse_position`, `AC_get_mouse_table`, `AC_press_mouse`, `AC_release_mouse`, `AC_mouse_scroll`, `AC_mouse_left`, `AC_mouse_right`, `AC_mouse_middle` |
486+
| 鍵盤 | `AC_type_keyboard`, `AC_press_keyboard_key`, `AC_release_keyboard_key`, `AC_write`, `AC_hotkey`, `AC_check_key_is_press`, `AC_get_keyboard_keys_table` |
487487
| 圖像 | `AC_locate_all_image`, `AC_locate_image_center`, `AC_locate_and_click` |
488488
| 螢幕 | `AC_screen_size`, `AC_screenshot` |
489489
| Accessibility | `AC_a11y_list`, `AC_a11y_find`, `AC_a11y_click` |
490490
| VLM（AI 定位） | `AC_vlm_locate`, `AC_vlm_click` |
491491
| OCR | `AC_locate_text`, `AC_click_text`, `AC_wait_text` |
492492
| 剪貼簿 | `AC_clipboard_get`, `AC_clipboard_set` |
493-
| 錄製 | `AC_record`, `AC_stop_record` |
493+
| 視窗 | `AC_list_windows`, `AC_focus_window`, `AC_wait_window`, `AC_close_window` |
494+
| 流程控制 | `AC_loop`, `AC_break`, `AC_continue`, `AC_if_image_found`, `AC_if_pixel`, `AC_while_image`, `AC_wait_image`, `AC_wait_pixel`, `AC_sleep`, `AC_retry` |
495+
| 錄製 | `AC_record`, `AC_stop_record`, `AC_set_record_enable` |
494496
| 報告 | `AC_generate_html`, `AC_generate_json`, `AC_generate_xml`, `AC_generate_html_report`, `AC_generate_json_report`, `AC_generate_xml_report` |
497+
| 執行紀錄 | `AC_history_list`, `AC_history_clear` |
495498
| 專案 | `AC_create_project` |
496499
| Shell | `AC_shell_command` |
497500
| 程序 | `AC_execute_process` |
498-
| 執行器 | `AC_execute_action`, `AC_execute_files` |
501+
| 執行器 | `AC_execute_action`, `AC_execute_files`, `AC_add_package_to_executor`, `AC_add_package_to_callback_executor` |
502+
| MCP 伺服器 | `AC_start_mcp_server`, `AC_start_mcp_http_server` |
499503

500504
### MCP 伺服器（讓 Claude 使用 AutoControl）
501505

‎je_auto_control/utils/callback/callback_function_executor.py‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
from typing import Callable, Any
1+
from typing import Any, Callable
22

33
# utils cv2_utils
44
from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot
@@ -7,6 +7,7 @@
77
from je_auto_control.utils.logging.logging_instance import autocontrol_logger
88
# executor
99
from je_auto_control.utils.executor.action_executor import execute_action, execute_files
10+
from je_auto_control.utils.executor.mouse_aliases import MOUSE_BUTTON_COMMANDS
1011
# file process
1112
from je_auto_control.utils.file_process.get_dir_file_list import get_dir_files_as_list
1213
# html report
@@ -22,7 +23,7 @@
2223
# project
2324
from je_auto_control.utils.project.create_project_structure import create_project_dir
2425
# shell
25-
from je_auto_control.utils.shell_process.shell_exec import ShellManager
26+
from je_auto_control.utils.shell_process.shell_exec import default_shell_manager
2627
# socket server
2728
from je_auto_control.utils.socket_server.auto_control_socket_server import start_autocontrol_socket_server
2829
# process
@@ -61,9 +62,7 @@ def __init__(self):
6162
# 事件字典，對應字串名稱到實際函式
6263
self.event_dict: dict = {
6364
# mouse 滑鼠相關
64-
"AC_mouse_left": click_mouse,
65-
"AC_mouse_right": click_mouse,
66-
"AC_mouse_middle": click_mouse,
65+
**MOUSE_BUTTON_COMMANDS,
6766
"AC_click_mouse": click_mouse,
6867
"AC_get_mouse_table": get_mouse_table,
6968
"AC_get_mouse_position": get_mouse_position,
@@ -129,7 +128,7 @@ def __init__(self):
129128
"AC_add_package_to_callback_executor": package_manager.add_package_to_callback_executor,
130129

131130
# shell command
132-
"AC_shell_command": ShellManager().exec_shell,
131+
"AC_shell_command": default_shell_manager.exec_shell,
133132

134133
# process
135134
"AC_execute_process": start_exe,

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
from je_auto_control.utils.executor.flow_control import (
2323
BLOCK_COMMANDS, LoopBreak, LoopContinue,
2424
)
25+
from je_auto_control.utils.executor.mouse_aliases import MOUSE_BUTTON_COMMANDS
2526
from je_auto_control.utils.ocr.ocr_engine import (
2627
click_text as ocr_click_text,
2728
locate_text_center as ocr_locate_text_center,
@@ -38,7 +39,7 @@
3839
from je_auto_control.utils.mcp_server.server import start_mcp_stdio_server
3940
from je_auto_control.utils.package_manager.package_manager_class import package_manager
4041
from je_auto_control.utils.project.create_project_structure import create_project_dir
41-
from je_auto_control.utils.shell_process.shell_exec import ShellManager
42+
from je_auto_control.utils.shell_process.shell_exec import default_shell_manager
4243
from je_auto_control.utils.start_exe.start_another_process import start_exe
4344
from je_auto_control.utils.test_record.record_test_class import record_action_to_list, test_record_instance
4445
from je_auto_control.wrapper.auto_control_image import locate_all_image, locate_and_click, locate_image_center
@@ -124,9 +125,7 @@ def __init__(self):
124125
# 事件字典，對應字串名稱到函式
125126
self.event_dict: dict = {
126127
# Mouse 滑鼠相關
127-
"AC_mouse_left": click_mouse,
128-
"AC_mouse_right": click_mouse,
129-
"AC_mouse_middle": click_mouse,
128+
**MOUSE_BUTTON_COMMANDS,
130129
"AC_click_mouse": click_mouse,
131130
"AC_get_mouse_table": get_mouse_table,
132131
"AC_get_mouse_position": get_mouse_position,
@@ -178,7 +177,7 @@ def __init__(self):
178177
"AC_create_project": create_project_dir,
179178

180179
# Shell
181-
"AC_shell_command": ShellManager().exec_shell,
180+
"AC_shell_command": default_shell_manager.exec_shell,
182181

183182
# Process
184183
"AC_execute_process": start_exe,

‎je_auto_control/utils/executor/mouse_aliases.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
"""Single-button mouse-click aliases shared by the action and callback executors.
2+
3+
Each ``AC_mouse_<button>`` command name needs a callable whose signature is
4+
``(x=None, y=None)`` so the button itself is implied by the command name and
5+
not threaded through ``mouse_keycode``. Generating the three callables from
6+
one factory keeps the registration sites identical across executors.
7+
"""
8+
from typing import Callable, Optional, Tuple
9+
10+
from je_auto_control.wrapper.auto_control_mouse import click_mouse
11+
12+
ClickFn = Callable[[Optional[int], Optional[int]], Tuple[int, int, int]]
13+
14+
15+
def _make_button_click(button: str) -> ClickFn:
16+
def click(x: Optional[int] = None,
17+
y: Optional[int] = None) -> Tuple[int, int, int]:
18+
return click_mouse(button, x, y)
19+
20+
click.__name__ = f"click_{button}"
21+
click.__qualname__ = click.__name__
22+
click.__doc__ = f"Click the {button.replace('_', ' ')} mouse button at (x, y)."
23+
return click
24+
25+
26+
click_mouse_left: ClickFn = _make_button_click("mouse_left")
27+
click_mouse_right: ClickFn = _make_button_click("mouse_right")
28+
click_mouse_middle: ClickFn = _make_button_click("mouse_middle")
29+
30+
MOUSE_BUTTON_COMMANDS: dict = {
31+
"AC_mouse_left": click_mouse_left,
32+
"AC_mouse_right": click_mouse_right,
33+
"AC_mouse_middle": click_mouse_middle,
34+
}
35+
36+
__all__ = [
37+
"click_mouse_left", "click_mouse_right", "click_mouse_middle",
38+
"MOUSE_BUTTON_COMMANDS",
39+
]

‎je_auto_control/wrapper/auto_control_image.py‎

Lines changed: 1 addition & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
1-
from typing import List, Tuple, Optional, Union
1+
from typing import List, Tuple, Union
22

33
from je_auto_control.utils.cv2_utils import template_detection
4-
from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot
54
from je_auto_control.utils.exception.exception_tags import cant_find_image_error_message
65
from je_auto_control.utils.exception.exceptions import ImageNotFoundException
76
from je_auto_control.utils.logging.logging_instance import autocontrol_logger
@@ -81,23 +80,4 @@ def locate_and_click(image, mouse_keycode: Union[int, str],
8180
except (OSError, RuntimeError, AttributeError, TypeError, ValueError, ImageNotFoundException) as error:
8281
record_action_to_list("locate_and_click", {"image": image}, repr(error))
8382
autocontrol_logger.error(f"locate_and_click failed: {repr(error)}")
84-
raise
85-
86-
87-
def screenshot(file_path: Optional[str] = None, region: Optional[List[int]] = None):
88-
"""
89-
擷取螢幕畫面
90-
Take a screenshot
91-
92-
:param file_path: 儲存路徑 (None = 不儲存)
93-
:param region: 擷取區域 [x1, y1, x2, y2]
94-
:return: PIL Image
95-
"""
96-
autocontrol_logger.info(f"screenshot, file={file_path}, region={region}")
97-
try:
98-
record_action_to_list("screenshot", {"file_path": file_path, "region": region})
99-
return pil_screenshot(file_path, region)
100-
except (OSError, RuntimeError, AttributeError, TypeError, ValueError, ImageNotFoundException) as error:
101-
record_action_to_list("screenshot", {"file_path": file_path, "region": region}, repr(error))
102-
autocontrol_logger.error(f"screenshot failed: {repr(error)}")
10383
raise

‎je_auto_control/wrapper/auto_control_record.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ def stop_record() -> list:
4040
if action[0] == "AC_type_keyboard":
4141
new_list.append([action[0], {"keycode": action[1]}])
4242
else:
43-
new_list.append([action[0], {"mouse_keycode": action[0], "x": action[1], "y": action[2]}])
43+
new_list.append([action[0], {"x": action[1], "y": action[2]}])
4444
record_action_to_list("stop_record", None)
4545
return new_list
4646
except (OSError, RuntimeError, AttributeError, TypeError, ValueError, AutoControlException, AutoControlJsonActionException) as error:

‎test/unit_test/argparse/test1.json‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@
88
[
99
"AC_mouse_left",
1010
{
11-
"mouse_keycode": "mouse_left",
1211
"x": 500,
1312
"y": 500
1413
}

‎test/unit_test/argparse/test2.json‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@
88
[
99
"AC_mouse_left",
1010
{
11-
"mouse_keycode": "mouse_left",
1211
"x": 500,
1312
"y": 500
1413
}

0 commit comments

Comments
 (0)