Skip to content

Commit 2a5d22c

Browse files
committed
Add clipboard, hotkeys, triggers, cron, REST, CLI, plugins, i18n
Every new feature ships with a headless Python API, an AC_* executor command (where applicable), and a GUI affordance — keeping the facade Qt-free. New modules: clipboard, hotkey daemon, trigger engine, cron scheduling, REST API server, plugin loader, subcommand CLI, script variables, OCR, recording editor, window manager, watcher. GUI: menu bar with File/View/Tools/Language/Help, closable tabs with a View → Tabs restore menu, new Hotkeys/Triggers/Plugins tabs, socket-server tab split into TCP + REST sections, and live language switching across English / Traditional Chinese / Simplified Chinese / Japanese. CLAUDE.md now requires every feature to expose both headless and GUI access paths, verified by an import-time PySide6 leak check. Docs: new /new_features/ page (Eng + Zh), updated CLI page to document the python -m je_auto_control.cli subcommand interface. Tests: 40 new headless unit tests (cron, hotkey parse, plugin loader, trigger engine, REST server, CLI, clipboard round-trip).
1 parent e435a6f commit 2a5d22c

74 files changed

Lines changed: 6374 additions & 273 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,27 @@ python -m pytest test/integrated_test/
7474
python -m build
7575
```
7676

77+
## Feature Delivery Rules
78+
79+
### Every feature must ship both a headless API and a GUI surface
80+
81+
No feature is complete unless it can be driven entirely without the GUI **and** has a corresponding GUI affordance. Concretely:
82+
83+
- **Headless core in `utils/` or `wrapper/`**: all business logic lives in a module with zero `PySide6` imports. Users must be able to `import je_auto_control` and call the feature without ever instantiating a Qt class.
84+
- **Re-export from the package facade**: add the public functions / classes to `je_auto_control/__init__.py` and its `__all__` so `import je_auto_control as ac; ac.<feature>(...)` works out of the box.
85+
- **Executor command coverage**: wire an `AC_*` command into `utils/executor/action_executor.py` so the feature is usable from JSON action files, the socket server, the scheduler, and the visual script builder — all without Python glue.
86+
- **GUI tab or control is a thin wrapper**: the Qt widget must only translate user input into calls on the headless core. It must not contain business logic that would be unreachable headlessly.
87+
- **The top-level package stays Qt-free**: `import je_auto_control` MUST NOT import `PySide6`. The GUI entry point is loaded lazily inside `start_autocontrol_gui()`. Verify with:
88+
89+
```python
90+
import sys, je_auto_control # noqa
91+
assert not any("PySide6" in m for m in sys.modules)
92+
```
93+
94+
- **Tests cover the headless path**: at least one unit test in `test/unit_test/` must exercise the feature through its non-GUI API with no Qt imports.
95+
96+
Features that are inherently interactive (e.g. region picking with the mouse, template cropping) still count as GUI-only — but they must accept programmatic equivalents (e.g. `screenshot(screen_region=[...])` with explicit coordinates) so scripts can replay the same effect headlessly.
97+
7798
## Coding Standards
7899

79100
### Security First

docs/source/Eng/doc/cli/cli_doc.rst

Lines changed: 59 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,47 +2,90 @@
22
Command-Line Interface
33
=======================
44

5-
AutoControl can be used directly from the command line to execute automation scripts.
5+
Two CLI entry points are provided:
66

7-
Execute a Single Action File
8-
=============================
7+
- ``python -m je_auto_control`` — legacy flag-style runner for one-off
8+
execute / create-project operations. Also launches the GUI when called
9+
with no arguments.
10+
- ``python -m je_auto_control.cli`` — subcommand-based runner for running
11+
scripts, listing scheduler jobs, and starting the socket / REST servers.
12+
13+
Subcommand CLI (``python -m je_auto_control.cli``)
14+
==================================================
15+
16+
Run a script
17+
------------
918

1019
.. code-block:: bash
1120
12-
python -m je_auto_control --execute_file "path/to/actions.json"
21+
python -m je_auto_control.cli run script.json
22+
python -m je_auto_control.cli run script.json --var count=10 --var name=alice
23+
python -m je_auto_control.cli run script.json --dry-run
24+
25+
``--var name=value`` is parsed as JSON when the value parses, otherwise
26+
it is treated as a plain string. ``--dry-run`` records every action
27+
through the executor without invoking any side effects.
28+
29+
List scheduler jobs
30+
-------------------
31+
32+
.. code-block:: bash
33+
34+
python -m je_auto_control.cli list-jobs
35+
36+
Start the TCP socket server
37+
---------------------------
38+
39+
.. code-block:: bash
40+
41+
python -m je_auto_control.cli start-server --host 127.0.0.1 --port 9938
1342
14-
# Short form
43+
Start the REST API server
44+
-------------------------
45+
46+
.. code-block:: bash
47+
48+
python -m je_auto_control.cli start-rest --host 127.0.0.1 --port 9939
49+
50+
Endpoints: ``GET /health``, ``GET /jobs``, ``POST /execute`` with
51+
``{"actions": [...]}``.
52+
53+
Legacy flag-style CLI (``python -m je_auto_control``)
54+
=====================================================
55+
56+
Execute a single action file
57+
----------------------------
58+
59+
.. code-block:: bash
60+
61+
python -m je_auto_control --execute_file "path/to/actions.json"
1562
python -m je_auto_control -e "path/to/actions.json"
1663
17-
Execute All Files in a Directory
18-
================================
64+
Execute all files in a directory
65+
--------------------------------
1966

2067
.. code-block:: bash
2168
2269
python -m je_auto_control --execute_dir "path/to/action_files/"
23-
24-
# Short form
2570
python -m je_auto_control -d "path/to/action_files/"
2671
27-
Execute a JSON String Directly
28-
==============================
72+
Execute a JSON string directly
73+
------------------------------
2974

3075
.. code-block:: bash
3176
3277
python -m je_auto_control --execute_str '[["AC_screenshot", {"file_path": "test.png"}]]'
3378
34-
Create a Project Template
35-
=========================
79+
Create a project template
80+
-------------------------
3681

3782
.. code-block:: bash
3883
3984
python -m je_auto_control --create_project "path/to/my_project"
40-
41-
# Short form
4285
python -m je_auto_control -c "path/to/my_project"
4386
4487
Launch the GUI
45-
==============
88+
--------------
4689

4790
.. code-block:: bash
4891
Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
=====================
2+
New Features (2026-04)
3+
=====================
4+
5+
This page documents the April 2026 additions to AutoControl. Every new
6+
feature ships with a headless Python API **and** a GUI affordance, and is
7+
wired into the executor so it works from JSON scripts, the socket server,
8+
the REST API, and the CLI without any Python glue.
9+
10+
.. contents::
11+
:local:
12+
:depth: 2
13+
14+
15+
Clipboard
16+
=========
17+
18+
Headless::
19+
20+
import je_auto_control as ac
21+
ac.set_clipboard("hello")
22+
text = ac.get_clipboard()
23+
24+
Action-JSON commands::
25+
26+
[["AC_clipboard_set", {"text": "hello"}]]
27+
[["AC_clipboard_get", {}]]
28+
29+
Backends: Windows (Win32 via ``ctypes``), macOS (``pbcopy``/``pbpaste``),
30+
Linux (``xclip`` or ``xsel``). A ``RuntimeError`` is raised if no backend
31+
is available.
32+
33+
34+
Dry-run / step-debug executor
35+
=============================
36+
37+
Run an action list through the executor without invoking any side effects
38+
— useful for validating JSON scripts::
39+
40+
from je_auto_control.utils.executor.action_executor import executor
41+
record = executor.execute_action(actions, dry_run=True)
42+
43+
``step_callback`` lets you observe each action before it runs::
44+
45+
executor.execute_action(actions, step_callback=lambda a: print(a))
46+
47+
From the CLI::
48+
49+
python -m je_auto_control.cli run script.json --dry-run
50+
51+
52+
Global hotkey daemon (Windows)
53+
==============================
54+
55+
Bind OS-level hotkeys to action-JSON scripts::
56+
57+
from je_auto_control import default_hotkey_daemon
58+
default_hotkey_daemon.bind("ctrl+alt+1", "scripts/greet.json")
59+
default_hotkey_daemon.start()
60+
61+
Supported modifiers: ``ctrl``, ``alt``, ``shift``, ``win`` / ``super`` /
62+
``meta``. Keys: letters, digits, ``f1`` … ``f12``, arrows, ``space``,
63+
``enter``, ``tab``, ``escape``, ``home``, ``end``, ``insert``, ``delete``,
64+
``pageup``, ``pagedown``.
65+
66+
macOS and Linux currently raise ``NotImplementedError`` on
67+
``start()`` — the Strategy-pattern interface is in place so backends can
68+
be added later.
69+
70+
GUI: **Hotkeys** tab (bind/unbind, start/stop daemon, live fired count).
71+
72+
73+
Event triggers
74+
==============
75+
76+
Poll-based triggers fire an action script when a screen/state change is
77+
detected::
78+
79+
from je_auto_control import default_trigger_engine, ImageAppearsTrigger
80+
default_trigger_engine.add(ImageAppearsTrigger(
81+
trigger_id="", script_path="scripts/click_ok.json",
82+
image_path="templates/ok_button.png", threshold=0.85,
83+
repeat=True,
84+
))
85+
default_trigger_engine.start()
86+
87+
Available trigger types:
88+
89+
- ``ImageAppearsTrigger`` — template match on the current screen
90+
- ``WindowAppearsTrigger`` — title substring match
91+
- ``PixelColorTrigger`` — pixel color within tolerance
92+
- ``FilePathTrigger`` — mtime change on a path
93+
94+
GUI: **Triggers** tab (add/remove/start/stop, live fired count).
95+
96+
97+
Cron scheduling
98+
===============
99+
100+
Five-field cron (``minute hour day-of-month month day-of-week``) with
101+
``*``, comma-lists, ``*/step``, and ``start-stop`` ranges::
102+
103+
from je_auto_control import default_scheduler
104+
job = default_scheduler.add_cron_job(
105+
script_path="scripts/daily.json",
106+
cron_expression="0 9 * * 1-5", # 09:00 on weekdays
107+
)
108+
default_scheduler.start()
109+
110+
Interval and cron jobs coexist in the same scheduler; ``job.is_cron``
111+
tells them apart. GUI: **Scheduler** tab has cron/interval radio.
112+
113+
114+
Plugin loader
115+
=============
116+
117+
A plugin file is any ``.py`` defining top-level callables whose names
118+
start with ``AC_``. Each one becomes a new executor command::
119+
120+
# my_plugins/greeting.py
121+
def AC_greet(args=None):
122+
return f"hello, {args['name']}"
123+
124+
::
125+
126+
from je_auto_control import (
127+
load_plugin_directory, register_plugin_commands,
128+
)
129+
commands = load_plugin_directory("my_plugins/")
130+
register_plugin_commands(commands)
131+
132+
# Now usable from JSON:
133+
# [["AC_greet", {"name": "world"}]]
134+
135+
GUI: **Plugins** tab (browse directory, one-click register).
136+
137+
.. warning::
138+
Plugin files execute arbitrary Python. Only load from directories
139+
under your own control.
140+
141+
142+
REST API server
143+
===============
144+
145+
A stdlib-only HTTP server that exposes the executor and scheduler::
146+
147+
from je_auto_control import start_rest_api_server
148+
server = start_rest_api_server(host="127.0.0.1", port=9939)
149+
150+
Endpoints:
151+
152+
- ``GET /health`` — liveness probe
153+
- ``GET /jobs`` — scheduler job list
154+
- ``POST /execute`` with body ``{"actions": [...]}`` — run actions
155+
156+
GUI: **Socket Server** tab now has a separate REST section with its own
157+
host/port and a ``0.0.0.0`` opt-in.
158+
159+
.. note::
160+
Defaults to ``127.0.0.1`` per CLAUDE.md policy. Bind to ``0.0.0.0``
161+
only when you have authenticated the network boundary.
162+
163+
164+
CLI runner
165+
==========
166+
167+
A thin subcommand-based CLI over the headless APIs::
168+
169+
python -m je_auto_control.cli run script.json
170+
python -m je_auto_control.cli run script.json --var name=alice --dry-run
171+
python -m je_auto_control.cli list-jobs
172+
python -m je_auto_control.cli start-server --port 9938
173+
python -m je_auto_control.cli start-rest --port 9939
174+
175+
``--var name=value`` is parsed as JSON when possible (so ``count=10``
176+
becomes an int), otherwise treated as a string.
177+
178+
179+
Multi-language GUI (i18n)
180+
=========================
181+
182+
Live language switching via the **Language** menu. Built-in packs:
183+
184+
- English
185+
- Traditional Chinese (繁體中文)
186+
- Simplified Chinese (简体中文)
187+
- Japanese (日本語)
188+
189+
Register additional languages at runtime::
190+
191+
from je_auto_control.gui.language_wrapper.multi_language_wrapper import (
192+
language_wrapper,
193+
)
194+
language_wrapper.register_language("French", {"menu_file": "Fichier", ...})
195+
196+
Missing keys fall through to the English default, so a feature ships
197+
with usable labels even before its translations land.
198+
199+
200+
Closable tabs + menu bar
201+
========================
202+
203+
The main window is now a ``QMainWindow`` with:
204+
205+
- **File** → Open Script, Exit
206+
- **View → Tabs** → checkable entries for every tab (restore closed tabs)
207+
- **Tools** → Start hotkey daemon / scheduler / trigger engine
208+
- **Language** → select a registered language pack
209+
- **Help** → About
210+
211+
Close any tab with its ✕ button; re-open it via *View → Tabs*.

docs/source/Eng/eng_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,4 @@ Comprehensive guides for all AutoControl features.
2222
doc/critical_exit/critical_exit_doc
2323
doc/cli/cli_doc
2424
doc/create_project/create_project_doc
25+
doc/new_features/new_features_doc

0 commit comments

Comments
 (0)