Merge pull request #184 from Integration-Automation/dev

Profiler, run-history timeline, secrets, webhook + email triggers, embedder fixes

Author: JE-Chen

README.md

@@ -64,6 +64,7 @@
 - **LLM Action Planner** — translate a plain-language description into a validated `AC_*` action list using Claude
 - **Runtime Variables & Control Flow** — `${var}` substitution at execution time, plus `AC_set_var` / `AC_inc_var` / `AC_if_var` / `AC_for_each` / `AC_loop` / `AC_retry` for data-driven scripts
 - **Remote Desktop** — stream this machine's screen and accept remote input over a token-authenticated TCP protocol, *or* connect to another machine and view + control it (host + viewer GUIs included). Optional TLS (HTTPS-grade encryption), WebSocket transport (ws:// + wss:// for browser / firewall-friendly clients), persistent 9-digit Host ID, host→viewer audio streaming, bidirectional clipboard sync (text + image), and chunked file transfer (drag-drop + progress bar; arbitrary destination path; no size cap). Plus folder sync (additive mirror — local deletions never propagate) and a self-hosted coturn TURN config bundle generator (turnserver.conf + systemd unit + docker-compose + README). **AnyDesk-style popout**: when the viewer authenticates, the live remote desktop opens in its own resizable top-level window so the control panel stays uncluttered. The Remote Desktop tabs are wrapped in `QScrollArea` so the panel stays usable on small windows and stretches edge-to-edge on 4K displays. Driveable headlessly via `je_auto_control` and over MCP through the new `ac_remote_*` tools
+- **Driver-level input backends (opt-in)** — for games / apps that ignore SendInput (Win) or XTest (Linux): **Interception driver backend** for Windows (HID-layer keyboard / mouse injection via Oblita's WHQL-signed driver, opt-in via `JE_AUTOCONTROL_WIN32_BACKEND=interception`), **uinput backend** for Linux (kernel `/dev/uinput` synthetic HID device, opt-in via `JE_AUTOCONTROL_LINUX_BACKEND=uinput`), and **ViGEm virtual gamepad** for Windows games that read controllers (virtual Xbox 360 pad with friendly button / dpad / stick / trigger API, exposed as `AC_gamepad_*` executor commands and `ac_gamepad_*` MCP tools). All three fall back gracefully when the driver isn't installed, so existing deployments keep working unchanged
 - **Clipboard** — read/write system clipboard text on Windows, macOS, and Linux
 - **Screenshot & Screen Recording** — capture full screen or regions as images, record screen to video (AVI/MP4)
 - **Action Recording & Playback** — record mouse/keyboard events and replay them

README/README_zh-CN.md

@@ -63,6 +63,7 @@
 - **LLM 动作规划器** — 用 Claude 把自然语言描述翻译成验证过的 `AC_*` 动作清单
 - **运行期变量与流程控制** — 执行时 `${var}` 替换，加上 `AC_set_var` / `AC_inc_var` / `AC_if_var` / `AC_for_each` / `AC_loop` / `AC_retry` 让脚本数据驱动
 - **远程桌面** — 用 token 认证的 TCP 协议串流本机画面并接收输入，**或** 连接到他机观看与控制（host + viewer GUI 内置）。可选 TLS（HTTPS 级加密）、WebSocket 传输（``ws://`` + ``wss://``，穿墙／浏览器友好）、持久化 9 位数 Host ID、host→viewer 音频串流、双向剪贴板同步（文字 + 图片）、分块文件传输（拖放 + 进度条；任意目的路径；无大小上限）。另含文件夹同步（增量镜像 — 本地删除不会传出去）与自建 coturn TURN 配置包生成器（turnserver.conf + systemd unit + docker-compose + README）。**AnyDesk 风格弹出窗口**：viewer 认证成功后远程桌面会开在独立的可调整大小顶层窗口，控制面板保持简洁；Remote Desktop 子分页外层包了 `QScrollArea`，小窗口下可滚动、4K 屏幕下会铺满。同时支持 headless API 与 MCP 工具 (`ac_remote_*`) 直接驱动
+- **驱动级输入后端（可选）** — 针对忽略 SendInput（Win）或 XTest（Linux）的游戏／应用：**Interception driver 后端**（Windows，HID 层鍵鼠注入，使用 Oblita WHQL-signed driver，通过 `JE_AUTOCONTROL_WIN32_BACKEND=interception` 启用）、**uinput 后端**（Linux，kernel `/dev/uinput` 合成 HID 设备，通过 `JE_AUTOCONTROL_LINUX_BACKEND=uinput` 启用），以及 **ViGEm 虚拟手柄**（Windows，针对只认手柄的游戏，提供虚拟 Xbox 360 手柄 + 友善的 button / dpad / stick / trigger API，并暴露为 `AC_gamepad_*` 执行器命令与 `ac_gamepad_*` MCP 工具）。三者在 driver 没装时都会优雅 fallback，不影响既有部署
 - **剪贴板** — 于 Windows / macOS / Linux 读写系统剪贴板文本
 - **截图与屏幕录制** — 捕获全屏或指定区域为图片，录制屏幕为视频（AVI/MP4）
 - **动作录制与回放** — 录制鼠标/键盘事件并重新播放

README/README_zh-TW.md

@@ -63,6 +63,7 @@
 - **LLM 動作規劃器** — 用 Claude 把自然語言描述翻譯成驗證過的 `AC_*` 動作清單
 - **執行期變數與流程控制** — 執行時 `${var}` 取代，加上 `AC_set_var` / `AC_inc_var` / `AC_if_var` / `AC_for_each` / `AC_loop` / `AC_retry` 讓腳本資料驅動
 - **遠端桌面** — 用 token 認證的 TCP 協定串流本機畫面並接收輸入，**或** 連線到他機觀看與控制（host + viewer GUI 皆內建）。可選 TLS（HTTPS 級加密）、WebSocket 傳輸（``ws://`` + ``wss://``，穿牆／瀏覽器友善）、持久化 9 位數 Host ID、host→viewer 音訊串流、雙向剪貼簿同步（文字 + 圖片）、分塊檔案傳輸（拖放 + 進度條；任意目的路徑；無大小上限）。另含資料夾同步（增量鏡像 — 本地刪除不會傳出去）與自架 coturn TURN 設定包產生器（turnserver.conf + systemd unit + docker-compose + README）。**AnyDesk 風格彈出視窗**：viewer 認證成功後遠端桌面會開在獨立的可調整大小頂層視窗，控制面板維持簡潔；Remote Desktop 子分頁外層包了 `QScrollArea`，小視窗下可捲動、4K 螢幕下會延展到整寬。同時可由 headless API 與 MCP 工具（`ac_remote_*`）直接驅動
+- **驅動層輸入後端（可選）** — 針對忽略 SendInput（Win）或 XTest（Linux）的遊戲／應用:**Interception driver 後端**(Windows,HID 層鍵鼠注入,使用 Oblita WHQL-signed driver,以 `JE_AUTOCONTROL_WIN32_BACKEND=interception` 啟用)、**uinput 後端**(Linux,kernel `/dev/uinput` 合成 HID 裝置,以 `JE_AUTOCONTROL_LINUX_BACKEND=uinput` 啟用),以及 **ViGEm 虛擬手把**(Windows,針對只認手把的遊戲,提供虛擬 Xbox 360 手把 + 友善的 button / dpad / stick / trigger API,並暴露為 `AC_gamepad_*` 執行器指令與 `ac_gamepad_*` MCP 工具)。三者在 driver 沒裝時都會優雅 fallback,不影響既有部署
 - **剪貼簿** — 於 Windows / macOS / Linux 讀寫系統剪貼簿文字
 - **截圖與螢幕錄製** — 擷取全螢幕或指定區域為圖片，錄製螢幕為影片（AVI/MP4）
 - **動作錄製與回放** — 錄製滑鼠/鍵盤事件並重新播放

docs/source/Eng/doc/new_features/new_features_doc.rst

@@ -790,3 +790,123 @@ The status / observer tools (``ac_remote_host_status``,
 server's ``--readonly`` filter; everything that mutates state is
 correctly tagged ``destructiveHint: true`` so MCP clients can
 prompt for user confirmation.
+
+
+Driver-level input backends — drive games that ignore SendInput / XTest
+========================================================================
+
+The default Windows (SendInput) and Linux (XTest) input paths sit at
+the user-mode / X-server layer. Modern games that read input via
+``GetRawInputData`` (Win) or ``evdev`` (Linux) skip those layers
+entirely and ignore synthetic events. Three optional backends bridge
+the gap.
+
+Interception (Windows)
+----------------------
+
+Oblita's WHQL-signed Interception driver
+(https://github.com/oblitum/Interception) injects keyboard / mouse
+events at the HID layer; the OS sees them as real-hardware events.
+
+* New sub-package: ``je_auto_control/windows/interception/``
+  (``_dll.py`` ctypes bindings + ``keyboard.py`` + ``mouse.py``).
+* Same public surface as ``win32_ctype_keyboard_control`` /
+  ``win32_ctype_mouse_control`` — the platform wrapper just swaps
+  modules, no caller changes.
+* Opt-in via ``JE_AUTOCONTROL_WIN32_BACKEND=interception``; the
+  wrapper falls back to SendInput with a warning when the driver is
+  missing, so deployments can roll the driver out lazily.
+* Override device IDs with ``JE_AUTOCONTROL_INTERCEPTION_KEYBOARD``
+  / ``JE_AUTOCONTROL_INTERCEPTION_MOUSE`` (defaults: ``1`` / ``11``).
+
+Operator setup::
+
+   # 1. Install the driver as Administrator (one-time, requires reboot)
+   install-interception.exe /install
+
+   # 2. Tell AutoControl to route through it
+   setx JE_AUTOCONTROL_WIN32_BACKEND interception
+
+uinput (Linux)
+--------------
+
+The kernel's synthetic-input gateway. Events emitted via
+``/dev/uinput`` show up as a brand-new HID device, so anything reading
+``evdev`` (most games + SDL2 apps) sees them as real input.
+
+* New sub-package: ``je_auto_control/linux_with_x11/uinput/``
+  (``_device.py`` ctypes wrapper around ``ioctl`` + ``keyboard.py`` +
+  ``mouse.py``).
+* No third-party dependency — direct ``ctypes`` + ``ioctl`` to
+  ``/dev/uinput``.
+* Opt-in via ``JE_AUTOCONTROL_LINUX_BACKEND=uinput``; falls back to
+  XTest with a warning when ``/dev/uinput`` isn't writable.
+
+Operator setup::
+
+   # Load the kernel module if it isn't already.
+   sudo modprobe uinput
+
+   # Grant write access. For one-off testing:
+   sudo chmod 666 /dev/uinput
+
+   # For persistent provisioning, drop a udev rule:
+   echo 'KERNEL=="uinput", GROUP="input", MODE="0660"' \
+     | sudo tee /etc/udev/rules.d/99-autocontrol-uinput.rules
+   sudo udevadm control --reload && sudo udevadm trigger
+   sudo usermod -aG input $USER  # log out / back in to apply
+
+   # Then opt in:
+   export JE_AUTOCONTROL_LINUX_BACKEND=uinput
+
+ViGEm virtual gamepad (Windows)
+-------------------------------
+
+For games that don't take keyboard input at all but read controllers,
+ViGEmBus exposes a virtual Xbox 360 / DualShock 4 controller that
+AutoControl drives through the third-party ``vgamepad`` Python
+package.
+
+* New module: ``je_auto_control/utils/gamepad/`` with a friendly
+  ``VirtualGamepad`` API (string-keyed buttons / dpad / sticks /
+  triggers, context manager).
+* Headless::
+
+     from je_auto_control import VirtualGamepad
+     with VirtualGamepad() as pad:
+         pad.click_button("a")               # face button A
+         pad.set_left_stick(16000, 0)        # int16 stick offsets
+         pad.set_right_trigger(255)          # 0..255 pressure
+         pad.set_dpad("up")                  # hold dpad up
+         pad.update()                        # flush → driver
+
+* Executor commands: ``AC_gamepad_press``, ``AC_gamepad_release``,
+  ``AC_gamepad_click``, ``AC_gamepad_dpad``,
+  ``AC_gamepad_left_stick`` / ``_right_stick``,
+  ``AC_gamepad_left_trigger`` / ``_right_trigger``, and
+  ``AC_gamepad_reset``.
+
+* MCP tools: same names with the ``ac_`` prefix
+  (``ac_gamepad_press``, ``ac_gamepad_left_stick``, …) — so a model
+  can play a gamepad-only game over MCP.
+
+Operator setup::
+
+   # 1. Install the ViGEmBus driver (one-time, requires reboot)
+   #    https://github.com/nefarius/ViGEmBus/releases
+   # 2. Install the Python wrapper:
+   pip install vgamepad
+
+Anti-cheat caveat (all three)
+-----------------------------
+
+Driver-level injection is harder to detect than SendInput / XTest,
+but anti-cheat systems with a kernel-mode driver of their own
+(Vanguard, Easy Anti-Cheat with kernel module, BattlEye) can still
+enumerate Interception / ViGEmBus / a freshly-created uinput device
+and refuse to launch.
+
+These backends target legitimate use cases — accessibility software,
+GUI testing of games that lock out user-mode input, controlling a
+remote game-running machine from a headless setup — and aren't a
+generic anti-cheat bypass.

docs/source/Zh/doc/new_features/new_features_doc.rst

@@ -739,3 +739,118 @@ registry 包成工具,工廠函式為
 ``ac_remote_viewer_status``)為唯讀,可以通過 MCP server 的
 ``--readonly`` 過濾;會修改狀態的工具都正確帶上
 ``destructiveHint: true``,MCP client 端可以據此跳出使用者確認。
+
+
+驅動層輸入後端 — 驅動不接受 SendInput / XTest 的遊戲
+=====================================================
+
+預設的 Windows(SendInput)與 Linux(XTest)輸入路徑落在 user-mode
+/ X-server 那一層;會用 ``GetRawInputData``(Win)或 ``evdev``
+(Linux)直接讀 raw input 的遊戲會跳過這些層,完全忽略合成事件。
+新增三個可選的後端可以解決這個問題。
+
+Interception(Windows)
+------------------------
+
+Oblita 的 WHQL-signed Interception driver
+(https://github.com/oblitum/Interception)在 HID 層注入鍵鼠事件,
+OS 看到的就是「真實裝置」事件。
+
+* 新增子套件:``je_auto_control/windows/interception/``
+  (``_dll.py`` ctypes bindings + ``keyboard.py`` + ``mouse.py``)。
+* 與 ``win32_ctype_keyboard_control`` /
+  ``win32_ctype_mouse_control`` 公開介面完全一致 — wrapper 在啟動
+  時直接換模組,呼叫端不需要任何修改。
+* 透過 ``JE_AUTOCONTROL_WIN32_BACKEND=interception`` 啟用;若
+  driver 沒裝,wrapper 會打 warning 並回到 SendInput,所以可以分
+  階段佈署。
+* 用 ``JE_AUTOCONTROL_INTERCEPTION_KEYBOARD`` /
+  ``JE_AUTOCONTROL_INTERCEPTION_MOUSE`` 覆寫 device id(預設
+  ``1`` / ``11``)。
+
+操作步驟::
+
+   # 1. 以系統管理員身份安裝 driver(一次性,需要重開機)
+   install-interception.exe /install
+
+   # 2. 告訴 AutoControl 走這條路
+   setx JE_AUTOCONTROL_WIN32_BACKEND interception
+
+uinput(Linux)
+----------------
+
+kernel 自帶的合成輸入閘道。透過 ``/dev/uinput`` 送出的事件會被
+建立成一個全新的 HID 裝置,任何讀 ``evdev`` 的程式(包含大部分
+遊戲與 SDL2 app)都會視為真實輸入。
+
+* 新增子套件:``je_auto_control/linux_with_x11/uinput/``
+  (``_device.py`` 直接用 ctypes + ioctl 包 ``/dev/uinput`` +
+  ``keyboard.py`` + ``mouse.py``)。
+* 無第三方依賴 — 全程 ctypes + ioctl。
+* 透過 ``JE_AUTOCONTROL_LINUX_BACKEND=uinput`` 啟用;若
+  ``/dev/uinput`` 沒寫入權限,會 warning 後回退到 XTest。
+
+操作步驟::
+
+   # 載入 kernel module
+   sudo modprobe uinput
+
+   # 一次性測試:直接放寬權限
+   sudo chmod 666 /dev/uinput
+
+   # 持續性權限,寫一個 udev rule:
+   echo 'KERNEL=="uinput", GROUP="input", MODE="0660"' \
+     | sudo tee /etc/udev/rules.d/99-autocontrol-uinput.rules
+   sudo udevadm control --reload && sudo udevadm trigger
+   sudo usermod -aG input $USER  # 重新登入後生效
+
+   # 啟用後端
+   export JE_AUTOCONTROL_LINUX_BACKEND=uinput
+
+ViGEm 虛擬手把(Windows)
+-------------------------
+
+針對「完全不吃鍵鼠、只認手把」的遊戲,可以用 ViGEmBus 建立一個虛
+擬 Xbox 360 / DualShock 4 控制器;AutoControl 透過第三方 ``vgamepad``
+套件來驅動它。
+
+* 新增模組:``je_auto_control/utils/gamepad/`` 提供友善的
+  ``VirtualGamepad`` API(字串名稱的 button / dpad / stick /
+  trigger,支援 context manager)。
+* Headless::
+
+     from je_auto_control import VirtualGamepad
+     with VirtualGamepad() as pad:
+         pad.click_button("a")               # A 鍵
+         pad.set_left_stick(16000, 0)        # int16 stick 偏移
+         pad.set_right_trigger(255)          # 0..255 力度
+         pad.set_dpad("up")                  # 按住方向鍵上
+         pad.update()                        # 把狀態 flush 給 driver
+
+* Executor 指令:``AC_gamepad_press``、``AC_gamepad_release``、
+  ``AC_gamepad_click``、``AC_gamepad_dpad``、
+  ``AC_gamepad_left_stick`` / ``_right_stick``、
+  ``AC_gamepad_left_trigger`` / ``_right_trigger``,以及
+  ``AC_gamepad_reset``。
+
+* MCP 工具:同名加上 ``ac_`` 前綴(``ac_gamepad_press``、
+  ``ac_gamepad_left_stick`` …),所以模型可以透過 MCP 玩只支援
+  手把的遊戲。
+
+操作步驟::
+
+   # 1. 安裝 ViGEmBus driver(一次性,需要重開機)
+   #    https://github.com/nefarius/ViGEmBus/releases
+   # 2. 安裝 Python wrapper:
+   pip install vgamepad
+
+反作弊注意事項
+---------------
+
+驅動層注入比 SendInput / XTest 更難偵測,但帶 kernel-mode driver
+的反作弊(Vanguard、有 kernel module 的 Easy Anti-Cheat、
+BattlEye)依然可以列舉 Interception / ViGEmBus / 新建立的 uinput
+裝置然後拒絕啟動。
+
+這三個後端針對的是合法用途 — 輔助科技、遊戲 GUI 測試、從 headless
+環境控制執行遊戲的遠端機器 — **不是** 通用反作弊繞過工具。

je_auto_control/__init__.py

@@ -78,6 +78,11 @@
 from je_auto_control.utils.remote_desktop.registry import (
     registry as remote_desktop_registry,
 )
+from je_auto_control.utils.gamepad import (
+    GamepadUnavailable, VirtualGamepad,
+    default_gamepad as default_virtual_gamepad,
+    is_available as is_virtual_gamepad_available,
+)
 # MCP server (headless stdio bridge for Claude / other MCP clients)
 from je_auto_control.utils.mcp_server import (
     AuditLogger, HttpMCPServer, MCPContent, MCPPrompt, MCPPromptArgument,
@@ -122,6 +127,15 @@
     ConfigBundleExporter, ConfigBundleImporter, ImportReport,
     export_config_bundle, import_config_bundle,
 )
+# Profiler (headless)
+from je_auto_control.utils.profiler import (
+    ActionProfiler, ActionStats, default_profiler,
+)
+# Secrets (headless)
+from je_auto_control.utils.secrets import (
+    SecretManager, SecretStoreError, SecretStoreLocked,
+    default_secret_manager, default_secret_store_path,
+)
 # Run history (headless)
 from je_auto_control.utils.run_history.history_store import (
     HistoryStore, RunRecord, default_history_store,
@@ -131,6 +145,12 @@
     FilePathTrigger, ImageAppearsTrigger, PixelColorTrigger, TriggerEngine,
     WindowAppearsTrigger, default_trigger_engine,
 )
+from je_auto_control.utils.triggers.webhook_server import (
+    WebhookTrigger, WebhookTriggerServer, default_webhook_server,
+)
+from je_auto_control.utils.triggers.email_trigger import (
+    EmailTrigger, EmailTriggerWatcher, default_email_trigger_watcher,
+)
 # Recording editor (headless helpers)
 from je_auto_control.utils.recording_edit.editor import (
     adjust_delays, filter_actions, insert_action, remove_action,
@@ -297,6 +317,14 @@ def start_autocontrol_gui(*args, **kwargs):
     "TriggerEngine", "default_trigger_engine",
     "ImageAppearsTrigger", "WindowAppearsTrigger",
     "PixelColorTrigger", "FilePathTrigger",
+    "WebhookTrigger", "WebhookTriggerServer", "default_webhook_server",
+    "EmailTrigger", "EmailTriggerWatcher",
+    "default_email_trigger_watcher",
+    # Profiler
+    "ActionProfiler", "ActionStats", "default_profiler",
+    # Secret manager
+    "SecretManager", "SecretStoreError", "SecretStoreLocked",
+    "default_secret_manager", "default_secret_store_path",
     # Run history
     "HistoryStore", "RunRecord", "default_history_store",
     # Accessibility
@@ -312,6 +340,9 @@ def start_autocontrol_gui(*args, **kwargs):
     "RemoteDesktopHost", "RemoteDesktopViewer",
     "RemoteDesktopAuthError", "RemoteDesktopInputError",
     "RemoteDesktopProtocolError", "remote_desktop_registry",
+    # Virtual gamepad (ViGEm)
+    "VirtualGamepad", "GamepadUnavailable",
+    "default_virtual_gamepad", "is_virtual_gamepad_available",
     "generate_html", "generate_html_report", "generate_json", "generate_json_report", "generate_xml",
     "generate_xml_report", "get_dir_files_as_list", "create_project_dir", "start_autocontrol_socket_server",
     "callback_executor", "package_manager", "ShellManager", "default_shell_manager",