Skip to content

Commit 3696152

Browse files
committed
Update READMEs (features + mermaid + directory tree) for rounds 22-47
EN / zh-TW / zh-CN, kept structurally parallel (38 feature bullets each). Features section gains: - Folder sync + coturn TURN bundle (folded into Remote Desktop) - Hardened Remote Automation: bearer auth, rate limit, audit hook, Prometheus, dashboard, full endpoint inventory - Multi-host admin console - Tamper-evident audit log (SHA-256 hash chain) - WebRTC packet inspector - USB device enumeration + hotplug events - System diagnostics - OpenAPI 3.1 + Swagger UI at /docs - Configuration bundle export/import - USB passthrough chain (host + viewer + ACL + libusb / WinUSB skel / IOKit skel) — clearly marked experimental + opt-in Mermaid architecture diagram extended with: - Browser client surface (/dashboard + /docs) - WebRTC sessions transport row - Operations Layer subgraph (admin, audit, inspector, diagnostics, config_bundle) - USB subgraph (enumeration + hotplug + passthrough) - Remote Desktop subgraph - New cross-edges: REST → Ops/USB, WebRTC → Remote/UsbPass, GUIUser/Library → Ops, Audit ⤳ REST/USB, UsbPass → Backends Directory tree adds: - utils/llm/, utils/admin/, utils/diagnostics/, utils/config_bundle/, utils/usb/ (with passthrough/ subdir contents listed), utils/remote_desktop/ (one-line summary of its 30+ files) - REST description expanded to mention auth / audit / OpenAPI / metrics / dashboard / Swagger UI
1 parent 7fd79bb commit 3696152

3 files changed

Lines changed: 174 additions & 12 deletions

File tree

README.md

Lines changed: 58 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@
6363
- **OCR** — extract text from screen regions using Tesseract; wait for, click, or locate rendered text; regex search and full-region dump
6464
- **LLM Action Planner** — translate a plain-language description into a validated `AC_*` action list using Claude
6565
- **Runtime Variables & Control Flow**`${var}` substitution at execution time, plus `AC_set_var` / `AC_inc_var` / `AC_if_var` / `AC_for_each` / `AC_loop` / `AC_retry` for data-driven scripts
66-
- **Remote Desktop** — stream this machine's screen and accept remote input over a token-authenticated TCP protocol, *or* connect to another machine and view + control it (host + viewer GUIs included). Optional TLS (HTTPS-grade encryption), WebSocket transport (ws:// + wss:// for browser / firewall-friendly clients), persistent 9-digit Host ID, host→viewer audio streaming, bidirectional clipboard sync (text + image), and chunked file transfer (drag-drop + progress bar; arbitrary destination path; no size cap)
66+
- **Remote Desktop** — stream this machine's screen and accept remote input over a token-authenticated TCP protocol, *or* connect to another machine and view + control it (host + viewer GUIs included). Optional TLS (HTTPS-grade encryption), WebSocket transport (ws:// + wss:// for browser / firewall-friendly clients), persistent 9-digit Host ID, host→viewer audio streaming, bidirectional clipboard sync (text + image), and chunked file transfer (drag-drop + progress bar; arbitrary destination path; no size cap). Plus folder sync (additive mirror — local deletions never propagate) and a self-hosted coturn TURN config bundle generator (turnserver.conf + systemd unit + docker-compose + README)
6767
- **Clipboard** — read/write system clipboard text on Windows, macOS, and Linux
6868
- **Screenshot & Screen Recording** — capture full screen or regions as images, record screen to video (AVI/MP4)
6969
- **Action Recording & Playback** — record mouse/keyboard events and replay them
@@ -74,7 +74,7 @@
7474
- **Run History** — SQLite-backed run log across scheduler / triggers / hotkeys / REST with auto error-screenshot artifacts
7575
- **Report Generation** — export test records as HTML, JSON, or XML reports with success/failure status
7676
- **MCP Server** — JSON-RPC 2.0 Model Context Protocol server (stdio + HTTP/SSE) so Claude Desktop / Claude Code / custom tool-use loops can drive AutoControl. ~90 tools, full protocol coverage (resources, prompts, sampling, roots, logging, progress, cancellation, elicitation), bearer-token auth + TLS, audit log, rate limit, plugin hot-reload, CI fake backend
77-
- **Remote Automation** — TCP socket server **and** REST API server to receive automation commands
77+
- **Remote Automation** — TCP socket server **and** hardened REST API: bearer-token auth, per-IP rate limit + lockout, SQLite audit hook, Prometheus `/metrics`, OpenAPI-style endpoint table (`/health`, `/screen_size`, `/sessions`, `/screenshot`, `/execute`, `/audit/list`, `/audit/verify`, `/inspector/recent`, `/usb/devices`, `/diagnose`, ...), and a vanilla-JS browser dashboard at `/dashboard` (any phone with HTTP reach can monitor the host)
7878
- **Plugin Loader** — drop `.py` files exposing `AC_*` callables into a directory and register them as executor commands at runtime
7979
- **Shell Integration** — execute shell commands within automation workflows with async output capture
8080
- **Callback Executor** — trigger automation functions with callback hooks for chaining operations
@@ -84,6 +84,15 @@
8484
- **GUI Application** — built-in PySide6 graphical interface with live language switching (English / 繁體中文 / 简体中文 / 日本語)
8585
- **CLI Runner**`python -m je_auto_control.cli run|list-jobs|start-server|start-rest`
8686
- **Cross-Platform** — unified API across Windows, macOS, and Linux (X11)
87+
- **Multi-Host Admin Console** — register N AutoControl REST endpoints in one address book, poll them in parallel for health/sessions/jobs, broadcast actions to all of them. Persisted to `~/.je_auto_control/admin_hosts.json` (mode 0600 on POSIX). Bad-token hosts surface as unhealthy with the actual HTTP error
88+
- **Tamper-Evident Audit Log** — SQLite events table with SHA-256 hash chain (`prev_hash` + `row_hash` per row); editing any past row breaks the chain. `verify_chain()` walks rows top-down and reports the first broken link. Legacy tables get backfilled at startup ("trust on first use")
89+
- **WebRTC Packet Inspector** — process-global rolling window of `StatsSnapshot` samples (default 600 / ~10 min @ 1Hz) fed by the existing WebRTC stats pollers. Per-metric `last/min/max/avg/p95` for RTT, FPS, bitrate, packet loss, jitter
90+
- **USB Device Enumeration** — read-only cross-platform device listing. Tries pyusb (libusb) first; falls back to platform-specific (Windows `Get-PnpDevice`, macOS `system_profiler`, Linux `/sys/bus/usb/devices`). Phase 2 (passthrough) intentionally deferred pending design review
91+
- **System Diagnostics** — single-command "is everything OK?" probe across platform, optional deps, executor command count, audit chain, screenshot, mouse, disk space, REST registry. CLI exits 0 if all green / 1 otherwise; REST `/diagnose`; severity-tagged GUI tab
92+
- **USB Hotplug Events** — polling-based hotplug watcher (`UsbHotplugWatcher`) with bounded ring buffer + sequence-numbered events; `GET /usb/events?since=N` lets late subscribers catch up. GUI auto-refresh toggle on the USB tab.
93+
- **OpenAPI 3.1 + Swagger UI**`GET /openapi.json` (auth-gated, generated from the live route table) + `GET /docs` (browser Swagger UI with bearer token bar). Drift test in CI catches new routes added without metadata.
94+
- **Configuration Bundle** — single-file JSON export/import of user config (admin hosts, address book, trusted viewers, known hosts, host service, IDs). Atomic write with `<name>.bak.<timestamp>` backups; CLI `python -m je_auto_control.utils.config_bundle export|import`; `POST /config/{export,import}`; GUI buttons on the REST API tab.
95+
- **USB Passthrough (experimental, opt-in)** — wire-level protocol over a WebRTC `usb` DataChannel (10 opcodes, CREDIT-based flow control, 16 KiB payload cap). Host-side `UsbPassthroughSession` end-to-end on the Linux libusb backend; Windows `WinUSB` backend with full ctypes wiring (hardware-unverified); macOS `IOKit` skeleton. Viewer-side blocking client (`UsbPassthroughClient``ClientHandle.control_transfer / bulk_transfer / interrupt_transfer`). Persistent ACL (`~/.je_auto_control/usb_acl.json`, default deny, mode 0600) with host-side prompt QDialog and tamper-evident audit-log integration. Default off — opt-in via `enable_usb_passthrough(True)` or `JE_AUTOCONTROL_USB_PASSTHROUGH=1`. Phase 2e external security review checklist included; default-on requires sign-off.
8796

8897
---
8998

@@ -105,6 +114,7 @@ flowchart LR
105114
APIUser[["Custom Anthropic /<br/>OpenAI tool loops"]]
106115
HTTPClient[["HTTP / SSE clients"]]
107116
TCPClient[["Socket / REST clients"]]
117+
Browser[["Browser<br/>(/dashboard · /docs)"]]
108118
GUIUser[["PySide6 GUI"]]
109119
CLIUser[["python -m<br/>je_auto_control[.cli]"]]
110120
Library[["Library users<br/>(import je_auto_control)"]]
@@ -114,8 +124,9 @@ flowchart LR
114124
direction TB
115125
Stdio["MCP stdio<br/>JSON-RPC 2.0"]
116126
HTTPMCP["MCP HTTP /<br/>SSE + auth + TLS"]
117-
REST["REST server<br/>:9939"]
127+
REST["REST server :9939<br/>bearer auth · rate-limit ·<br/>OpenAPI · /metrics · /dashboard"]
118128
Socket["Socket server<br/>:9938"]
129+
WebRTC["WebRTC sessions<br/>(remote desktop ·<br/>files · audio · USB)"]
119130
end
120131
121132
subgraph MCP["mcp_server/"]
@@ -137,6 +148,28 @@ flowchart LR
137148
IOUtils["clipboard/ · cv2_utils/ ·<br/>shell_process/ · json/"]
138149
end
139150
151+
subgraph Ops["Operations Layer (utils/)"]
152+
direction TB
153+
Admin["admin/<br/>multi-host poll +<br/>broadcast"]
154+
Audit["remote_desktop/<br/>audit_log<br/>(SHA-256 chain)"]
155+
Inspector["remote_desktop/<br/>webrtc_inspector"]
156+
Diag["diagnostics/<br/>self-test"]
157+
ConfigB["config_bundle/<br/>export/import"]
158+
end
159+
160+
subgraph USB["USB"]
161+
direction TB
162+
UsbEnum["usb/<br/>list + hotplug events"]
163+
UsbPass["usb/passthrough/<br/>session · client · ACL ·<br/>libusb · WinUSB · IOKit"]
164+
end
165+
166+
subgraph Remote["Remote Desktop (utils/remote_desktop/)"]
167+
direction TB
168+
RDHost["host · webrtc_host ·<br/>signaling · multi_viewer"]
169+
RDFiles["webrtc_files · file_sync ·<br/>clipboard_sync · audio"]
170+
RDTrust["trust_list · fingerprint ·<br/>turn_config · lan_discovery"]
171+
end
172+
140173
subgraph Backends["Per-OS Backends"]
141174
direction TB
142175
Win["windows/<br/>Win32 ctypes"]
@@ -149,6 +182,7 @@ flowchart LR
149182
HTTPClient --> HTTPMCP
150183
TCPClient --> Socket
151184
TCPClient --> REST
185+
Browser --> REST
152186
153187
Stdio --> Dispatcher
154188
HTTPMCP --> Dispatcher
@@ -167,13 +201,27 @@ flowchart LR
167201
Resources --> Wrapper
168202
169203
REST --> Executor
204+
REST --> Ops
205+
REST --> USB
170206
Socket --> Executor
207+
WebRTC --> Remote
208+
WebRTC --> UsbPass
171209
172210
GUIUser --> Wrapper
173211
GUIUser --> Recorder
212+
GUIUser --> Ops
213+
GUIUser --> USB
214+
GUIUser --> Remote
174215
CLIUser --> Executor
175216
Library --> Wrapper
176217
Library --> Executor
218+
Library --> Ops
219+
220+
Admin --> REST
221+
Inspector -.- WebRTC
222+
Audit -.- REST
223+
Audit -.- USB
224+
UsbPass --> Backends
177225
178226
Wrapper --> Backends
179227
Vision -.- Wrapper
@@ -203,11 +251,17 @@ je_auto_control/
203251
├── vision/ # VLM-based locator (Anthropic / OpenAI backends)
204252
├── ocr/ # Tesseract-backed text locator
205253
├── clipboard/ # Cross-platform clipboard (text + image)
254+
├── llm/ # Plain-language → AC_* action planner
206255
├── scheduler/ # Interval + cron scheduler
207256
├── hotkey/ # Global hotkey daemon
208257
├── triggers/ # Image/window/pixel/file triggers
209258
├── run_history/ # SQLite run log + error-screenshot artifacts
210-
├── rest_api/ # Stdlib HTTP/REST server
259+
├── rest_api/ # Stdlib HTTP/REST server — auth · audit · rate-limit · OpenAPI · /metrics · dashboard · Swagger UI
260+
├── admin/ # Multi-host AdminConsoleClient (poll + broadcast)
261+
├── diagnostics/ # System self-test runner + CLI
262+
├── config_bundle/ # Single-file user-config export / import
263+
├── usb/ # Cross-platform enumeration, hotplug events, passthrough/{protocol, session, viewer client, ACL, libusb / WinUSB / IOKit}
264+
├── remote_desktop/ # WebRTC host + viewer, signalling, multi-viewer, file/clipboard/audio sync, audit log (hash chain), trust list, TURN config, mDNS discovery, WebRTC stats inspector
211265
├── plugin_loader/ # Dynamic AC_* plugin discovery
212266
├── socket_server/ # TCP socket server for remote automation
213267
├── shell_process/ # Shell command manager

0 commit comments

Comments
 (0)