You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Expose remote desktop over MCP and document recent UX changes
MCP tool surface
- New ``remote_desktop_tools()`` factory in
``je_auto_control/utils/mcp_server/tools/_factories.py`` exposes
the same singleton remote-desktop registry the GUI's Remote
Desktop tab uses:
ac_remote_host_start (token, bind, port, fps, quality, …)
ac_remote_host_stop (timeout)
ac_remote_host_status (read-only)
ac_remote_viewer_connect (host, port, token, expected_host_id)
ac_remote_viewer_disconnect (timeout)
ac_remote_viewer_status (read-only)
ac_remote_viewer_send_input (action: dict)
- Adapter handlers in ``_handlers.py`` import the registry lazily so
the existing tool group stays cheap to load.
- Status / observer tools (`*_status`) carry ``READ_ONLY`` so they
survive ``--readonly`` mode; ``send_input`` is correctly tagged
``destructiveHint`` so MCP clients can prompt for confirmation.
Tests
- ``test_mcp_server.test_remote_desktop_tools_are_registered`` —
schema + annotation sanity check.
- ``test_mcp_server.test_remote_desktop_status_tools_survive_read_only_mode``
— confirms the read-only filter keeps status tools and drops the
destructive ones.
- 591 / 591 headless pytest pass; ruff clean.
Docs
- ``docs/source/Eng|Zh/doc/new_features/new_features_doc.rst`` —
three new sections: AnyDesk-style popout viewer window,
responsive ``QScrollArea`` sub-tab sizing, and the new
``ac_remote_*`` MCP tool surface (with a worked example).
- ``docs/source/Eng|Zh/doc/mcp_server/mcp_server_doc.rst`` — the
Remote Desktop tool group is listed in the tool catalogue.
- ``README.md`` and the two CN/TW READMEs — Remote Desktop entry
now mentions the popout, ``QScrollArea`` resizing, and the
headless / MCP driveability; MCP entry highlights the new
``ac_remote_*`` tools and the bumped tool count (~100).
Copy file name to clipboardExpand all lines: README.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -63,7 +63,7 @@
63
63
-**OCR** — extract text from screen regions using Tesseract; wait for, click, or locate rendered text; regex search and full-region dump
64
64
-**LLM Action Planner** — translate a plain-language description into a validated `AC_*` action list using Claude
65
65
-**Runtime Variables & Control Flow** — `${var}` substitution at execution time, plus `AC_set_var` / `AC_inc_var` / `AC_if_var` / `AC_for_each` / `AC_loop` / `AC_retry` for data-driven scripts
66
-
-**Remote Desktop** — stream this machine's screen and accept remote input over a token-authenticated TCP protocol, *or* connect to another machine and view + control it (host + viewer GUIs included). Optional TLS (HTTPS-grade encryption), WebSocket transport (ws:// + wss:// for browser / firewall-friendly clients), persistent 9-digit Host ID, host→viewer audio streaming, bidirectional clipboard sync (text + image), and chunked file transfer (drag-drop + progress bar; arbitrary destination path; no size cap). Plus folder sync (additive mirror — local deletions never propagate) and a self-hosted coturn TURN config bundle generator (turnserver.conf + systemd unit + docker-compose + README)
66
+
- **Remote Desktop** — stream this machine's screen and accept remote input over a token-authenticated TCP protocol, *or* connect to another machine and view + control it (host + viewer GUIs included). Optional TLS (HTTPS-grade encryption), WebSocket transport (ws:// + wss:// for browser / firewall-friendly clients), persistent 9-digit Host ID, host→viewer audio streaming, bidirectional clipboard sync (text + image), and chunked file transfer (drag-drop + progress bar; arbitrary destination path; no size cap). Plus folder sync (additive mirror — local deletions never propagate) and a self-hosted coturn TURN config bundle generator (turnserver.conf + systemd unit + docker-compose + README). **AnyDesk-style popout**: when the viewer authenticates, the live remote desktop opens in its own resizable top-level window so the control panel stays uncluttered. The Remote Desktop tabs are wrapped in `QScrollArea` so the panel stays usable on small windows and stretches edge-to-edge on 4K displays. Driveable headlessly via `je_auto_control` and over MCP through the new `ac_remote_*` tools
67
67
-**Clipboard** — read/write system clipboard text on Windows, macOS, and Linux
68
68
-**Screenshot & Screen Recording** — capture full screen or regions as images, record screen to video (AVI/MP4)
69
69
-**Action Recording & Playback** — record mouse/keyboard events and replay them
@@ -73,7 +73,7 @@
73
73
-**Event Triggers** — fire scripts when an image appears, a window opens, a pixel changes, or a file is modified
74
74
-**Run History** — SQLite-backed run log across scheduler / triggers / hotkeys / REST with auto error-screenshot artifacts
75
75
-**Report Generation** — export test records as HTML, JSON, or XML reports with success/failure status
76
-
-**MCP Server** — JSON-RPC 2.0 Model Context Protocol server (stdio + HTTP/SSE) so Claude Desktop / Claude Code / custom tool-use loops can drive AutoControl. ~90 tools, full protocol coverage (resources, prompts, sampling, roots, logging, progress, cancellation, elicitation), bearer-token auth + TLS, audit log, rate limit, plugin hot-reload, CI fake backend
76
+
-**MCP Server** — JSON-RPC 2.0 Model Context Protocol server (stdio + HTTP/SSE) so Claude Desktop / Claude Code / custom tool-use loops can drive AutoControl. ~100 tools, full protocol coverage (resources, prompts, sampling, roots, logging, progress, cancellation, elicitation), bearer-token auth + TLS, audit log, rate limit, plugin hot-reload, CI fake backend. New in this release: `ac_remote_host_start` / `ac_remote_host_stop` / `ac_remote_host_status` / `ac_remote_viewer_connect` / `ac_remote_viewer_disconnect` / `ac_remote_viewer_status` / `ac_remote_viewer_send_input` wrap the same singleton remote-desktop registry the GUI uses, so a model can spin up a host, open a viewer to another machine, and forward mouse / keyboard / type / hotkey actions through the active session
77
77
-**Remote Automation** — TCP socket server **and** hardened REST API: bearer-token auth, per-IP rate limit + lockout, SQLite audit hook, Prometheus `/metrics`, OpenAPI-style endpoint table (`/health`, `/screen_size`, `/sessions`, `/screenshot`, `/execute`, `/audit/list`, `/audit/verify`, `/inspector/recent`, `/usb/devices`, `/diagnose`, ...), and a vanilla-JS browser dashboard at `/dashboard` (any phone with HTTP reach can monitor the host)
78
78
-**Plugin Loader** — drop `.py` files exposing `AC_*` callables into a directory and register them as executor commands at runtime
79
79
-**Shell Integration** — execute shell commands within automation workflows with async output capture
0 commit comments