gui-agent

Monolithic FastMCP Server for Linux & Windows Desktop Computer Use

The Philosophy: Why gui-agent?

Autonomous AI agents interacting with modern graphical user interfaces are frequently burdened by fragmented architectures, high latency, and fragile computer vision loops. Traditional automation setups force models to perceive operating systems exclusively through repetitive raster screenshots, incurring prohibitive round-trip times (RTT) of 2 to 5 seconds per motor action. This reductionist approach causes severe spatial misalignments under fractional scaling, loses transient UI components like disappearing toasts or click-away menus, and quickly exhausts context windows with redundant image payloads.

gui-agent redefines desktop interaction by aligning agent decisions with the actual ontology of modern operating systems: structured processes in RAM, accessibility trees (AT-SPI2 / D-Bus), window compositors (X11 / Wayland), and kernel event subsystems (uinput, evdev). Built on two foundational pillars—Progressive Escalation (L3/L2/L1) and the CodeAct Local REPL Engine—the server enables models to actuate targets in RAM within 50 ms via native Rust mediation (gui-agent-atspi), fallback to local RapidOCR or calibrated Cartesian grids when needed, and execute entire multi-step action sequences locally in host memory with sub-5ms latency.

Operating through a single, resilient standard input/output (stdio) FastMCP connection, gui-agent functions with a lean baseline memory footprint below 50 MB RAM, fully preserving dual-core host responsiveness without external cloud vision dependencies or persistent background daemons. Platform implementations reside in dedicated, sealed root directories (linux/, windows/, macos/) with dynamic XDG Base Directory path resolution, zero hardcoded user paths, and full continuous screen video recording preservation (gui_start_video_recording, gui_stop_video_recording) for deterministic auditing.

Core Capabilities

The table below outlines the Target Architecture v1.0 (15 unified primitives) specifying the target contract across the modular roadmap, alongside the active MCP tools currently exposed by the Linux runtime under their operational gui_* and AT-SPI2 namespaces.

Tool Name	Domain	Description	Status
`execute_action_batch`		Executes local Python/Bash scripts with preloaded `mcp_core` (Open Interpreter model), running multi-action batches without RTT (Phase 2).
`process_run`		Spawns interactive shell commands via PTY to inject credentials and bypass Polkit/sudo security dialogs (Phase 3).
`process_list`		Inspects the `/proc` filesystem in read-only mode to probe active processes without altering desktop state (Phase 3).
`activate_window`		Switches focus at the display compositor level using unique Window IDs (active runtime alias: `gui_window_focus`).
`get_app_state`		Inspects the AT-SPI2 accessibility tree via native Rust mediation (`gui-agent-atspi`) directly in RAM.
`perform_action`		Triggers semantic actions directly in RAM via D-Bus IPC in sub-50ms without physical pointer movements.
`set_value`		Mutates text or numerical values directly into component memory variables without physical keystrokes.
`find_text`		Discovers on-screen text coordinates via local OCR without sending image payloads (active runtime alias: `gui_find_text`).
`screen_capture`		Captures raw framebuffer with calibrated Cartesian grid overlays for opaque surfaces (active runtime alias: `gui_take_screenshot`).
`mouse_click_at`		Injects physical mouse click events (`left`, `right`, `middle`, double) at target `(x, y)` coordinates (active runtime alias: `gui_mouse_click`).
`mouse_drag_smooth`		Dispatches continuous kinematic mouse trajectory interpolation to overcome UI drag thresholds (active runtime alias: `gui_mouse_drag`).
`mouse_scroll`		Executes hardware wheel scrolls to materialize virtualized DOM elements (active runtime alias: `gui_mouse_scroll`).
`key_tap`		Injects standard keystrokes, navigation hotkeys, and chord combinations (active runtime aliases: `gui_keyboard_press`, `gui_keyboard_type`).
`gui_start_video_recording`		Starts background low-overhead screen video recording via FFmpeg (`x11grab` / H.264 ultrafast).
`gui_stop_video_recording`		Cleanly halts the active FFmpeg recording, flushes the MP4 container, and prevents descriptor leaks.

How It Works

gui-agent operates as a closed-loop Computer Use bridge between frontier LLM reasoning engines and the host operating system, combining semantic RAM actuation, local script execution, and visual-motor feedback.

Technical Execution Pipeline & Foundational Pillars

Pillar I : Progressive Escalation & Layered Actuation: Rather than enforcing a single interaction mode, the architecture prioritizes cognitive and execution efficiency across layered stages: Level L3 accesses the OS accessibility tree (AT-SPI2 / D-Bus via the compiled Rust mediator gui-agent-atspi) directly in RAM for deterministic sub-50ms actuation with zero image tokens; Level L2 runs decoupled local OCR (RapidOCR/Tesseract) on typography without model inference overhead; Level L1 operates as the ultimate hardware safety net using calibrated Cartesian grid screenshots with native input dispatchers (with direct kernel uinput/evdev drivers scheduled on the roadmap); and an interactive PTY shell layer provides seamless handling of privileged commands.
Pillar II : High-Efficiency Execution Architecture: Paving the way to eliminate multi-turn network round-trip time (RTT) latency, the project architecture designs an isolated local execution environment (execute_action_batch via core/repl.py). Models will project multi-step inspection and action logic directly as Python code executed in host memory via the unified mcp_core SDK. Complex condition checking, kinematic drag calculations, and dynamic polling resolve in a single cognitive round-trip with sub-5ms execution speed and less than 15 MB RAM consumption (slated for Phase 2 roadmap).
Sub-second Screen Ingestion & Cartesian Grid Overlay: When an agent requests visual state via gui_take_screenshot, the server captures the raw framebuffer through MSS, with automatic fallback to KDE Spectacle or Scrot on XWayland surfaces. The engine overlays a millimeter Cartesian coordinate grid with adaptive contrast-buffered labels at configurable intervals (e.g., 100px), allowing models to infer target coordinates with mathematical certainty.
Dual Coordinate Normalization Engine: The server accepts coordinates in either absolute physical pixels (x, y) or normalized ratios [0, 1000] across any display geometry or multi-monitor setup. An automatic converter handles boundary clamping, DPI scaling, and coordinate translation transparently.
Native OS Input & Window Dispatcher: Keystrokes, hotkeys, mouse clicks, and drag operations are routed through low-latency native drivers (xdotool and python-xlib under Linux, Win32 API under Windows). Humanized delays and micro-jitter emulate natural user interaction. Window management commands (wmctrl / xprop) inspect and manipulate window states without window manager locks.
Local Vision, OCR & Playwright Automation: Template matching (cv2.matchTemplate) enables robust icon detection even under theme variations. Text discovery combines Tesseract OCR with RapidOCR ONNX fallback. Web automation leverages Playwright to inspect ARIA trees and manipulate DOM nodes directly without visual ambiguity.

Multi-Platform Root Architecture & Dynamic Path Resolution

The codebase organizes platform implementations into dedicated root directories with zero hardcoded filesystem paths:

linux/ : Complete Linux implementation featuring the core server (server.py), native Rust AT-SPI2 / D-Bus mediator (linux/crates/atspi_mediator compiled to gui-agent-atspi), automated install/uninstall scripts (install.sh, uninstall.sh), dedicated tests and examples, and dynamic XDG Base Directory path resolution (paths.py).
windows/ : Dedicated Windows directory (install.ps1, uninstall.ps1, native UI Automation backend in active development).
macos/ : Dedicated macOS directory reserved for upcoming NSAccessibility and Quartz Event Taps implementations. All runtime paths—including screenshots ($XDG_CACHE_HOME/gui-agent/screenshots or GUI_AGENT_SCREENSHOTS_DIR), persistent continuous video captures ($XDG_CACHE_HOME/gui-agent/videos or GUI_AGENT_VIDEOS_DIR), and data storage ($XDG_DATA_HOME/gui-agent)—are resolved dynamically at runtime.

Installation

For detailed OS-specific instructions, troubleshooting matrices, and offline setups, see the Detailed Installation Guide (INSTALL.md).

1. Automated Installation (Recommended)

Linux (Bash)

Run the automated installer to check dependencies, install Astral uv, build the native Rust mediator, and register the MCP server:

# Download and execute the automated installer via curl
curl -fsSL https://raw.githubusercontent.com/leandre755/gui_agent/7a49514/linux/install.sh | bash

# Or execute locally from a cloned repository
./linux/install.sh

Microsoft Windows (PowerShell)

Launch PowerShell (standard user or administrator) and execute the automated setup script:

# Download and execute the installation script
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/leandre755/gui_agent/7a49514/windows/install.ps1" -OutFile "install.ps1"
powershell -ExecutionPolicy Bypass -File .\install.ps1

# Or execute locally from a cloned repository
.\windows\install.ps1 -Local

2. Isolated Deployment via uv tool

Install gui-agent directly into an isolated environment with global CLI entrypoints:

# Install from PyPI
uv tool install gui-agent

# Or install from GitHub repository
uv tool install "git+https://github.com/leandre755/gui_agent.git"

# Upgrade to latest release
uv tool upgrade gui-agent

3. Linux System Prerequisites

Under Linux, install the native window management, OCR, multimedia, AT-SPI accessibility, and Rust build libraries:

# Debian / Ubuntu / Linux Mint
sudo apt-get update && sudo apt-get install -y \
  xdotool wmctrl spectacle ffmpeg xclip tesseract-ocr libgl1 libatspi-dev cargo rustc

# Fedora / RHEL
sudo dnf install -y \
  xdotool wmctrl spectacle ffmpeg xclip tesseract libglvnd-glx at-spi2-core-devel cargo rust

# Arch Linux / Manjaro
sudo pacman -S --needed \
  xdotool wmctrl spectacle ffmpeg xclip tesseract at-spi2-core cargo rust

MCP Client Configuration

1. Claude Code CLI

Register the server with Claude Code CLI in a single command:

# If installed via uv tool
claude mcp add gui-agent -- gui-agent

# Direct on-the-fly execution via uvx (zero pre-installation)
claude mcp add gui-agent -- uvx --from gui-agent gui-agent

2. Antigravity CLI

Add the server definition to your Antigravity global MCP configuration:

Linux / macOS: ~/.gemini/config/mcp_config.json
Windows: %USERPROFILE%\.gemini\config\mcp_config.json

{
  "mcpServers": {
    "gui-agent": {
      "command": "gui-agent",
      "args": [],
      "env": {
        "DISPLAY": ":0"
      }
    }
  }
}

(Note: The alias binary mcp-gui-server can also be used as the command target).

3. Cursor & VSCode

Add the following entry to your Cursor mcp.json (~/.cursor/mcp.json or .vscode/mcp.json):

{
  "mcpServers": {
    "gui-agent": {
      "command": "uvx",
      "args": ["--from", "gui-agent", "gui-agent"]
    }
  }
}

Toolset & CLI Reference

Target Architecture v1.0 Primitives (13 tools)

`execute_action_batch`

Executes multi-step Python or Bash action blocks directly in host memory with preloaded mcp_core SDK (Open Interpreter paradigm), eliminating network RTT.

Parameters:
- language (str, default "python"): Execution runtime environment ("python" or "bash").
- code (str): Multi-step script containing conditional logic, loops, and rapid polling routines.
- timeout (float, default 30.0): Execution deadline in seconds before terminating the runner process.
Returns: dict containing execution status, captured stdout, stderr, and execution elapsed_seconds.

`process_run`

Executes shell commands in a pseudo-terminal (PTY) session, allowing credential injection to bypass security modals.

Parameters:
- command (str | list[str]): Command line string or argument list to execute.
- background (bool, default False): Spawns detached in background (True) or waits synchronously (False).
- sudo_password (str | None, default None): Password injected into PTY stdin for Polkit/sudo escalation.
Returns: dict containing execution status, exit code returncode, stdout, and stderr.

`process_list`

Inspects the /proc filesystem in read-only mode to probe active desktop processes and hierarchy without mutation.

Parameters: None.
Returns: list[dict] containing active system process entries with pid, name, and status metadata.

`activate_window`

Switches desktop focus directly at the display compositor level using unique Window IDs, avoiding PID collision (active alias: gui_window_focus).

Parameters:
- window_id (str | int): Target compositor Window ID to raise and focus.
Returns: dict containing operation status and confirmed active window identifier.

`get_app_state`

Inspects the accessibility tree via native Rust mediation (gui-agent-atspi) directly in RAM with zero image tokens.

Parameters:
- include_screenshot (bool, default False): Attaches an optional visual framebuffer capture.
Returns: dict containing structured tree nodes, numeric element_index, bounds, states, and snapshot_id.

`perform_action`

Invokes semantic actions directly in target application memory via D-Bus IPC in sub-50ms without pointer motion.

Parameters:
- element_id (str | int): Node identifier or cache index from the active snapshot_id.
- action (str, default "activate"): Semantic action name ("activate", "click", "press").
Returns: dict containing execution status, target identifier, and action verification response.

`set_value`

Mutates text or numerical values directly into component memory variables without emitting physical keystrokes.

Parameters:
- element_id (str | int): Target editable field or widget identifier.
- value (str): Text or numerical value to assign directly into component memory.
Returns: dict containing mutation status, target identifier, and assigned value confirmation.

`find_text`

Discovers on-screen text coordinates via decoupled local OCR engines (RapidOCR / Tesseract; active alias: gui_find_text).

Parameters:
- text (str): Target text string to identify across the desktop screen.
- confidence (float, default 0.85): Minimum detection confidence score (0.0 to 1.0).
Returns: dict containing detected text centroid {"x": int, "y": int}, bounding box, and match confidence.

`screen_capture`

Captures the raw display framebuffer with an optional calibrated Cartesian coordinate grid overlay (active alias: gui_take_screenshot).

Parameters:
- show_grid (bool, default True): Overlays a Cartesian coordinate grid with adaptive contrast labels.
- grid_step (int, default 100): Pixel distance between coordinate grid lines (minimum 20px).
- output_path (str | None, default None): Output destination path with atomic reservation protection.
Returns: dict containing resolved screenshot_path, image dimensions, format, and grid status.

`mouse_click_at`

Injects physical mouse click events directly via low-level input subsystems at exact target coordinates (active alias: gui_mouse_click).

Parameters:
- x (int | float): Absolute X pixel coordinate.
- y (int | float): Absolute Y pixel coordinate.
- button (str, default "left"): Mouse button identifier ("left", "right", "middle").
- double (bool, default False): Dispatches a consecutive double-click sequence when enabled.
Returns: dict confirming click execution status, target coordinates, and dispatched button.

`mouse_drag_smooth`

Dispatches an interpolated continuous mouse trajectory to overcome GUI drag-and-drop breakaway thresholds (active alias: gui_mouse_drag).

Parameters:
- from_x (int | float): Starting horizontal X coordinate.
- from_y (int | float): Starting vertical Y coordinate.
- to_x (int | float): Terminating horizontal X coordinate.
- to_y (int | float): Terminating vertical Y coordinate.
- duration (float, default 0.5): Total animation interpolation duration in seconds.
Returns: dict confirming kinematic drag completion across the spatial trajectory.

`mouse_scroll`

Simulates hardware mouse wheel movements to force dynamic rendering of virtualized lists (active alias: gui_mouse_scroll).

Parameters:
- x (int | float): Horizontal position where the scroll event is injected.
- y (int | float): Vertical position where the scroll event is injected.
- direction (str, default "down"): Scroll direction axis ("up", "down", "left", "right").
- amount (int, default 5): Step count of scroll ticks to dispatch.
Returns: dict confirming scroll action dispatch, coordinate target, and step count.

`key_tap`

Sends hardware-level keyboard keypresses, system hotkeys, and chord sequences directly to focused window (active aliases: gui_keyboard_press, gui_keyboard_type).

Parameters:
- key (str): Key identifier (e.g., "Return", "Escape", "Tab", "space").
- modifiers (list[str] | str | None, default None): Key modifiers (e.g., ["ctrl"], ["alt"], "super").
Returns: dict confirming keystroke injection status and dispatched chord combination.

Display & Cursor Tools (10 tools)

`gui_get_screen_info`

Retrieves display parameters, monitor topologies, active resolution, and session variables.

Parameters: None.
Returns: dict containing resolution, width, height, monitors list, display_env, and failsafe_enabled.

`gui_take_screenshot`

Captures full-screen or cropped images with an optional Cartesian coordinate grid overlay.

Parameters:
- monitor_index (int, default 1): Target monitor index (0 for virtual canvas).
- crop_box (list[int] | None, default None): Sub-region [x, y, width, height].
- apply_grid (bool, default True): Overlays the Cartesian coordinate grid.
- grid_interval (int, default 100): Interval in pixels between grid lines (minimum 20).
- format (str, default "png"): Output image format ("png" or "jpeg").
- quality (int, default 80): Compression quality (1-100) for JPEG output.
- output_path (str | None, default None): Destination file path. Relative paths are resolved to absolute paths and missing parent directories are created. Empty paths and existing directories are rejected. If the path lacks an extension, the extension corresponding to format is automatically appended. Incompatible extensions are rejected. If the target file already exists, atomic reservation with incremental suffixes such as (1) and (2) protects existing files from overwrite. screenshot_path returns the actual resolved absolute path used. If omitted, defaults to a timestamped image in the screenshots directory.
- include_base64 (bool, default False): Returns Base64-encoded string representation.
Returns: dict containing screenshot_path (resolved absolute path), raw_screenshot_path, format, resolution, cropped, grid_applied, grid_interval, renamed_due_to_conflict, message, and base64_data (present when include_base64 is enabled).

`gui_mouse_move`

Smoothly translates the mouse cursor to target coordinates.

Parameters:
- x (float): Target X position.
- y (float): Target Y position.
- duration (float, default 0.2): Movement interpolation duration in seconds.
- normalized (bool, default False): Set to True when using [0, 1000] coordinates.
- monitor_index (int, default 1): Reference monitor for coordinate calculations.

`gui_mouse_click`

Executes single, double, or multi-clicks at specific coordinates.

Parameters:
- x (float): Target X position.
- y (float): Target Y position.
- button (str, default "left"): Mouse button ("left", "right", "middle").
- clicks (int, default 1): Number of clicks to perform.
- normalized (bool, default False): Set to True for [0, 1000] coordinates.
- monitor_index (int, default 1): Reference monitor.

`gui_mouse_drag`

Performs a smooth click-and-drag gesture between two spatial locations.

Parameters:
- x1 (float): Starting X position.
- y1 (float): Starting Y position.
- x2 (float): Ending X position.
- y2 (float): Ending Y position.
- duration (float, default 0.5): Drag animation duration in seconds.
- normalized (bool, default False): Set to True for [0, 1000] coordinates.
- monitor_index (int, default 1): Reference monitor.

`gui_mouse_scroll`

Simulates mouse wheel scrolling along vertical or horizontal axes.

Parameters:
- clicks (int): Number of scroll ticks (positive integer).
- direction (str, default "down"): Direction ("up", "down", "left", "right").

`gui_keyboard_type`

Types text sequentially with natural human-like timing variations.

Parameters:
- text (str): String content to type.
- delay (float, default 0.06): Base delay between keystrokes in seconds.

`gui_keyboard_press`

Simulates individual key presses or complex modifier combinations.

Parameters:
- key (str): Key identifier or chord (e.g., "Return", "Escape", "ctrl+c", "alt+tab", "super").

`gui_clipboard_get`

Reads current textual content from the system clipboard.

Parameters: None.
Returns: dict containing clipboard text, character length, and retrieval method.

`gui_clipboard_set`

Writes string content into the OS clipboard.

Parameters:
- text (str): Text content to store in the clipboard.

Window & Process Control (5 tools)

`gui_window_list`

Enumerates all active desktop windows with metadata.

Parameters: None.
Returns: dict with windows array containing window id, title, pid, and wm_class.

`gui_window_focus`

Activates and brings a specified window to the front.

Parameters:
- window_id (int): Numeric window ID obtained from gui_window_list.

`gui_window_resize_move`

Reposition and resize an application window in a single atomic operation.

Parameters:
- window_id (int): Target numeric window ID.
- x (int): New top-left X coordinate.
- y (int): New top-left Y coordinate.
- width (int): New window width in pixels.
- height (int): New window height in pixels.

`gui_window_close`

Sends an orderly close request to a target window.

Parameters:
- window_id (int): Target numeric window ID.

`gui_app_launch`

Spawns an operating system process or binary.

Parameters:
- command (str): Shell command line or binary path to launch.
- background (bool, default True): Run asynchronously detached (True) or wait synchronously (False).

Vision & OCR Automation (3 tools)

`gui_find_template`

Performs normalized template matching via OpenCV to locate graphical elements.

Parameters:
- template_path (str): File path to the reference template image.
- threshold (float, default 0.8): Confidence threshold (between 0.01 and 1.0).
- monitor_index (int, default 1): Monitor index to inspect.
Returns: dict containing match center coordinates (x, y) and matching confidence.

`gui_find_text`

Extracts text bounding boxes via OCR (Tesseract / RapidOCR) and calculates centroid coordinates.

Parameters:
- text (str): Target string to discover.
- confidence (float, default 0.6): Minimum OCR confidence score (0.0 to 1.0).
- monitor_index (int, default 1): Monitor index to search.
Returns: dict containing text_found, centroid (x, y), confidence, and bounding box [x, y, w, h].

`gui_click_text`

Executes an OCR search and dispatches a mouse click directly to the centroid of the discovered text.

Parameters:
- text (str): Target text string to locate and click.
- button (str, default "left"): Mouse button to click ("left", "right", "middle").
- clicks (int, default 1): Number of clicks to perform.
- monitor_index (int, default 1): Target monitor.

Web & Multimedia Recording (3 tools)

`gui_web_action`

Interacts directly with web pages via headless Chromium powered by Playwright.

Parameters:
- url (str): Web address or local file URL to navigate to.
- action (str, default "aria_tree"): Action to perform ("aria_tree", "click", "type", "screenshot").
- selector (str | None, default None): CSS or XPath selector for click and type actions.
- text (str | None, default None): Text payload to input when action="type".
- viewport_width (int, default 1280): Browser viewport width.
- viewport_height (int, default 720): Browser viewport height.
- timeout_ms (int, default 30000): Navigation and locator timeout in milliseconds.

`gui_start_video_recording`

Launches an asynchronous screen recording sub-process using FFmpeg with minimal CPU overhead.

Parameters:
- output_path (str | None, default None): Destination file path (defaults to timestamped MP4 in videos dir).
- fps (int, default 5): Video capture frame rate (1 to 30 FPS).
- monitor_index (int, default 1): Target monitor index.
- duration (int | None, default None): Optional automatic duration limit in seconds.

`gui_stop_video_recording`

Cleanly terminates the ongoing FFmpeg recording and validates the generated MP4 file container.

Parameters: None.
Returns: dict containing output_path, file_exists, and file_size_bytes.

Environment Variables (Configuration)

Variable	Description	Default Value
`DISPLAY`	Target X11 display server identifier.	`:0`
`GUI_AGENT_SCREENSHOTS_DIR`	Directory where screenshots and cropped frames are saved.	`$XDG_CACHE_HOME/gui-agent/screenshots`
`GUI_AGENT_VIDEOS_DIR`	Directory where continuous MP4 screen video recordings are saved.	`$XDG_CACHE_HOME/gui-agent/videos`
`GUI_AGENT_ATSPI_BIN`	Custom filesystem path to the native `gui-agent-atspi` Rust mediator binary.	Auto-discovered

Clean Uninstallation

To cleanly purge gui-agent, delete isolated environments, and remove registered MCP configurations:

1. Linux (Bash)

# Download and execute the automated uninstaller
curl -fsSLO https://raw.githubusercontent.com/leandre755/gui_agent/7a49514/linux/uninstall.sh
chmod +x uninstall.sh && ./uninstall.sh --purge-data --yes

# Or local uninstall with full data and cache purge
./linux/uninstall.sh --purge-data --yes

2. Microsoft Windows (PowerShell)

# Download and execute the automated uninstaller
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/leandre755/gui_agent/7a49514/windows/uninstall.ps1" -OutFile "uninstall.ps1"
powershell -ExecutionPolicy Bypass -File .\uninstall.ps1 -PurgeData -Yes

# Or local uninstall with full data and cache purge
.\windows\uninstall.ps1 -PurgeData -Yes

What the uninstaller cleans:

Removes gui-agent, mcp-gui-server, and gui-agent-atspi binaries from standard binary paths (~/.local/bin or virtualenv).
Unregisters the MCP server from Claude Code CLI configuration.
Cleans JSON entries from Antigravity mcp_config.json.
Purges temporary runtimes and optionally deletes all screenshots and recordings (--purge-data / -PurgeData).

Development & Quality-Gate

The project enforces strict software engineering standards, verified by an 8-layer pre-commit quality-gate pipeline and full test coverage.

1. Local Environment Setup

# Clone the repository
git clone https://github.com/leandre755/gui_agent.git
cd gui_agent

# Initialize virtual environment with Astral UV
uv venv
source .venv/bin/activate

# Install editable package with development dependencies and build native Rust extensions (requires Cargo)
uv pip install -e ".[dev]"

2. Running Test Suites

# Run unit and integration tests across platform layers
pytest -v linux/tests/

3. Quality-Gate 8-Layer Pre-Commit Verification

Every commit is gated through 8 strict static validation layers to eliminate technical debt and security vulnerabilities:

# Run the 8-layer quality-gate validation hook locally
ALLOW_CONFIG_EDIT=1 ./.githooks/pre-commit

Layer	Validator	Scope & Quality Invariants Enforced
1	`anti-leak`	Blocks secret tokens, private keys, and `.env` credentials from staged files.
2	`pip-audit`	Audits Python dependency tree against known CVE vulnerability databases.
3	`ruff check`	Enforces zero lint warnings, PEP 8 standards, and modern Python 3.10+ idioms.
4	`ruff format`	Verifies deterministic, uniform code formatting across all Python sources.
5	`mypy`	Strict static type checking with zero untyped definitions permitted.
6	`sonar/smells`	Checks cognitive complexity (McCabe C90 <= 25), bug hazards, and simplifications.
7	`bandit`	Static AST security analysis preventing insecure subprocess calls and patterns.
8	`semgrep`	SAST security scanner detecting code injection and system boundary risks.

License

This project is licensed under the terms of the MIT License.

Name		Name	Last commit message	Last commit date
.GCC		.GCC
.agents		.agents
.githooks		.githooks
.github		.github
linux		linux
macos		macos
windows		windows
.coderabbit.yaml		.coderabbit.yaml
.gitignore		.gitignore
AGENTS.md		AGENTS.md
AGENT_POLICY.md		AGENT_POLICY.md
CLAUDE.md		CLAUDE.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
INSTALL.md		INSTALL.md
LICENSE		LICENSE
README.fr.md		README.fr.md
README.md		README.md
ci.sh		ci.sh
hatch_build.py		hatch_build.py
maj.md		maj.md
mcp_final_specification.md		mcp_final_specification.md
note-de-conception-fondements-et-architecture.md		note-de-conception-fondements-et-architecture.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

gui-agent

The Philosophy: Why gui-agent?

Core Capabilities

How It Works

Technical Execution Pipeline & Foundational Pillars

Multi-Platform Root Architecture & Dynamic Path Resolution

Installation

1. Automated Installation (Recommended)

Linux (Bash)

Microsoft Windows (PowerShell)

2. Isolated Deployment via uv tool

3. Linux System Prerequisites

MCP Client Configuration

1. Claude Code CLI

2. Antigravity CLI

3. Cursor & VSCode

Toolset & CLI Reference

execute_action_batch

process_run

process_list

activate_window

get_app_state

perform_action

set_value

find_text

screen_capture

mouse_click_at

mouse_drag_smooth

mouse_scroll

key_tap

gui_get_screen_info

gui_take_screenshot

gui_mouse_move

gui_mouse_click

gui_mouse_drag

gui_mouse_scroll

gui_keyboard_type

gui_keyboard_press

gui_clipboard_get

gui_clipboard_set

gui_window_list

gui_window_focus

gui_window_resize_move

gui_window_close

gui_app_launch

gui_find_template

gui_find_text

gui_click_text

gui_web_action

gui_start_video_recording

gui_stop_video_recording