A system substrate for building embodied intelligence across heterogeneous robots.
robonix.ai · Documentation · Package catalog · Quick start
Robonix is a general-purpose agentic operating system designed for embodied artificial intelligence. It employs large language models (LLMs) and vision-language models (VLMs) to convert natural-language tasks into Robot Task Description Language (RTDL) programs, which are then executed to complete the specified tasks. The primary advantages of Robonix include the following:
- (1) Hardware–software decoupling. Robonix utilizes layered capability abstractions (Task, Skill, Service, Primitive) to separate software functionality from robot-specific hardware. These abstractions establish standard interface definition language (IDL) interfaces, enabling the reuse of compatible primitives, services, and skills across different robotic platforms.
- (2) Adaptability to diverse user tasks, robot capabilities, and environments. Robonix leverages LLMs and VLMs to generate RTDL programs tailored to each user's task, the robot's capabilities, registered skills, environmental conditions, and body state. This approach enables Robonix to accommodate a wide range of natural language tasks, robotic hardware, and operational environments.
- (3) Observable and verifiable execution. Robonix explicitly represents the model's program in RTDL, allowing for inspection of its structure. The system performs program validation prior to execution and can verify specified outcomes upon completion.
Figure 1. The architecture of Robonix.
Figure 1 illustrates the overall architecture of Robonix. The user interacts with Liaison, which verifies the user with Keystone. Liaison delegates the task to Pilot. The model orchestrates skills and services for the task by writing an RTDL program; Pilot checks it and forwards it to Executor. Sentinel monitors Executor, while Scene keeps the semantic map (objects and their relations) and the spatial map of the outer world. Custom services sit beside Scene: navigation, SLAM mapping, object detection, grasp-pose estimation, motion planning, memory, and more from the package catalog. Atlas, Nexus, Chronos, Scribe, and Vitals provide the capability catalog, messaging, time, logging, and health that the rest build on. At the bottom, Soma is the body abstraction. A URDF gives the body's geometry and coordinate frames; a YAML body description gives its component tree (base, arm, gripper, camera) and the primitive capabilities each component provides. On the right, the skill library holds skills such as picking, handing over an item, pouring, and folding, often vision-language-action (VLA) models; skills can be learned in a simulator such as Webots, MuJoCo, or Isaac Sim and then deployed to real robots through Robonix. Where it meets Soma sit the preset skills the vendor ships with the body, such as a dance or a backflip.
| Abstraction | Definition |
|---|---|
| primitive | A single hardware function — sensing or actuation — behind a software interface. |
| service | Reusable software that implements a Robonix interface, such as planning, execution, interaction and scene management. |
| skill | A learned model or an algorithmic procedure, packaged as a reusable executable unit. |
| task | A goal, its constraints, and the conditions that count as completion. |
Each primitive, service, and skill specifies its offerings in a versioned contract and registers with Atlas at startup. Callers bind to the contract rather than the underlying implementation, enabling task transfer to another body and allowing implementations to be replaced without modifying callers.
The System Architecture section lists every component.
Robonix runs on the robot. It can be installed and managed through Robonix Console, a web console on the robot, or built from source and run with the rbnx command-line interface (CLI). Tasks are given through Robonix Client on your computer or Android phone.
The steps below run the TIAGo robot in the Webots simulator and give it a first task from your own computer or phone. The simulator runs on a Linux computer that plays the role of the robot: Ubuntu 22.04 (x86_64 or aarch64), with Docker installed. The user who runs the installer must be able to run docker without sudo (for example, as a member of the docker group), because the simulator is started as that user. Below, ROBOT_IP stands for an address at which your computer or phone can reach the robot computer, such as a LAN or VPN address; once Robonix Console is installed, its Overview page lists the robot's addresses under Robot Host.
On the robot computer, in a terminal, as a regular user:
curl -fsSL https://hub.robonix.ai/install | bashThe installer installs Robonix and Robonix Console. It asks for the model that Pilot, the Robonix planner, uses to plan tasks: an OpenAI-compatible endpoint URL, an API key, and a model name.
-
In a browser on the robot computer, open
http://127.0.0.1:4880and sign in with a Robonix Hub account. From another computer, first runssh -N -L 4880:127.0.0.1:4880 USER@ROBOT_IPthere. -
On the Overview page of Robonix Console, click Choose a robot configuration, select PAL Robotics TIAGo, and click Create deployment.
-
On the same page, click Boot hooks, enter the two commands below, and click Save:
Field Command Before boot bash sim/start.sh --detachAfter exit bash sim/stop.sh -
On the same page, click Allow other machines and confirm, so that Robonix Client can connect from your computer or phone. Use this only on a trusted network.
-
Click Start. The first start builds the simulator and takes several minutes; View run logs shows the progress.
Robonix Client is the operator's app: it gives the robot tasks by text or voice, shows the plan and each capability call as the robot carries it out, and displays the robot's camera view, health, and logs. It runs on a computer or a phone that can reach the robot at ROBOT_IP.
To give tasks from a computer, first install uv, which also provides the Python that Robonix Client needs:
# Linux or macOS
curl -LsSf https://astral.sh/uv/install.sh | sh# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"Open a new terminal, then install Robonix Client and start it with the robot's address:
uv tool install --python 3.11 git+https://github.com/syswonder/robonix-client.git
robonix-client --robot-host ROBOT_IPOpen http://127.0.0.1:7860 on that computer.
To give tasks from a phone, Robonix Client is currently available for Android 8.0 or later. Install the APK from the latest release, enter ROBOT_IP as Robot Host in Settings, and tap Save and connect.
Once the client reaches the robot, it shows Connected. Send Turn left 90 degrees, take a photo, and describe what you see. TIAGo turns in the simulator, the reply describes what the camera sees, and the RTDL tree shows the navigate and snapshot calls.
- A real robot: run the installer on the robot and install its robot configuration from the catalog instead of TIAGo.
- Change the planning model: the model entered during installation is kept on the Settings page of Robonix Console under Robot variables, as VLM endpoint URL, VLM key, and VLM model. Change it there and restart Robonix.
Install these from their own documentation first — Robonix does not provide or install them for you.
| Tool | Why it is needed | Install |
|---|---|---|
| Rust (stable) | Most components are Rust; make install builds them with cargo |
rustup.rs |
| uv | Resolves and runs the Python workspace — services and primitives | docs.astral.sh/uv |
| Docker | Runs the Webots simulator stack, and any capability provider you choose to containerise | docs.docker.com |
| NVIDIA Container Toolkit | Only on a machine with an NVIDIA GPU: Scene then runs its detector in a container with --gpus all, which fails without it |
docs.nvidia.com |
| System packages | A C toolchain, pkg-config and OpenSSL headers for the Rust build, and ALSA's arecord/aplay for the audio driver |
sudo apt install build-essential pkg-config libssl-dev git curl alsa-utils |
Rust and uv install into your home directory, so put them on PATH before continuing:
export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"git clone --recursive https://github.com/syswonder/robonix.git
cd robonix
make installThis builds the Rust components and the rbnx command-line interface (CLI) into ~/.cargo/bin. See Host Platforms for what is regularly tested.
Start the Webots simulator in one terminal:
export DISPLAY=:0
bash examples/webots/sim/start.shBoot Robonix in a second terminal with any OpenAI-compatible VLM endpoint:
export RMW_IMPLEMENTATION=rmw_zenoh_cpp
export VLM_BASE_URL=https://api.openai.com/v1
export VLM_API_KEY=sk-...
export VLM_MODEL=your-model-name
cd examples/webots
rbnx build
rbnx bootThen run rbnx chat in a third terminal. Try go to room 101, what can you see?, or explore the office. See the Getting Started guide for the complete walkthrough.
The Robonix package template contains a mock primitive, a service, and a skill that boot without robot hardware:
git clone https://github.com/syswonder/template-rbnx.git
cd template-rbnx
cp .env.example .env
# Fill in the three VLM values in .env.
set -a; source .env; set +a
rbnx build
rbnx bootRun rbnx caps to inspect the live providers, then try rbnx chat and ask the robot to say hello. Each example package keeps its manifest, config.spec, build/start scripts, implementation, and optional capability definitions in one directory. Start there, then follow the package integration guide to publish a reusable package.
Robot bodies published to the catalog, with more on the way: wheeled, tracked, and quadruped bases, fixed and dual arms, standalone dexterous hands, a humanoid, and four simulated bodies. They span several vendors' chassis software development kits (SDKs), both versions of the Robot Operating System (ROS 1 and ROS 2), and both grippers and five-finger hands, while running the same components, capability contracts, and skills.
Per-robot hardware and links
| Robot | Hardware | Maintainer | Links |
|---|---|---|---|
| AgileX Ranger Mini v3 | Ranger Mini v3 chassis; Livox MID-360 lidar and IMU; RealSense D435i RGB-D; optional Piper arm; audio | syswonder | repo · catalog |
| DEEP Robotics Lite3 | Lite3 quadruped; Livox MID-360 lidar and IMU; Orbbec Gemini 330 RGB-D | Bunnycxk | repo · catalog |
| DEEP Robotics Lynx S10 | Lynx S10 wheeled-quadruped over UDP; Orbbec Gemini 336L RGB-D; InternVLA navigation | 1mujue | repo · catalog |
| Unitree Go2 | Go2 quadruped; onboard lidar, camera, IMU; audio bridge | Origamii520 | repo · catalog |
| Yobotics Y20W | Y20W chassis and posture; Livox MID-360 lidar; RealSense D435i RGB-D; speech | chenx1118 | repo · catalog |
| MirrorMe BPX | BPX quadruped; odometry; guarded stand and sit posture control | nonkr | repo · catalog |
| WHEELTEC R550 | R550 tracked chassis and IMU; LSLIDAR N10P; Orbbec Astra S RGB-D | sherry-part | repo · catalog |
| Yahboom ROSMASTER X3 | Mecanum chassis on Jetson TX2 NX; RPLidar; guarded ROS 1 navigation | luoyg0831-a11y | repo · catalog |
| Hantewin Benben | Benben chassis; Livox MID-360 and LSLIDAR LakiBeam1; RealSense camera; audio | Futaba19-c | repo · catalog |
| BeingBeyond D1 | Fixed-base 6-DOF arm; 2-DOF head; five-finger hand; head RGB-D; pick, place, stack, sort | Ciliphen | repo · catalog |
| AgileX Dual Piper | Two Piper arms and CAN grippers on separate buses; per-arm joint telemetry | syswonder | repo · catalog |
| WowRobo Roboarm | Five-axis LeRobot Koch arm; Orbbec Gemini 215 RGB-D; audio | gaoyz1235 | repo · catalog |
| AUBO i5H | i5H six-axis collaborative arm over the official AUBO SDK; wave skill | gaoyz1235 | repo · catalog |
| LinkerHand O6 | Standalone six-axis five-finger hand over CAN; gesture and finger-motion skills | Ciliphen | repo · catalog |
| Webots TIAGo Lite (sim) | Simulated differential-drive base; head RGB-D; Hokuyo lidar; audio | syswonder | repo · catalog |
| Minecraft Bot (sim) | Minecraft player body; camera, chassis, world state, inventory, navigation | ZZJJWarth | repo · catalog |
| Ranger + Piper (sim) | MuJoCo Ranger Mini v3 and Piper; web or native viewer; mapping, navigation, manipulation | syswonder | repo · catalog |
| Unitree Go2 (sim) | MuJoCo Go2 quadruped; web or native viewer; exploration and semantic navigation | syswonder | repo · catalog |
Each deployment links the complete robot manifest and its primitive, service, and skill dependencies. Published deployment metadata does not replace the hardware-specific safety, commissioning, and acceptance gates documented by each repository. See the robot catalog for published integrations.
Every robot above is assembled from packages, not written as one program. A package declares the capabilities it provides against the shared contracts, so what it offers does not depend on which body it was written for, and a deployment can swap one implementation for another without the layers above noticing.
Browse them in the package catalog, or publish your own with the package integration guide.
The Rust components and the Python packages are written to stay portable across architectures; x86-64, arm64 (NVIDIA Jetson), and LoongArch64 are tested.
| Arch | OS / Distribution | Status |
|---|---|---|
| x86_64 | Ubuntu 22.04 | ✅ Tested |
| x86_64 | Debian 13 | ✅ Tested |
| arm64 | NVIDIA Jetson — JetPack 6.2 (L4T 36.4.3, Ubuntu 22.04) | ✅ Tested |
| LoongArch64 | Loongson 3A6000 + AMD Radeon 7900 XTX — Loong ArchLinux 2026.08.07 | ✅ Tested |
| x86_64 / arm64 | Ubuntu 24.04 and newer | 🚧 Planned |
Note (LoongArch64): Robonix itself runs on the Loongson 3A6000 host, while the simulation platform (Webots) runs on a separate x86_64 machine with Ubuntu 22.04. The two machines are connected over Ethernet on the same local area network (LAN).
"Tested" means the full Robonix pipeline runs end-to-end on that platform — in simulation or on a real robot: voice & interaction, task execution, body movement, scene & mapping (semantic map + spatial map), navigation, and skill execution. Other Linux distributions will likely work but are not regularly verified.
Capability providers that use ROS 2 are built and tested against ROS 2 Humble.
Robonix ships the components below. Deployments add custom services from the package catalog, such as navigation, SLAM mapping, object detection, grasp-pose estimation, motion planning with MoveIt, and memory.
| Component | Responsibility |
|---|---|
| liaison | User input — text and voice |
| keystone | Identity, configuration, and access policy |
| pilot | Turns a task into an RTDL program with the model, and validates it |
| executor | Runs the program, dispatching each call to its provider |
| sentinel | Decides whether a capability call is allowed |
| scene | The environment: a semantic map of objects and their relations, and a spatial map |
| atlas | Capability catalog — every running provider and its contract |
| nexus | Communication over gRPC, the Model Context Protocol (MCP), and ROS 2 |
| chronos | One time source across sensors, actuators, and components |
| scribe | Structured logging |
| vitals | Onboard health: temperatures, voltage, joint motors |
| soma | Body abstraction — URDF for geometry and frames, a YAML component tree with each component's primitive capabilities, and the body's runtime state |
Preset skills, such as a dance or a backflip, ship with a robot body. The skill library holds the rest, such as picking, pouring, and folding. Soma launches deployed skills, Pilot discovers them through Atlas, and Executor invokes them.
Every contract lives in capabilities/, whether a service or a package implements it. Primitives and skills live in their own repositories. Currently, all 12 components are under the system directory for historical reasons.
Full documentation lives at book.robonix.ai.
Getting started
- Quickstart — the full version of the Webots walkthrough in this README
- Host Platforms — what is tested, and what is not
Understanding the system
- Architecture overview — the control plane, and one full request end to end
- Namespaces & contracts — how
robonix/primitive/*,robonix/service/*,robonix/skill/*, androbonix/system/*relate - Interface catalog — every primitive and service contract, generated from
capabilities/
Building on it
- Package integration guide — write a package and publish it to the catalog
- Package catalog — every published package, browsable by kind
- Robot catalog — every published deployment, with its full dependency tree
Release history lives in CHANGELOG.md, formatted per Keep a Changelog. Contributors add entries under ## [Unreleased]; they are moved into a versioned section at release time.
See CONTRIBUTING.md for the repository's license headers, code style, validation commands, commit format, human-authorship policy, and AI assistance disclosure rules.
Mulan Permissive Software License, Version 2 (MulanPSL-2.0). See LICENSE.