diff --git a/examples/tutorials/003_OpenVINO_Optimization.ipynb b/examples/tutorials/003_OpenVINO_Optimization.ipynb index 443fcd94..bae987ff 100644 --- a/examples/tutorials/003_OpenVINO_Optimization.ipynb +++ b/examples/tutorials/003_OpenVINO_Optimization.ipynb @@ -2,388 +2,417 @@ "cells": [ { "cell_type": "markdown", - "id": "4d74971e-0379-4237-96f2-d2755405ec96", + "id": "title003", "metadata": {}, "source": [ - "# OpenVINO Physical AI APIs enabling deployment of optimized model" - ] - }, - { - "cell_type": "markdown", - "id": "08eaf7c2-a51a-497b-8f98-41967e972426", - "metadata": {}, - "source": [ - "### Minimal code to run a Pi0.5 model on an SO101 robot" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "id": "40e9e0bb-3b87-42cd-a587-ae6acc72409e", - "metadata": {}, - "source": [ - "" + "# Deploy an OpenVINO Policy on an SO101 Robot\n", + "\n", + "This notebook loads an exported robot policy with the OpenVINO Physical AI runtime, checks that its inputs and outputs match an SO101 deployment, and optionally runs it on real hardware.\n", + "\n", + "Real robot execution is disabled by default. Review the model, calibration, camera names, workspace, and emergency-stop procedure before enabling it." ] }, { "cell_type": "markdown", - "id": "39464140-04c5-4f8a-b386-6a7bb735836b", + "id": "install1", "metadata": {}, "source": [ - "## 1) Train a model using the Physical AI Studio or LeRobot\n", - "* Use Pi0.5 policy supported in Physical AI Studio or LeRobot to train a model.
\n", - "* The output of this stage should be a trained Pi0.5 model
\n", - "[Open Notebook](002_Using_Physical_AI_Studio.ipynb)" + "## 1) Install OpenVINO Physical AI\n", + "\n", + "The runtime APIs used below are newer than the latest PyPI release. Install the current package directly from the Physical AI repository with the SO101 and shared-camera transport extras.\n", + "\n", + "The general notebook environment should already be installed as described in [the tutorials README](README.md)." ] }, { - "cell_type": "markdown", - "id": "a34dcf91-070a-4772-ad8f-dde8c6483c57", + "cell_type": "code", + "execution_count": null, + "id": "install2", "metadata": {}, + "outputs": [], "source": [ - "## 2) Install OpenVINO Physical AI on your target platform" + "%pip install -q \"physicalai[so101,transport] @ git+https://github.com/openvinotoolkit/physicalai.git@main\"" ] }, { "cell_type": "markdown", - "id": "3c28a826-a284-4623-8a7b-74765529bdd6", - "metadata": {}, - "source": [ - "Clone the OpenVINO Physical AI repo and install." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "907c2724-8da0-4745-bea7-2d44337e1645", + "id": "model001", "metadata": {}, - "outputs": [], "source": [ - "from pathlib import Path\n", + "## 2) Load an OpenVINO Policy\n", + "\n", + "The default path downloads the official `OpenVINO/pi05-libero-fp16-ov` package from Hugging Face with `InferenceModel.from_pretrained`. It lets every developer validate model download and OpenVINO loading without preparing a local export.\n", "\n", - "requirements_file = Path(\"requirements.txt\")\n", - "if not requirements_file.exists():\n", - " requirements_file = Path(\"notebooks/requirements.txt\")\n", + "Use one of two loading paths:\n", "\n", - "if not Path(\"physicalai\").exists():\n", - " !git clone https://github.com/openvinotoolkit/physicalai.git\n", + "1. **Hugging Face (default):** keep the official model ID for a loading test, or replace it with your own published OpenVINO policy package.\n", + "2. **Local OpenVINO package:** point to an extracted policy package on disk. For real robot deployment, obtain this package using one of these workflows:\n", + " - **Physical AI Studio export (recommended):** train or import the policy in a current Studio version, export the OpenVINO backend, download the export, and extract it locally. This workflow preserves the manifest and runtime metadata expected by the current Physical AI APIs.\n", + " - **External training and export:** train on another platform, such as a CUDA system, then export a complete OpenVINO policy package yourself. Keep the model IR, manifest, preprocessing/postprocessing assets, normalization statistics, and feature schema together. A standalone `.xml`/`.bin` pair may not contain enough deployment metadata.\n", "\n", - "%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu -r {requirements_file}\n", - "%pip install -q -e physicalai" + "The official Pi0.5 checkpoint is trained for the LIBERO benchmark. Its 8-dimensional state and 7-dimensional action interface are not compatible with a six-motor SO101, so keep `RUN_ON_ROBOT = False` when using this default model.\n", + "\n", + "See the [Physical AI Studio documentation](https://github.com/open-edge-platform/physical-ai-studio/blob/main/application/README.md) for its model and hardware workflows." ] }, { "cell_type": "markdown", - "id": "55c25328-afc8-48e1-9448-db67a492caaf", + "id": "model002", "metadata": {}, "source": [ - "Import libraries, and set up a working directory" + "### Action required: Select and configure a model source\n", + "\n", + "Keep `MODEL_SOURCE = \"huggingface\"` for the default loading validation, or set it to `\"local\"` for an extracted Studio export or an externally exported package.\n", + "\n", + "- For `\"huggingface\"`, keep the official model ID or set `HUGGINGFACE_MODEL_ID` to your published policy package.\n", + "- For `\"local\"`, set `LOCAL_MODEL_PATH` to the root of the extracted OpenVINO policy package.\n", + "- Set `TASK` to the instruction used by a language-conditioned policy, or `None` for a policy that does not use language.\n", + "- Current Studio exports should declare feature metadata. For an older or external package that does not, set `FALLBACK_CAMERA_NAMES` to the exact camera feature names used during data collection and training.\n", + "\n", + "Keep `RUN_ON_ROBOT = False` until the compatibility section confirms that the policy is intended for this SO101 and its six-dimensional state and action interfaces." ] }, { "cell_type": "code", "execution_count": null, - "id": "9cd0bf28-873b-4cce-bb84-21c4ef513ea2", + "id": "model003", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", - "import json\n", - "import os\n", - "import time\n", "\n", - "import numpy as np\n", - "import openvino as ov\n", - "from huggingface_hub import hf_hub_download, snapshot_download\n", - "from physicalai.inference import InferenceModel\n", + "MODEL_SOURCE = \"huggingface\"\n", "\n", - "WORKSPACE = Path.cwd().resolve()\n", - "RUNTIME_ROOT = WORKSPACE / \"physicalai\"\n", - "ROOT = RUNTIME_ROOT\n", - "os.chdir(ROOT)" - ] - }, - { - "cell_type": "markdown", - "id": "65ff5c43-c32a-44ca-ac2a-9e4bee141acc", - "metadata": {}, - "source": [ - "Install extras, mainly for hardware acceleration (e.g. for SO101 or specific camera types)" + "LOCAL_MODEL_PATH = Path(\"physicalai_assets/models/my-so101-openvino-export\")\n", + "\n", + "HUGGINGFACE_MODEL_ID = \"OpenVINO/pi05-libero-fp16-ov\"\n", + "HUGGINGFACE_REVISION = None\n", + "HUGGINGFACE_CACHE_DIR = Path(\"physicalai_assets/models\")\n", + "\n", + "DEVICE = \"AUTO\"\n", + "TASK = \"pick up the box\"\n", + "FALLBACK_CAMERA_NAMES = [\"image\", \"image2\"]\n", + "FPS = 30.0\n", + "DURATION_S = 60.0\n", + "RUN_ON_ROBOT = False" ] }, { "cell_type": "code", "execution_count": null, - "id": "28eab95d-26c2-4e88-a67d-c19b3cc7e9dc", + "id": "model004", "metadata": {}, "outputs": [], "source": [ - "INSTALL_HARDWARE_DEPS = \"True\"\n", - "USE_SHARED_CAMERA = \"True\"\n", + "from physicalai.inference import InferenceModel\n", "\n", - "if INSTALL_HARDWARE_DEPS:\n", - " import subprocess\n", - " import sys\n", + "if MODEL_SOURCE == \"local\":\n", + " model_path = LOCAL_MODEL_PATH.expanduser().resolve()\n", + " if not model_path.is_dir():\n", + " raise FileNotFoundError(\n", + " f\"Local policy directory not found: {model_path}. \"\n", + " \"Update LOCAL_MODEL_PATH or select MODEL_SOURCE='huggingface'.\"\n", + " )\n", + " policy = InferenceModel(\n", + " model_path,\n", + " backend=\"openvino\",\n", + " device=DEVICE,\n", + " )\n", + " print(f\"Loaded local policy: {model_path}\")\n", + "elif MODEL_SOURCE == \"huggingface\":\n", + " policy = InferenceModel.from_pretrained(\n", + " HUGGINGFACE_MODEL_ID,\n", + " revision=HUGGINGFACE_REVISION,\n", + " cache_dir=HUGGINGFACE_CACHE_DIR,\n", + " backend=\"openvino\",\n", + " device=DEVICE,\n", + " )\n", + " print(f\"Loaded Hugging Face policy: {HUGGINGFACE_MODEL_ID}\")\n", + "else:\n", + " raise ValueError(\"MODEL_SOURCE must be 'local' or 'huggingface'.\")\n", "\n", - " hardware_extras = \"transport,so101\" if USE_SHARED_CAMERA else \"so101\"\n", - " runtime_with_hardware_extras = str(WORKSPACE / \"physicalai\") + f\"[{hardware_extras}]\"\n", - " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"-e\", runtime_with_hardware_extras])\n", - " print(f\"[DONE] Installed PhysicalAI hardware extras: {hardware_extras}\")\n", + "print(policy)\n", + "if policy.input_features:\n", + " print(\"Input features:\")\n", + " for feature in policy.input_features:\n", + " print(f\" {feature.name}: {feature.ftype}, shape={feature.shape}, dtype={feature.dtype}\")\n", "else:\n", - " print(\"[SKIP] Hardware extras are not installed. Set INSTALL_HARDWARE_DEPS = True if this environment needs them.\")\n" - ] - }, - { - "cell_type": "markdown", - "id": "2694f43e-7753-4ce5-9124-7faf9bc32490", - "metadata": {}, - "source": [ - "## 3) Discover and connect to Robot and Cameras" - ] - }, - { - "cell_type": "markdown", - "id": "7c427976-07a9-49ac-86da-8b68a2a6dbdd", - "metadata": {}, - "source": [ - "Discover cameras." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fad5138f-f397-4143-8e94-9d44ff7ad1a6", - "metadata": {}, - "outputs": [], - "source": [ - "# Discover available cameras — use the device IDs from this output in the config below\n", - "from physicalai.capture import discover_all\n", - "\n", - "for driver, devices in discover_all().items():\n", - " if not devices:\n", - " continue\n", - " print(f\"\\n[{driver}]\")\n", - " for dev in devices:\n", - " print(f\" {dev.device_id} — {dev.name}\")\n" + " print(\"The package does not declare input feature metadata; configure the documented fallbacks before deployment.\")\n", + "\n", + "if policy.output_features:\n", + " print(\"Output features:\")\n", + " for feature in policy.output_features:\n", + " print(f\" {feature.name}: {feature.ftype}, shape={feature.shape}, dtype={feature.dtype}\")\n", + "else:\n", + " print(\"The package does not declare output feature metadata; confirm its SO101 action schema before deployment.\")" ] }, { - "attachments": {}, "cell_type": "markdown", - "id": "85241b2b-6119-4bba-8986-aa2fa0f57123", + "id": "hardware", "metadata": {}, "source": [ - "To set up cameras
\n", - "* Use the output of previous cell in the next cell
\n", - "* Use the same names of camera as set in the dataset
\n", - "\"Description\"\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "55d09462-1ef0-4d92-a03e-5b2b21684bad", - "metadata": {}, - "outputs": [], - "source": [ - "# Cameras — use the same names and setup as used for the dataset collection\n", - "# ============================================================\n", - "# 🔴 USER PARAMETERS — Edit these before running 🔧\n", - "# ============================================================\n", - "CAMERAS = [\n", - " (, \"uvc\", ),\n", - " (, \"uvc\", ),\n", - "]\n", + "## 3) Configure SO101 Hardware\n", "\n", - "CAMERA_WIDTH = 640\n", - "CAMERA_HEIGHT = 480\n", - "CAMERA_FPS = 30\n", + "Skip this section when `RUN_ON_ROBOT` is `False`.\n", + "\n", + "For a real deployment, keep the model, calibration, and camera configuration from the same data-collection workflow:\n", + "\n", + "1. **Studio-assisted workflow (recommended):** download the OpenVINO model export and SO101 calibration JSON from a current Physical AI Studio setup. During camera selection, use the exact camera names, resolution, and FPS from the Studio environment used to collect the training dataset.\n", + "2. **External/manual workflow:** provide a compatible OpenVINO policy package and SO101 calibration JSON produced outside current Studio, then configure the physical cameras explicitly. The feature names and ordering must still match the training dataset.\n", "\n", - "# Runtime\n", - "FPS = 30\n", - "DURATION_S = 60.0" + "### Linux device permissions\n", + "\n", + "The notebook user needs access to the robot serial port, camera devices, and, when applicable, the OpenVINO GPU device. Check `groups` and device ownership before starting Jupyter. A typical one-time setup is:\n", + "\n", + "```bash\n", + "sudo usermod -aG dialout,video,render \"$USER\"\n", + "```\n", + "\n", + "Log out and back in after changing group membership. Do not run Jupyter as root." ] }, { - "attachments": {}, "cell_type": "markdown", - "id": "19ae4018-7f3d-4c98-917c-843c58074b8a", + "id": "robot001", "metadata": {}, "source": [ - "Define robot and get the robot ID from Physical AI Studio

\n", - "\"Description\"\n", - "\n" + "### Action required: Select the calibration source\n", + "\n", + "Choose one of two calibration paths:\n", + "\n", + "1. **Studio download (recommended):** download the calibration JSON for this SO101 from a current Physical AI Studio robot configuration, place it in a local directory, and set `STUDIO_CALIBRATION_PATH`.\n", + "2. **Custom path:** for an older Studio setup, LeRobot, or another calibration workflow, set `CUSTOM_CALIBRATION_PATH` to its compatible SO101 calibration JSON.\n", + "\n", + "Set `CALIBRATION_SOURCE` to `\"studio_download\"` or `\"custom_path\"`. The notebook does not assume a Studio cache location. The selected file must belong to this physical robot and contain calibration data for all six motors." ] }, { "cell_type": "code", "execution_count": null, - "id": "35aa390b-c7e4-4bc2-ab08-13aecff652d5", + "id": "robot002", "metadata": {}, "outputs": [], "source": [ - "# ============================================================\n", - "# 🔴 USER PARAMETERS — Edit these before running 🔧\n", - "# ============================================================\n", - "# Robot\n", "SO101_PORT = \"/dev/ttyACM0\"\n", "\n", - "# ============================================================\n", - "# 🔴 USER PARAMETERS — Edit these before running 🔧\n", - "# ============================================================\n", - "# Calibration file location depends on how you collected data:\n", - "# Physical AI Studio: ~/.cache/physicalai/robots//calibrations/.json\n", - "# LeRobot: ~/.cache/calibration/so101_follower.json\n", + "CALIBRATION_SOURCE = \"studio_download\"\n", + "STUDIO_CALIBRATION_PATH = Path(\"physicalai_assets/calibrations/so101_calibration.json\")\n", + "CUSTOM_CALIBRATION_PATH = Path(\"/path/to/existing-so101-calibration.json\")\n", "\n", - "#SO101_CALIBRATION = \"~/.cache/physicalai/robots//calibrations/.json\"\n", - "SO101_CALIBRATION = \"/home/intel/.cache/physicalai/robots/7549dd0c-a292-41c4-a8c0-712aae14c55f/calibrations/b28a2ed2-66db-4f61-848e-b65c88827d48.json\"" - ] - }, - { - "cell_type": "markdown", - "id": "6c86a75f-9cd6-46ba-99fb-880521a689c8", - "metadata": {}, - "source": [ - "Connect cameras and robot together. This is an important step as it sets up an environment for robot and camera together.\n", - "* Make sure Physical AI Studio application is closed and no other application is using the cameras or robot" + "if CALIBRATION_SOURCE == \"studio_download\":\n", + " SO101_CALIBRATION = STUDIO_CALIBRATION_PATH.expanduser()\n", + "elif CALIBRATION_SOURCE == \"custom_path\":\n", + " SO101_CALIBRATION = CUSTOM_CALIBRATION_PATH.expanduser()\n", + "else:\n", + " raise ValueError(\"CALIBRATION_SOURCE must be 'studio_download' or 'custom_path'.\")\n", + "\n", + "CAMERA_WIDTH = 640\n", + "CAMERA_HEIGHT = 480\n", + "CAMERA_FPS = 30\n", + "\n", + "CAMERA_SOURCE = \"interactive\"\n", + "MANUAL_CAMERA_CONFIGS = [\n", + " {\n", + " \"name\": \"image\",\n", + " \"camera_type\": \"uvc\",\n", + " \"init_args\": {\"device\": \"/dev/v4l/by-id/usb-example-overhead-video-index0\"},\n", + " },\n", + " {\n", + " \"name\": \"image2\",\n", + " \"camera_type\": \"uvc\",\n", + " \"init_args\": {\"device\": \"/dev/v4l/by-id/usb-example-arm-video-index0\"},\n", + " },\n", + "]" ] }, { "cell_type": "code", "execution_count": null, - "id": "b56dabbf-20f5-4896-894f-7bafc7283625", + "id": "robot003", "metadata": {}, "outputs": [], "source": [ - "from physicalai.capture import SharedCamera\n", "from physicalai.robot import SO101\n", "\n", - "# Cameras\n", - "cameras = {}\n", - "for name, driver, device_id in CAMERAS:\n", - " kwargs = {\"serial_number\": device_id} if driver == \"realsense\" else {\"device\": device_id}\n", - " cam = SharedCamera(driver, **kwargs, width=CAMERA_WIDTH, height=CAMERA_HEIGHT, fps=CAMERA_FPS)\n", - " cam.connect()\n", - " cameras[name] = cam\n", - " print(f\"Camera '{name}' connected: {cam.actual_width}x{cam.actual_height} @ {cam.actual_fps}fps\")\n", - "\n", - "# Robot\n", - "robot = SO101(port=SO101_PORT, calibration=SO101_CALIBRATION, role=\"follower\")\n", - "robot.connect()\n", - "print(f\"Robot connected on {SO101_PORT}\")" - ] - }, - { - "cell_type": "markdown", - "id": "66334671-4691-44d7-a37b-59bc92693798", - "metadata": {}, - "source": [ - "## 4) Load the model" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "4f9581ef-405e-4719-a201-8908a2940414", - "metadata": {}, - "outputs": [], - "source": [ - "# ============================================================\n", - "# 🔴 USER PARAMETERS — Edit these before running 🔧\n", - "# ============================================================\n", - "EXPORT_DIR = \"exports/pi05-pick-place-purple-cube\"\n", - "DEVICE = \"GPU\" # OpenVINO can run on \"GPU\", \"CPU\", \"NPU\"\n", - "TASK = \"pick up the box\"" + "robot = None\n", + "if RUN_ON_ROBOT:\n", + " if not SO101_CALIBRATION.is_file():\n", + " raise FileNotFoundError(f\"Calibration file not found: {SO101_CALIBRATION}\")\n", + " robot = SO101(\n", + " port=SO101_PORT,\n", + " calibration=SO101_CALIBRATION,\n", + " role=\"follower\",\n", + " )\n", + "else:\n", + " print(\"Robot setup skipped. Set RUN_ON_ROBOT = True after validating a compatible policy.\")" ] }, { "cell_type": "markdown", - "id": "d4ed8ff1-4c78-4060-b9d2-6e8fb9e9c891", + "id": "compat01", "metadata": {}, "source": [ - "
Make sure to copy all the model files from the studio to the $EXPORT_DIR\n", - "
Then, load the model to the target device\n", - "

\n", - "\"Description\"" + "## 4) Validate Deployment Compatibility\n", + "\n", + "The runtime maps robot observations and named camera frames to the public feature schema in `policy.input_features`. The model state and action dimensions must match the robot joint count. Multi-camera names must match the suffixes of the model's visual feature names.\n", + "\n", + "This validation uses only public APIs; it does not construct a private runtime input." ] }, { "cell_type": "code", "execution_count": null, - "id": "667d3981-8a63-433e-9e43-f62952331c01", + "id": "compat02", "metadata": {}, "outputs": [], "source": [ - "import openvino_tokenizers \n", - "from physicalai.inference import InferenceModel\n", + "def features_of_type(features, feature_type):\n", + " return [feature for feature in features if feature.ftype == feature_type]\n", "\n", - "policy = InferenceModel.load(EXPORT_DIR, device=DEVICE)\n", - "print(f\"Model loaded on {DEVICE}\")" - ] - }, - { - "cell_type": "markdown", - "id": "3e99fb03-e422-4c37-beb7-6a0d70783606", - "metadata": {}, - "source": [ - "## 5) Run the model (policy) on the robot" + "\n", + "visual_features = features_of_type(policy.input_features, \"VISUAL\")\n", + "state_features = features_of_type(policy.input_features, \"STATE\")\n", + "action_features = features_of_type(policy.output_features, \"ACTION\")\n", + "\n", + "declared_camera_names = [\n", + " feature.name.removeprefix(\"images.\")\n", + " for feature in visual_features\n", + "]\n", + "if declared_camera_names:\n", + " expected_camera_names = declared_camera_names\n", + " camera_name_source = \"model feature metadata\"\n", + "else:\n", + " expected_camera_names = list(FALLBACK_CAMERA_NAMES)\n", + " camera_name_source = \"FALLBACK_CAMERA_NAMES\"\n", + " if not expected_camera_names:\n", + " raise ValueError(\n", + " \"The model has no visual feature metadata. \"\n", + " \"Set FALLBACK_CAMERA_NAMES to the camera names used during training.\"\n", + " )\n", + "\n", + "if len(set(expected_camera_names)) != len(expected_camera_names):\n", + " raise ValueError(f\"Camera names must be unique: {expected_camera_names}\")\n", + "\n", + "state_dims = sorted({feature.shape[-1] for feature in state_features if feature.shape})\n", + "action_dims = sorted({feature.shape[-1] for feature in action_features if feature.shape})\n", + "\n", + "print(f\"Expected cameras ({camera_name_source}): {expected_camera_names}\")\n", + "print(f\"Expected state dimensions: {state_dims or 'none declared'}\")\n", + "print(f\"Expected action dimensions: {action_dims or 'none declared'}\")\n", + "\n", + "if RUN_ON_ROBOT:\n", + " robot_dim = len(robot.joint_names)\n", + " if state_dims and state_dims != [robot_dim]:\n", + " raise ValueError(f\"Model state dimensions {state_dims} do not match SO101 joint count {robot_dim}.\")\n", + " if action_dims and action_dims != [robot_dim]:\n", + " raise ValueError(f\"Model action dimensions {action_dims} do not match SO101 joint count {robot_dim}.\")\n", + " print(f\"SO101 state/action dimensions match its {robot_dim} joints.\")" ] }, { "cell_type": "markdown", - "id": "13d14a2f-fe48-4c08-9846-775278cdb7e8", + "id": "camera01", "metadata": {}, "source": [ - "Use OpenVINO Physical AI to perform Inference." + "### Action required: Configure cameras\n", + "\n", + "Choose one of two camera paths:\n", + "\n", + "1. **Interactive selection (recommended):** keep `CAMERA_SOURCE = \"interactive\"`. The public `select_cameras_interactive` API discovers connected cameras. Select one index at a time and assign the exact feature names printed by the compatibility cell. For a Studio-trained policy, use the names from the Studio environment used during data collection.\n", + "2. **Manual configuration:** set `CAMERA_SOURCE = \"manual\"` and update `MANUAL_CAMERA_CONFIGS`. This is useful for an external setup or unattended reruns. Prefer stable Linux device IDs such as `/dev/v4l/by-id/...-video-index0` instead of `/dev/videoN`.\n", + "\n", + "Both paths create shared cameras through public Physical AI APIs. They are not connected yet; the `RobotRuntime` context manager connects and disconnects them together with the robot." ] }, { "cell_type": "code", "execution_count": null, - "id": "0d363fee-8116-4662-86fa-a074ac6ecb63", + "id": "camera02", "metadata": {}, "outputs": [], "source": [ - "from physicalai.runtime import ActionQueue, PolicyRuntime, SyncExecution\n", - "\n", - "runtime = PolicyRuntime(\n", - " robot=robot,\n", - " model=policy,\n", - " execution=SyncExecution(fps=FPS, request_threshold=0.5),\n", - " fps=FPS,\n", - " cameras=cameras,\n", - " action_queue=ActionQueue(),\n", - " task=TASK,\n", - ")\n", - "\n", - "with runtime:\n", - " print(f\"Running policy at {FPS} fps for {DURATION_S}s — task: {TASK!r}\")\n", - " stats = runtime.run(duration_s=DURATION_S)\n", - "\n", - "print(f\"\\nDone — {stats.steps} steps, {stats.inference_count} inferences, {stats.total_holds} holds\")" + "from physicalai.capture import create_camera, select_cameras_interactive\n", + "\n", + "cameras = {}\n", + "if RUN_ON_ROBOT:\n", + " if CAMERA_SOURCE == \"interactive\":\n", + " cameras = select_cameras_interactive(\n", + " width=CAMERA_WIDTH,\n", + " height=CAMERA_HEIGHT,\n", + " fps=CAMERA_FPS,\n", + " )\n", + " elif CAMERA_SOURCE == \"manual\":\n", + " for config in MANUAL_CAMERA_CONFIGS:\n", + " name = config[\"name\"]\n", + " if name in cameras:\n", + " raise ValueError(f\"Duplicate manual camera name: {name}\")\n", + " cameras[name] = create_camera(\n", + " config[\"camera_type\"],\n", + " shared=True,\n", + " width=CAMERA_WIDTH,\n", + " height=CAMERA_HEIGHT,\n", + " fps=CAMERA_FPS,\n", + " **config[\"init_args\"],\n", + " )\n", + " else:\n", + " raise ValueError(\"CAMERA_SOURCE must be 'interactive' or 'manual'.\")\n", + "\n", + " if len(cameras) != len(expected_camera_names):\n", + " raise ValueError(\n", + " f\"Selected {len(cameras)} camera(s), but {camera_name_source} \"\n", + " f\"expects {len(expected_camera_names)}: {expected_camera_names}.\"\n", + " )\n", + " if set(cameras) != set(expected_camera_names):\n", + " raise ValueError(\n", + " f\"Camera names {sorted(cameras)} do not match the expected names \"\n", + " f\"{sorted(expected_camera_names)} from {camera_name_source}.\"\n", + " )\n", + "else:\n", + " print(\"Camera selection skipped.\")" ] }, { "cell_type": "markdown", - "id": "c2cb63f2-e963-484d-aa8c-57fbf499aa1c", + "id": "runtime1", "metadata": {}, "source": [ - "## 6) Disconnect" + "## 5) Run the Policy on the Robot\n", + "\n", + "Clear the robot workspace and make the emergency stop accessible before running this cell. The `RobotRuntime` context manager owns hardware connection and cleanup, including cleanup after an exception or keyboard interrupt." ] }, { "cell_type": "code", "execution_count": null, - "id": "bae9cb87-132b-4a12-a412-80bf3d4e5760", + "id": "runtime2", "metadata": {}, "outputs": [], "source": [ - "for name, cam in cameras.items():\n", - " cam.disconnect()\n", - " print(f\"Camera '{name}' disconnected\")\n", + "from physicalai.runtime import PolicySource, RobotRuntime, SyncExecution\n", + "\n", + "if RUN_ON_ROBOT:\n", + " execution = SyncExecution(request_threshold=0.5)\n", + " policy_source = PolicySource(\n", + " model=policy,\n", + " execution=execution,\n", + " task=TASK,\n", + " )\n", + " runtime = RobotRuntime(\n", + " robot=robot,\n", + " action_source=policy_source,\n", + " fps=FPS,\n", + " cameras=cameras,\n", + " )\n", "\n", - "robot.disconnect()\n", - "print(\"Robot disconnected\")" + " with runtime:\n", + " steps = runtime.run(duration_s=DURATION_S)\n", + "\n", + " print(f\"Completed steps: {steps}\")\n", + " print(f\"Inference requests after warmup: {execution.inference_count}\")\n", + " print(f\"Action queue holds: {policy_source.action_queue.total_holds}\")\n", + "else:\n", + " print(\"Deployment skipped. Set RUN_ON_ROBOT = True only with a compatible SO101 policy.\")" ] } ],