Disclaimer: This library is currently in early development and is intended for testing and experimentation purposes only. It is not recommended for use in production environments. Please use at your own risk. Feedback and bug reports are highly appreciated as we continue to improve the library. Thank you for your understanding.
⚠️
The "RobotFramework-ImageDetection" library is a powerful tool designed to harness the capabilities of machine learning to train and detect photos effectively within the Robot Framework automation framework. With its seamless integration, this library enables users to build intelligent and sophisticated automation solutions by incorporating computer vision and image recognition techniques.
Before installing "RobotFramework-ImageDetection," please ensure you have Python 3.10 or higher installed on your system.
Setting up a Virtual Environment (Recommended)
python -m venv venv
source venv/bin/activate # On Windows, use: venv\Scripts\activateTo install the library, simply use pip:
pip install robotframework-imagedetectionTraining and detection automatically run on an NVIDIA GPU if one is available, falling back to CPU otherwise — no code or keyword changes needed. GPU acceleration is optional, not required: both training and detection work correctly on CPU too. The difference is speed, and it depends on what you're doing — training on CPU for anything beyond a tiny dataset will be noticeably slower, while running detection (Detect From Path, Detect From Webcam) on CPU with an already-trained model is generally fine for most single-photo use cases. A model can also be trained on one machine and deployed to run detection on a different, CPU-only machine — see CLAUDE.md for details on that workflow.
This library uses PyTorch, so to get GPU acceleration you need to install a CUDA-enabled PyTorch build instead of the default CPU-only one from PyPI. Follow the selector at pytorch.org/get-started/locally to get the exact command for your CUDA version — with uv:
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130(or plain pip install torch torchvision --index-url ... if not using uv). Use whichever cuXXX tag the selector on pytorch.org recommends for your GPU/driver — newer NVIDIA GPUs (e.g. the RTX 50 series / Blackwell architecture) need CUDA 12.8 or newer (cu128 or later); older tags like cu121 predate that hardware and won't work on it.
You can verify it worked with uv run python -c "import torch; print(torch.cuda.is_available())", or by running atest/smoke_test.robot (see atest/) and checking its "Report Device Info" test case.
Before training the image detection model with the "RobotFramework-ImageDetection" library, you need to collect and organize your data/photos in a specific way. Follow these steps to set up your data:
💡 For your convenience, we have provided a sample dataset in another repository, which you can clone and use for testing purposes. You can find the sample dataset at Data-Example. Clone the repository and follow the same folder structure to get started with the "RobotFramework-ImageDetection" library.
-
Create the Main Dataset Folder: Start by creating a new folder in your project called "Data." This folder will serve as the main directory for your training and validation data.
-
Create Training and Validation Subfolders: Inside the "Data" folder, create two subfolders: "training" and "validation." These folders will contain your training and validation datasets, respectively.
-
Organize Photos by Class: Within the "training" and "validation" folders, organize your photos into subfolders based on their classes or categories. For example, if you are classifying images into "Right," "Left," and "Straight," create subfolders named "Right," "Left," and "Straight" inside both the "training" and "validation" folders.
-
Split Data: Ensure that each class's photos are distributed appropriately between the "training" and "validation" folders. The "training" folder should contain photos of each class. Similarly, the "validation" folder should also contain photos of each class, but make sure to use different images than those present in the "training" folder. This separation allows for proper evaluation and testing of the image detection model.
Your data structure should look like this:
Data/
├── training/
│ ├── Right/
│ ├── Left/
│ └── Straight/
├── validation/
│ ├── Right/
│ ├── Left/
│ └── Straight/
Following this organized structure will enable smooth data loading and training using the "RobotFramework-ImageDetection" library. Remember to provide a sufficient amount of diverse and representative photos for accurate model training.
Manually cropping photos in an external editor and sorting the results into the right folders by hand is tedious and error-prone. The library ships an interactive annotation helper for exactly this: it shows you a photo (or a live camera feed), you drag a box around just the part you want the model to learn, and it saves that crop straight into the correct training/<class>/ or validation/<class>/ folder for you — no manual file management, no coordinate math.
# Annotate photos you've already taken:
python -m Imagedetection.annotate --input-dir path/to/photos --class Celsius --out Data
# Or annotate live from a webcam:
python -m Imagedetection.annotate --camera 0 --class Celsius --out DataEach saved crop is automatically split between training/ and validation/ (80/20 by default, --val-ratio to change it) so you never accidentally put the same photo in both — which would make your validation accuracy meaningless, since the model would just be "tested" on something it already memorized. Run it once per class, dragging a box around that class's subject each time, and your Data/ folder ends up correctly organized without any manual sorting.
A magnifier follows your cursor while you draw (switching to show the whole box plus margin once you've started one), so precise placement is possible even for small subjects. Full shortcut list and options: python -m Imagedetection.annotate --help.
If you're capturing live via --camera, the crop region you draw is also saved and automatically embedded into the model file the next time you run Train Model — see "Detecting from a live camera" below for why that matters.
Once you have organized your data as described above, you can use the library to train your image detection model and perform real-time detection within your Robot Framework automation projects.
Now create a new Test case using Robot framework to train a new Model:
*** Settings ***
Library Imagedetection
*** Variables ***
${training_folder} ${CURDIR}\\Data\\training
${validation_folder} ${CURDIR}\\Data\\validation
*** Test Cases ***
Training a New Model
Train Model ${training_folder} ${validation_folder}In this example, the Robot Framework test case named "Training a New Model" starts by importing the "Imagedetection" library under setting section.
The Variables section sets up two variables:
${training_}: Specifies the directory path containing the training data for the model. It points to the "training" folder inside the "Data" directory. ${validati_}: Specifies the directory path containing the validation data for the model. It points to the "validation" folder inside the "Data" directory. The actual test case named "Training a New Model" calls the "Train Model" keyword from the "Imagedetection" library. This keyword is designed to train an image detection model using the specified training and validation data directories.
Training a model can take some time, depends of course on the size of your dataset. Please be patient 🙂
Once you have set up your test cases like this example, you can run your Robot Framework tests to train and utilize your image detection model effectively.
after the test is finished, it will automaticlly generate a new file called model_<timestamp>.pt This file will contain the trained model, which can be used for image detection in your subsequent Robot Framework test cases. This model file is crucial, as it encapsulates the learned patterns and features from the training data, allowing it to accurately detect objects or patterns in new images. Once the file is generated, you can load and utilize the trained model to perform image detection tasks with ease.
Now, the easy part is to use another keyword Detect From Path to make predictions using the newly trained model. This keyword takes two arguments: model_name, which is the path to the generated model, and photo_path, which indicates the path to a test photo that will be used to check if our model can detect the object or not. By providing these two arguments, you can easily evaluate the performance of your trained model on new images and test its detection capabilities.
*** Settings ***
Library Imagedetection
*** Variables ***
${training_folder} ${CURDIR}\\Data\\training
${validation_folder} ${CURDIR}\\Data\\validation
${model_name} ${CURDIR}\\model_XXXXX.pt
*** Test Cases ***
Check the test photo
Detect From Path ${model_name} ${CURDIR}\\test\\Left_test2png.png${model_name}: Specifies the path to the generated model file (model_XXXXX.pt) that was trained using the specified training and validation data directories.
This test case "Check the test photo" calls the "Detect From Path" keyword to perform image detection on a single test photo using the trained model specified.
For hardware-in-the-loop testing (e.g. reading a symbol off a physical device's screen via USB camera), use Detect From Webcam instead of Detect From Path. It captures a frame from a camera, crops it down to just the region you care about, and classifies that crop — the rest of the frame is ignored entirely, since this library is a whole-image classifier, not an object detector, and has no way to locate a subject within a larger scene on its own.
*** Test Cases ***
Check The Live Reading
${prediction} Detect From Webcam ${model_name}
Log Detected: ${prediction}The crop region (x, y, width, height) doesn't need to be passed explicitly — if you built your dataset with annotate.py --camera (see above), the region you drew is already embedded in the trained model file, and Detect From Webcam reads it from there automatically. Pass any of the four arguments explicitly to override that stored region, e.g. while calibrating a new one. If your machine has more than one camera, pass camera_index to pick which one OpenCV should open.
Capture And Predict is the more general keyword Detect From Webcam is built on, for cases where you want to control the saved photo's path directly.
Get Device Info is a small diagnostic keyword worth running first in any suite — it logs (and returns) whether training/detection will run on cuda or cpu, plus PyTorch/CUDA version info, so a misconfigured GPU install shows up immediately instead of just being slower without explanation.
Congratulations! You've reached the end of the README for the "RobotFramework-ImageDetection" library. We hope this documentation has provided you with a clear understanding of how to use our library for image detection in your Robot Framework projects.
In this README, we covered the following topics:
- Introduction to the library and its features
- How to collect and organize your dataset, including the
annotate.pyhelper for building it without manual cropping/sorting - Training a new model and generating the
model_<timestamp>.ptfile - Using the trained model for image detection with
Detect From Path - Detecting from a live camera with
Detect From Webcam, and the diagnosticGet Device Infokeyword
Our library aims to simplify image detection tasks and empower you to build robust and accurate image detection systems within the Robot Framework.
If you encounter any issues, have questions, or want to contribute to the project, feel free to visit our GitHub repository. We value your feedback and are excited to grow the library together with the community.
Thank you for choosing "RobotFramework-ImageDetection" for your image detection needs. Happy testing and happy detecting!😄