You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Kinematic structures are very common in the real world. They range from simple articulated objects to complex mechanical systems. However, despite their relevance, most model-based 3D tracking methods only consider rigid objects. To overcome this limitation, we propose a flexible framework that allows the extension of existing 6DoF algorithms to kinematic structures. Our approach focuses on methods that employ Newton-like optimization techniques, which are widely used in object tracking. The framework considers both tree-like and closed kinematic structures and allows a flexible configuration of joints and constraints. To project equations from individual rigid bodies to a multi-body system, Jacobians are used. For closed kinematic chains, a novel formulation that features Lagrange multipliers is developed. In a detailed mathematical proof, we show that our constraint formulation leads to an exact kinematic solution and converges in a single iteration. Based on the proposed framework, we extend ICG, which is a state-of-the-art rigid object tracking algorithm, to multi-body tracking. For the evaluation, we create a highly-realistic synthetic dataset that features a large number of sequences and various robots. Based on this dataset, we conduct a wide variety of experiments that demonstrate the excellent performance of the developed framework and our multi-body tracker.
11
+
Kinematic structures are very common in the real world.
12
+
They range from simple articulated objects to complex mechanical systems.
13
+
However, despite their relevance, most model-based 3D tracking methods only consider rigid objects.
14
+
To overcome this limitation, we propose a flexible framework that allows the extension of existing 6DoF algorithms to kinematic structures.
15
+
Our approach focuses on methods that employ Newton-like optimization techniques, which are widely used in object tracking.
16
+
The framework considers both tree-like and closed kinematic structures and allows a flexible configuration of joints and constraints.
17
+
To project equations from individual rigid bodies to a multi-body system, Jacobians are used.
18
+
For closed kinematic chains, a novel formulation that features Lagrange multipliers is developed.
19
+
In a detailed mathematical proof, we show that our constraint formulation leads to an exact kinematic solution and converges in a single iteration.
20
+
Based on the proposed framework, we extend *ICG*, which is a state-of-the-art rigid object tracking algorithm, to multi-body tracking.
21
+
For the evaluation, we create a highly-realistic synthetic dataset that features a large number of sequences and various robots.
22
+
Based on this dataset, we conduct a wide variety of experiments that demonstrate the excellent performance of the developed framework and our multi-body tracker.
12
23
13
24
14
25
## Videos
@@ -29,4 +40,69 @@ Kinematic structures are very common in the real world. They range from simple a
29
40
</a>
30
41
31
42
32
-
Code and dataset coming soon ...
43
+
## The Robot Tracking Benchmark (RTB)
44
+
The *Robot Tracking Benchmark (RTB)* is a synthetic dataset that facilitates the quantitative evaluation of 3D tracking algorithms for multi-body objects.
45
+
It is publicly available on [Zenodo](https://zenodo.org/record/7548537).
46
+
The dataset was created using the procedural rendering pipeline [BlenderProc](https://github.com/DLR-RM/BlenderProc).
47
+
It contains photo-realistic sequences with [HDRi lighting](https://polyhaven.com/hdris) and physically-based materials.
48
+
Perfect ground truth annotations for camera and robot trajectories are provided in the [BOP format](https://github.com/thodan/bop_toolkit/blob/master/docs/bop_datasets_format.md).
49
+
Many physical effects, such as motion blur, rolling shutter, and camera shaking, are accurately modeled to reflect real-world conditions.
50
+
For each frame, four depth qualities exist to simulate sensors with different characteristics.
51
+
While the first quality provides perfect ground truth, the second considers measurements with the distance-dependent noise characteristics of the *Azure Kinect* time-of-flight sensor.
52
+
Finally, for the third and fourth quality, two stereo RGB images with and without a pattern from a simulated dot projector were rendered.
53
+
Depth images were then reconstructed using *Semi-Global Matching (SGM)*.
54
+
<palign="center">
55
+
<imgsrc="robots.png"width=600>
56
+
<br>
57
+
<em>Multi-body objects included in the RTB dataset</em>
58
+
</p>
59
+
60
+
The benchmark features six robotic systems with different kinematics, ranging from simple open-chain and tree topologies to structures with complex closed kinematics.
61
+
For each robotic system, three difficulty levels are provided: *easy*, *medium*, and *hard*.
62
+
In all sequences, the kinematic system is in motion.
63
+
While for *easy* sequences the camera is mostly static with respect to the robot, *medium* and *hard* sequences feature faster and shakier motions for both the robot and camera.
64
+
Consequently, motion blur increases, which also reduces the quality of stereo matching.
65
+
Finally, for each object, difficulty level, and depth image quality, 10 sequences with 150 frames are rendered.
66
+
In total, this results in 108.000 frames that feature different kinematic structures, motion patterns, depth measurements, scenes, and lighting conditions.
67
+
In summary, the *Robot Tracking Benchmark* allows to extensively measure, compare, and ablate the performance of multi-body tracking algorithms, which is essential for further progress in the field.
68
+
69
+
<palign="center">
70
+
<imgsrc="depth_qualities.png"width=600>
71
+
<br>
72
+
<em>Depth image qualities provided in the RTB dataset</em>
73
+
</p>
74
+
75
+
76
+
## Training Data Generation
77
+
To generate your own data similar to sequences provided in the *RTB* dataset, please use the provided BlenderProc scripts in the following [directory](https://github.com/DLR-RM/3DObjectTracking/tree/master/Mb-ICG/data_generation).
78
+
79
+
80
+
## Code
81
+
The *Mb-ICG* tracker was fully integrated into the multi-body, multi-modality, and multi-camera tracking library *M3T*.
82
+
In contrast to *Mb-ICG*, *M3T* provides additional functionality, such as a keypoint-based texture modality, capabilities for multi-region tracking, and soft constraints.
83
+
Note that it was ensured that evaluation results for *Mb-ICG* and *M3T* are exactly the same.
84
+
If you would like to reproduce results from our publication or use the tracker in your own application, please use the code for *M3T* from the following [directory](https://github.com/DLR-RM/3DObjectTracking/tree/master/M3T).
85
+
86
+
87
+
## Evaluation
88
+
The [code](https://github.com/DLR-RM/3DObjectTracking/tree/master/M3T/examples) in `examples/evaluate_RTB_dataset.cpp` and `examples/parameters_study_RTB.cpp` was used for the evaluation on the [RTB](https://zenodo.org/record/7548537) dataset.
89
+
To reproduce results, please download the dataset and adjust the `dataset_directory` in the source code.
90
+
Files for sparse viewpoint models will be created automatically and are stored in the specified `external_directory`.
91
+
To reproduce the evaluation results of *DART* on the *RTB* dataset, please unzip the `poses_rtb.zip` file and store its content in the respective `external_directory`.
92
+
In the code, you can then set `evaluator.set_evaluate_external(true);` and `evaluator.set_external_results_folder("dart");`.
93
+
Convergence experiments for the proposed constraints can be reproduced using `examples/constraint_convergence.cpp`.
94
+
To evaluate orthogonality constraints, which ensure that axes of reference frames are orthogonal, set the parameter in `include/m3t/constaint.h` to `kOrthogonalityConstraintExperiment = true`.
95
+
Results for the optimization time of the projected and constrained configuration over a varying number of bodies can be reproduced using `examples/optimization_time.cpp`.
96
+
In both experiments, outputs are generated that can be used together with the *LaTeX* package *PGFPlots* and that were used to generate the corresponding plots in the paper.
97
+
98
+
99
+
## Citation
100
+
If you find our work useful, please cite us with:
101
+
102
+
```
103
+
@Article{Stoiber2023,
104
+
author = {Stoiber, Manuel and Sundermeyer, Martin and Boerdijk, Wout and Triebel, Rudolph},
105
+
title = {A Multi-body Tracking Framework - From Rigid Objects to Kinematic Structures},
0 commit comments