For processing Insta360 TITAN .mp4 video into the format:
<sequence>/
images/ stitched frames (.png)
cam_timestamps.txt one frame timestamp per line (seconds, monotonically increasing)
imu_data.csv header + rows: timestamp,ax,ay,az,gx,gy,gz
Put input videos in:
data/videos/
Run preprocessing from the repo root:
.\.venv\Scripts\activate.ps1
python .\scripts\preprocessing\preprocess_video.py --name <sequence_name> --start <seconds or HH:MM:SS> --duration <seconds> --overwrite
If there is exactly one .mp4 in data/videos/ then --video can be omitted. If there are multiple videos:
python .\scripts\preprocessing\preprocess_video.py --video <your_video.mp4> --name <sequence_name>
The script writes interim files to:
data/tmp_interim_components/
The script writes final files to:
results/<sequence_name>/
results/<sequence_name>/gps.txt
results/<sequence_name>/<sequence_name>/images/
results/<sequence_name>/<sequence_name>/imu_data.csv
results/<sequence_name>/<sequence_name>/cam_timestamps.txt
Optional arguments:
--start <seconds or HH:MM:SS>
--duration <seconds>
--fps <output fps>
--pre-imu-samples 700
--post-imu-samples 200
--overwrite
--run-calibration
--calibration-frame-step 10
--overwrite deletes and recreates images/ before extracting frames. Without it the script will refuse to run if images/ already contains files, to avoid silently mixing frames from different runs.
The IMU output is automatically trimmed to keep 700 IMU recordings before the first processed frame and 200 after the last processed frame, or all available rows if fewer exist.
Passing --run-calibration runs refine_tbc_rotation_from_sequence.py at the end of the pipeline as a single combined command. The output is written to results/<sequence_name>/tbc_rotation.yaml. --calibration-frame-step controls how many frames are skipped between each pair used for calibration (larger values → larger per-pair motion → more reliable rotation estimate, but fewer pairs).
Extract embedded data (IMU, GPS) from the mp4
exiftool -ee path/to/<your_video.mp4> > data/tmp_interim_components/<sequence_name>_embedded.txt
The scripts use the first high precision GPS Date/Time line as t0 for both frame timestamps and IMU timestamps. Example:
GPS Date/Time : 2024:07:22 07:29:10.116064Z
This is already UTC. The scripts convert it to epoch seconds automatically.
Collect time stamps from the video with the following.
ffprobe -v error -select_streams v:0 -show_entries frame=best_effort_timestamp_time -of csv=p=0:nk=1 path/to/<your_video.mp4> > data/tmp_interim_components/<sequence_name>_cam_timestamps_raw.txt
Remove commas and add the epoch offset to timestamps by running:
python .\scripts\preprocessing\process_timestamps.py data/tmp_interim_components/<sequence_name>_cam_timestamps_raw.txt results/<sequence_name>/<sequence_name>/cam_timestamps.txt --embedded data/tmp_interim_components/<sequence_name>_embedded.txt --start <seconds> --duration <seconds>
The script removes ffprobe commas automatically. No Ctrl+h replacement is needed.
You can also pass t0 directly:
python .\scripts\preprocessing\process_timestamps.py <raw_timestamps.txt> <cam_timestamps.txt> --t0 1721633350.116064
embedded.txt will include a timestamp (e.g. "2024:07:22 07:29:10.116064Z") for video beginning. If you want to translate manually on Windows:
[datetime]::Parse("2024-07-22 07:29:10.116064Z").ToUniversalTime().Subtract([datetime]"1970-01-01").TotalSeconds
Properly format the IMU data from embedded.txt in a csv by running:
python .\scripts\preprocessing\process_imu.py data/tmp_interim_components/<sequence_name>_embedded.txt results/<sequence_name>/<sequence_name>/imu_data.csv --frame-timestamps results/<sequence_name>/<sequence_name>/cam_timestamps.txt
Note: If you choose to edit the IMU data, such as removing rows, do not do it in Excel. When you save the file again it will truncate decimals because Excel cannot hold large floats.
GPS is not needed by the VIO software, but it is useful for post-processing. Extract it from embedded.txt with:
python .\scripts\preprocessing\process_gps.py data/tmp_interim_components/<sequence_name>_embedded.txt results/<sequence_name>/gps.txt
This replaces the manual CammReader process for the normal preprocessing path.
Extract frames from the video
ffmpeg -i .\data\videos\<your_video.mp4> -vf "select=gte(t\,<start>)*lt(t\,<end>)" -vsync 0 results/<sequence_name>/<sequence_name>/images/%06d.png
Run:
python .\scripts\preprocessing\preprocess_video.py --name <sequence_name> --start <seconds or HH:MM:SS> --duration <seconds> --overwrite
This extracts frames over the interval, filters the frame timestamps over the same interval, and trims the IMU data around the resulting first and last frame.
Run:
python .\scripts\preprocessing\preprocess_video.py --name <sequence_name> --fps <choiceFPS>
Check that the script does not warn about image/timestamp count mismatch.
If you do not have a dedicated high-excitement calibration video, you can estimate a rough camera-to-body rotation from the processed sequence:
python .\scripts\preprocessing\refine_tbc_rotation_from_sequence.py .\results\<sequence_name>\<sequence_name>
When the sequence directory follows the results/<name>/<name>/ layout, --write-yaml is optional — the script automatically writes to results/<sequence_name>/tbc_rotation.yaml. You can override the path explicitly:
python .\scripts\preprocessing\refine_tbc_rotation_from_sequence.py .\results\<sequence_name>\<sequence_name> --write-yaml .\results\<sequence_name>\tbc_rotation.yaml
Copy the T_BC block from tbc_rotation.yaml into the VIO config file.
The script prints excitation statistics to help judge result quality, for example:
Used 187 / 210 frame pairs
Final mean squared angle residual : 3.2e-04 rad²
IMU axis excitation (singular values): 0.8821, 0.4103, 0.0612
WARNING: motion is close to one-axis — pitch/roll rows of R_BC may be unreliable.
The three singular values describe how well the collected motion spans all three rotation axes (1.0 = full coverage, 0.0 = no coverage). A rover U-turn and straight drive mostly excites yaw; if the third singular value is less than ~15% of the first, the pitch and roll components of R_BC are weakly constrained. In that case treat the estimate as a rough initial guess and validate against Kalibr or TartanCalib if possible (requires specific AprilPatch video sequence).
The complete script calls command-line programs:
ffmpeg
ffprobe
exiftool
These are not Python libraries. They must either be on PATH or passed explicitly:
python .\scripts\preprocessing\preprocess_video.py --ffmpeg C:\path\to\ffmpeg.exe --ffprobe C:\path\to\ffprobe.exe --exiftool C:\path\to\exiftool.exe
A conda environment may be simplest route because it can install the Python packages and the command-line tools into one environment:
conda install -c conda-forge ffmpeg exiftool
pip install -r requirements.txt
If you are using only venv/pip, install ffmpeg and exiftool separately and either add them to PATH or pass the executable paths above.