VGGT-Splat

English | 中文

Casual phone video → VGGT feed-forward poses, points and depth → 3D Gaussian Splatting, compared against the usual COLMAP → 3DGS pipeline.

Project page: https://chu-jiaming.github.io/vggt-splat/ · Release: v1.0.0 (trained models, self-captured data)

Garden, all frames, 7k steps: both models rendered along one shared novel orbit (26.3 vs. 21.8 dB test PSNR). Full-resolution videos are on the project page.

Questions

1. Frame budget: how many frames does a casual capture need? Compare COLMAP + 3DGS and VGGT → 3DGS at 8 / 16 / 32 / all training frames.
2. VGGT depth prior: with few frames, how much quality does VGGT depth supervision recover?

6 scenes (self-captured desk / lawn / room; Mip-NeRF 360 garden / bicycle / counter) × 4 frame budgets × 3 methods = 72 runs, plus seed repeats.

Method

flowchart LR
    A["Phone video or image set<br/>test split + N = 8 · 16 · 32 · all<br/>training frames"] --> B["COLMAP<br/>exhaustive matching + SfM"]
    A --> C["VGGT-1B<br/>one forward pass"]
    B -->|"poses, points"| D["3DGS, 7k steps<br/>+ per-camera pose refinement"]
    C -->|"poses, points"| D
    C -.->|"depth prior"| D
    D --> E["Align test poses<br/>Sim(3) + 500-step refine<br/>scene frozen"]
    E --> F["PSNR / SSIM / LPIPS<br/>seen vs. unseen pixels"]

Loading

• COLMAP: the standard front end, exhaustive feature matching and incremental SfM on the N training frames.
• VGGT: VGGT-1B predicts cameras, a point map and depth for all N frames in one forward pass; the top 50% most confident points initialize the Gaussians.
• VGGT + depth: the same, plus a Pearson loss between rendered inverse depth and VGGT depth on the training views (λ = 0.2).
• Evaluation: test cameras come from a separate joint run on training + test frames, are mapped into each model's frame by a Sim(3) fitted on the training cameras, and are refined with the Gaussians frozen. No test image updates the scene.

Full protocol: Protocol.

Conclusions

1. Frame budget. At 16 frames or fewer, VGGT is the only one of the two front ends that reliably gives a reconstruction: COLMAP fails on 4 of 6 scenes at N = 8 and 3 of 6 at N = 16, while VGGT never fails. With a full capture of 110–180 frames, COLMAP gives higher quality, leading by 2.1–4.6 dB on five of six scenes; VGGT is less accurate than COLMAP but 3.5–13× faster at estimating poses.
2. VGGT depth prior. No measurable gain in novel-view quality: it pulls rendered depth towards VGGT's depth, but across seed repeats only two of the differences exceed noise (−0.68 dB on desk N = 32, +0.17 dB on lawn N = 8). With few frames the error sits in regions that no training view sees, and a depth loss on training views cannot reach them; more views, or priors that cover unseen regions, are what would help.

Findings

1. At 16 frames or fewer, VGGT is reliable and COLMAP is not; with all frames, COLMAP is better. At N = 8, COLMAP fails outright on 4 of 6 scenes and registers only 3/8 and 6/8 training frames on the other two; at N = 16 it fails on 3 scenes and registers 3/16 (bicycle) and 7/16 (desk) on two more. VGGT never fails. With all frames, COLMAP leads by 2.1–4.6 dB on five scenes and 0.7 dB on desk.
2. With all frames, VGGT loses on precision, not coverage. Restricted to test pixels that the training views do see, VGGT is still behind (garden 21.7 vs 26.3 dB, bicycle 20.7 vs 22.8, counter 24.9 vs 27.2). Its renders are visibly smeared (higher PSNR than at N = 32 but worse LPIPS on lawn, garden and bicycle), likely because feed-forward poses are not mutually consistent enough for 3DGS to resolve fine detail.
3. The depth prior changes geometry but not novel-view quality. It raises the local correlation between rendered depth and VGGT depth on training views from 0.4–0.8 to 0.7–0.9, yet across seed repeats only two differences exceed noise: desk N = 32 gets 0.68 dB worse and lawn N = 8 gets 0.17 dB better.
4. With few frames, the bottleneck is coverage. Test pixels not seen by any training view are 0.5–9 dB worse than seen ones, and the per-frame unseen fraction is strongly anti-correlated with PSNR. A depth prior on training views cannot constrain those regions, which also explains finding 3.
5. Noise level. Re-running the same configuration with different seeds moves PSNR by a standard deviation of 0.06–0.35 dB, so single-run differences below about 0.3 dB are not conclusive.
6. Speed. VGGT estimates poses for all frames in 47–73 s; COLMAP (exhaustive matching, GPU) takes 167–790 s.

Results

Test PSNR (dB) after test-time pose optimization, averaged over the test frames that every usable method registered for that scene and N. fail = pose estimation or alignment failed, or the alignment was flagged as inconsistent (center error > 10%). Numbers in brackets are COLMAP's registered / total test frames when it did not register all of them; the other methods are averaged over the same subset.

Scene	Method	N=8	N=16	N=32	N=all
desk	COLMAP	fail	25.75 (7/16)	23.47	32.46
desk	VGGT	24.36	30.20	32.34	31.79
desk	VGGT + depth	24.36	30.20	31.79	30.97
lawn	COLMAP	13.75 (15/23)	fail	18.67 (22/23)	21.44 (22/23)
lawn	VGGT	15.66	17.47	18.51	19.35
lawn	VGGT + depth	15.61	17.41	18.64	19.39
room	COLMAP	fail	fail	fail	23.40 (14/17)
room	VGGT	13.77	16.62	18.24	21.20
room	VGGT + depth	13.77	16.52	18.54	21.64
garden	COLMAP	14.27	19.57	23.19	26.34
garden	VGGT	15.64	18.13	19.93	21.79
garden	VGGT + depth	15.70	18.17	19.99	22.39
bicycle	COLMAP	fail	12.03 (9/24)	14.25	23.40
bicycle	VGGT	14.42	16.45	18.26	21.30
bicycle	VGGT + depth	14.46	16.08	18.28	20.93
counter	COLMAP	fail	fail	21.04	27.25
counter	VGGT	14.12	19.02	21.58	25.04
counter	VGGT + depth	14.40	18.72	22.44	24.71

COLMAP at N = 32 on desk and bicycle registers only 15/32 and 10/32 training frames, which is why it drops below N = 16 there.

Pose estimation time (s, single RTX 4090, includes model loading and image I/O):

Scene	COLMAP all	VGGT all
desk (110 frames)	167	47
lawn (156)	790	60
room (114)	505	49
garden (161)	770	68
bicycle (166)	380	68
counter (180)	480	73

Protocol

• Data. Each scene's frames are split into a test set (every 8th frame after removing blurry frames, as in Mip-NeRF 360) and nested training sets of N = 8 / 16 / 32 / all frames sampled uniformly from the rest (scripts/extract_frames.py). Self-captured scenes are 1080p phone videos sampled at 3 fps and trained at 960×540; Mip-NeRF 360 scenes use images_4.
• Poses. COLMAP 4.2.1 with exhaustive matching (scripts/run_colmap.sh), or VGGT-1B in one forward pass with a shared camera and the top 50% most confident points as initialization (scripts/vggt_to_colmap.py).
• Training. gsplat 3DGS with per-camera pose refinement (--pose_opt), stopped at step 7,000 of the default 30k schedule. The depth-prior variant adds a Pearson loss between rendered inverse depth and VGGT depth, λ = 0.2 (chosen by a sweep on desk).
• Evaluation (scripts/eval.py). Test cameras come from a joint run on training + test frames; a Sim(3) fitted on the shared training cameras (rotation from camera orientations, scale and translation from centers) maps them into the trained model's frame; with the Gaussians frozen, each test pose is refined for 500 steps; PSNR / SSIM / LPIPS are measured after refinement. Methods are compared on the test frames that all of them registered.
• Analysis. scripts/coverage_err.py splits test error by whether a pixel is seen by any training view; scripts/depth_diag.py measures agreement between rendered depth and VGGT depth; scripts/seed_repeat.sh re-runs selected configurations with seeds 1 and 2.

Usage

Environment

Tested on Ubuntu 22.04, CUDA 12.8, one RTX 4090 (24 GB), Python 3.11:

Component	Version
PyTorch	2.9.1 + cu128
gsplat	commit 512d366 (built from source)
VGGT	commit a288dd0, weights facebook/VGGT-1B
COLMAP	4.2.1 built with CUDA

uv venv .venv --python 3.11 && source .venv/bin/activate
uv pip install torch==2.9.1 torchvision==0.24.1 --index-url https://download.pytorch.org/whl/cu128
git clone --recursive https://github.com/nerfstudio-project/gsplat.git && git -C gsplat checkout 512d366b67073d77ca099ede742683c165dfc23b && git -C gsplat submodule update --init --recursive
uv pip install setuptools wheel ninja && uv pip install -e gsplat --no-build-isolation
uv pip install -r gsplat/examples/requirements.txt --no-build-isolation
git clone https://github.com/facebookresearch/vggt.git && git -C vggt checkout a288dd0f14786c93483e45524328726ab7b1b4ce
uv pip install -e vggt --no-deps && uv pip install einops safetensors huggingface_hub trimesh hydra-core omegaconf
git clone https://github.com/chu-jiaming/VGGT-Splat.git

3DGS training uses this repository's trainer/, a modified copy of gsplat examples/ at the same commit (changes listed in trainer/NOTICE.md); the gsplat library and trainer/ must stay on the same commit.

Data

data/<scene>/frames/NNNN.jpg      all frames
data/<scene>/splits/              test.txt, train_{8,16,32,all}.txt (from extract_frames.py)

• Mip-NeRF 360: download 360_v2.zip, point data/<scene>/frames at 360_v2/<scene>/images_4 with a symlink, then run python scripts/extract_frames.py data/<scene>.
• Self-captured scenes (desk, lawn, room): download vggt-splat_data_<scene>.zip from the v1.0.0 release and unzip it inside data/; it contains the extracted frames/ and the exact splits/ used here (joint_*.txt are regenerated by run_all.sh). For a new capture, run python scripts/extract_frames.py data/<scene> on a folder containing video.mp4.

Run

scripts/run_all.sh runs pose estimation → joint run → training → evaluation for every scene × method × N and appends one row per run to results/<scene>.csv. It resumes from existing outputs. Paths come from environment variables: DATA, RUNS, RESULTS, VENV (the virtual environment, its bin/ goes first on PATH), HF_HOME / TORCH_HOME (weight caches) and HF_ENDPOINT (defaults to https://hf-mirror.com; can be set to https://huggingface.co). SCENES, METHODS and NS select a subset, e.g. SCENES=garden NS=8 for a ~10 minute check.

export VENV=$PWD/.venv DATA=$PWD/data RUNS=$PWD/runs RESULTS=$PWD/results
SCENES="desk lawn room garden bicycle counter" METHODS="colmap vggt" scripts/run_all.sh
SCENES="desk lawn room garden bicycle counter" METHODS=vggt_depth DEPTH_LAMBDAS=0.2 scripts/run_all.sh
python scripts/summarize.py --results results --runs runs       # tables, curves, failure list → results/summary/

Analysis and exports:

python scripts/coverage_err.py --runs runs --out results/coverage.csv     # --save_maps for GT | render | error | coverage images
python scripts/depth_diag.py --runs runs --out results/depth_diag.csv
scripts/seed_repeat.sh job desk vggt_depth 32 1 0.2 && scripts/seed_repeat.sh summary
python scripts/export_splats.py --runs runs --out results/export --list "garden/vggt_all garden/colmap_all"   # .splat + compressed .ply
RUNS=runs scripts/rerender_traj.sh garden/vggt_all garden/colmap_all   # slower fly-through videos (450 frames) from existing checkpoints
python scripts/render_orbit.py --runs runs --out results/orbit --scene garden --frames 600   # COLMAP and VGGT models along one shared novel orbit

Trained models

The v1.0.0 release has the all-frames COLMAP and VGGT models of every scene (<scene>_<colmap|vggt>_all, step 7,000), exported by scripts/export_splats.py in two formats:

• .splat (7–144 MB): base color only, for web viewers that read the antimatter15 .splat format.
• .compressed.ply (13–276 MB): PlayCanvas compressed PLY with spherical harmonics, for viewers such as SuperSplat.

Each model is in its own normalized world frame, so the COLMAP and VGGT models of a scene are not aligned with each other; scripts/render_orbit.py shows how to map one into the other.

Limitations

• One capture per scene and, for the main grid, one seed; seed repeats cover only a subset of configurations.
• Training stops at 7k steps for every run. On desk N = 16, the 30k checkpoint of the same run was slightly worse on test frames (extra steps overfit the training views) with the same method ranking; other configurations were not checked.
• COLMAP is not deterministic with few frames. Reproducing garden N = 8 in a fresh environment gave VGGT 15.67 dB (15.64 here), but COLMAP registered 5/8 training frames instead of 6/8 and reached 13.35 dB instead of 14.27; five further COLMAP runs on the same frames all registered 6/8.
• Whether a test pixel is "seen" by a training view is judged from the geometry each model reconstructs; that geometry may itself be inaccurate, so the split is only an approximation.
• Test cameras are obtained from a separate joint run and refined with the scene frozen; this is the standard protocol for pose-free methods but still differs from evaluating with ground-truth test poses.

Acknowledgements

• VGGT
• gsplat: 3DGS training; trainer/ is adapted from its examples/
• COLMAP
• Mip-NeRF 360

License

MIT, except trainer/ (Apache 2.0, see NOTICE.md).

VGGT weights are for non-commercial use only, under the license in the VGGT repository; other third-party dependencies keep their own licenses.