This repository provides the implementation of VoMP. TL;DR: Feed-forward, fine-grained, physically based volumetric material properties from Splats, Meshes, NeRFs, etc. which can be used to produce realistic worlds. We recommend reading the README_train.md if you need to fine-tune or train the model from scratch or know more details about the codebase.
Note
This repository now contains VoMP 1.0 and VoMP 1.1. VoMP 1.1 is a more robust model. The instructions in README work for either version, simply change the config file you use.
- 🔧 Dependencies and Installation
- 🌐 Quickstart: Web Demo
- 📥 Loading the Model
- 🎯 High-Level API
- 🎨 Visualizing Material Results
- 🔧 Low-Level API
- 🧩 Custom 3D Representations
- 🧬 Material Upsampler
- 💾 Using our Benchmark
- 📦 Simulation
- 🤗 Credits
- 📜 Citation
- License and Contact
All the instructions in this README are meant to be run from the root of the repository. Running simulations requires a separate set of dependencies than this setup which we mention later in the 📦 Simulation section.
First set up the environment. We recommend using Python 3.11, PyTorch>=2.1.0, and CUDA>=11.8. It is okay if some packages show warnings or fail to install due to version conflicts. The version conflicts are not a problem for the functionalities we use.
git clone --recursive https://github.com/nv-tlabs/VoMP
cd VoMP
# Install dependencies using the provided script (Linux only)
chmod +x install_env.sh
./install_env.shNote
Running install_env.sh without conda: The script includes optional conda-only steps (environment creation/activation, installing CUDA toolkit inside the env, and setting env vars). If you're using a Python venv and don't have conda, the script will fail when it tries to call conda. You can either install conda, or comment out the conda-specific lines (lines 93-115 and any conda install / conda env config vars set commands). The rest of the script relies on pip and standard bash commands and will work in a venv.
The mesh and USD pipeline uses bpy==4.2.0. It renders the geometry and saves the mesh used for voxelization in the same coordinate frame. The high-level APIs use this installed module by default:
results = model.get_mesh_materials("mesh.obj")
results = model.get_usd_materials("model.usd")A Blender 4.2 executable can also be passed through blender_path.
For rendering with Isaac Sim, you need to install it manually. This is not required for Gaussian splat processing.
Download Isaac Sim from here and follow the instructions to install it. On Linux, you would have a isaac-sim.sh file in the path you installed it. For Windows, you would have a isaac-sim.bat file in the path you installed it. Note the path to the isaac-sim.sh or isaac-sim.bat file.
Note
Provide the Isaac Sim binary path through isaac_sim_path when calling model.render_views_replicator(), or set the ISAAC_SIM_PATH environment variable.
Warning
Replicator renders USD assets. For meshes in other formats, first convert them to USD using existing tools.
We provide the trained models (3.44 GB for both versions, including fine-tuning checkpoints) in 
weights/, keeping the vomp_1_0/ and vomp_1_1/ folders. The examples below use VoMP 1.1: weights/vomp_1_1/matvae.safetensors and weights/vomp_1_1/geometry_transformer.pt.
For VoMP 1.0, use the matching files and inference config under weights/vomp_1_0/. Each version folder contains the files below. If you need to fine-tune the model, also download that version's ft/ directory, for example weights/vomp_1_1/ft/.
| File | Model |
|---|---|
matvae.safetensors |
MatVAE |
geometry_transformer.pt |
Geometry Transformer |
normalization_params.json |
Normalization Parameters |
inference.json |
Inference Configuration |
ft |
MatVAE and Geometry Transformer checkpoints (same as above but in a directory structure compatible for fine-tuning) |
We provide a simple web demo to quickly test out VoMP in a GUI. The web-demo uses some additional dependecies over the base environment, see gradio/requirements.txt. To start it, run:
pip install --no-build-isolation -r gradio/requirements.txt
python gradio/app.pyThen, you can access the demo at the address shown in the terminal. The web demo allows you to run the model, visualize the outputs of the model and download an artifact which can directly be used for 📦 Simulation.
Before using any of the VoMP APIs, you need to load the model. We provide flexible loading options:
The simplest way to load the model is using the inference configuration file:
from vomp.inference import Vomp
# Load model using inference config (recommended)
model = Vomp.from_checkpoint(
config_path="weights/vomp_1_1/inference.json",
use_trt=False # Set to True to enable TensorRT acceleration (significantly faster but requires `torch-tensorrt`)
)Note
Using the use_trt flag will compile the DINO model with TensorRT. This makes the from_checkpoint function slower.
For more control, you can specify exact checkpoint files, optionally overriding the inference config:
# Load model using direct file paths
model = Vomp.from_checkpoint(
config_path="weights/vomp_1_1/inference.json",
geometry_checkpoint_dir="weights/vomp_1_1/geometry_transformer.pt",
matvae_checkpoint_dir="weights/vomp_1_1/matvae.safetensors",
normalization_params_path="weights/vomp_1_1/normalization_params.json"
)
# Or override specific paths from inference config
model = Vomp.from_checkpoint(
config_path="configs/materials/v1.1/inference.json",
geometry_checkpoint_dir="custom/path/to/geometry_transformer.pt" # Override just this path
)Use this approach only if you are using the fine-tuning checkpoints i.e. the ft/ directory for your model version. This lets the model auto-find the latest geometry checkpoint:
from vomp.inference import Vomp
# Load model using checkpoint directories
model = Vomp.from_checkpoint(
config_path="weights/vomp_1_1/inference.json",
geometry_checkpoint_dir="weights/vomp_1_1/ft/geometry_transformer",
matvae_checkpoint_dir="weights/vomp_1_1/ft/matvae/final_model",
normalization_params_path="weights/vomp_1_1/ft/matvae/normalization_params.json",
geometry_ckpt="latest" # Can also be a specific step number
)We provide a flexible Python API with both high-level and low-level interfaces for material property estimation.
For Gaussian splats, use the high-level API for the easiest experience (see Loading the Model section first):
from vomp.inference.utils import save_materials
# Get materials directly from PLY (auto-handles Gaussian loading)
# By default, returns materials evaluated at each Gaussian splat center
results = model.get_splat_materials("path/to/your/gaussian_splat.ply")
# Or use Kaolin voxelizer for more accurate results
# results = model.get_splat_materials("path/to/your/gaussian_splat.ply", voxel_method="kaolin")
# Control where materials are evaluated using query_points:
# results = model.get_splat_materials("path/to/your/gaussian_splat.ply", query_points="splat_centers") # Default
# results = model.get_splat_materials("path/to/your/gaussian_splat.ply", query_points="voxel_centers") # Voxel centers (direct output of the model)
# results = model.get_splat_materials("path/to/your/gaussian_splat.ply", query_points=custom_points) # Custom (N,3) array
# Adjust DINO batch size for performance (higher values use more GPU memory)
# results = model.get_splat_materials("path/to/your/gaussian_splat.ply", dino_batch_size=32)
# Save results
save_materials(results, "materials.npz")For mesh objects, use the equivalent high-level mesh API (see Loading the Model section first):
from vomp.inference.utils import save_materials
# Get materials directly from mesh file (supports OBJ, PLY, STL, USD)
# By default, returns materials evaluated at each mesh vertex (not recommended if you have vertices only on the surface)
results = model.get_mesh_materials("path/to/your/mesh.obj")
# Control where materials are evaluated using query_points:
# results = model.get_mesh_materials("path/to/your/mesh.obj", query_points="mesh_vertices") # Default
# results = model.get_mesh_materials("path/to/your/mesh.obj", query_points="voxel_centers") # Voxel centers (direct output of the model)
# results = model.get_mesh_materials("path/to/your/mesh.obj", query_points=custom_points) # Custom (N,3) array
# Adjust DINO batch size for available GPU memory.
# results = model.get_mesh_materials("path/to/your/mesh.obj", dino_batch_size=32)
# Save results
save_materials(results, "materials.npz")USD files can come in many different formats with varying internal structures, materials, and organization. For USD assets, use the high-level USD API (see Loading the Model section first):
The USD API renders the asset's composed geometry and textures and uses that same geometry for voxelization:
from vomp.inference.utils import save_materials
# Render the USD with Blender and use the same geometry for voxelization
results = model.get_usd_materials(
usd_path="path/to/your/model.usd",
query_points="voxel_centers", # Default (direct output)
return_original_scale=True, # Return positions in the original asset's coordinates
)
# Control where materials are evaluated using query_points:
# results = model.get_usd_materials(..., query_points="voxel_centers") # Default (direct output)
# results = model.get_usd_materials(..., query_points=custom_points) # Custom (N,3) array in normalized coordinates
# Adjust DINO batch size for performance (higher values use more GPU memory):
# results = model.get_usd_materials(..., dino_batch_size=32)
# Save results
save_materials(results, "materials.npz")If your USD file is in the SimReady format (like the USD files in our dataset), you can use the following arguments:
from vomp.inference.utils import save_materials
results = model.get_usd_materials(
usd_path="model.usd",
use_simready_usd_format=True,
seed=42
)
# Control where materials are evaluated using query_points:
# results = model.get_usd_materials(..., query_points="voxel_centers") # Default (direct output)
# results = model.get_usd_materials(..., query_points=custom_points) # Custom (N,3) array
# Save results
save_materials(results, "materials.npz")After estimating material properties, you can visualize them using our interactive polyscope-based viewer.
# After getting results from any API
from vomp.inference.utils import save_materials
# Save your results
save_materials(results, "my_materials.npz")# Launch interactive property viewer
python scripts/viewer.py my_materials.npzThe viewer also saves the colorbars for visualizations as PNG images that look like this:
For fine-grained control with Gaussian splats, use the low-level API (see Loading the Model section first):
from vomp.representations.gaussian import Gaussian
from vomp.inference.utils import save_materials
# Load Gaussian splat
gaussian = Gaussian(sh_degree=3, aabb=[-1,-1,-1,2,2,2], device="cuda")
gaussian.load_ply("path/to/your/gaussian_splat.ply", transform=None)
gaussian = model._normalize_gaussian(gaussian)
# Step-by-step pipeline
output_dir = "outputs/materials"
renders_metadata = model.render_sampled_views(gaussian, output_dir, num_views=150)
voxel_centers = model._voxelize_gaussian(gaussian, output_dir)
coords, features = model._extract_dino_features(output_dir, voxel_centers, renders_metadata, save_features=True)
results = model.predict_materials(coords, features)
save_materials(results, "materials.npz")For fine-grained control with meshes, use the equivalent low-level mesh API (see Loading the Model section first):
from vomp.inference.utils import save_materials
# Step-by-step pipeline for meshes
output_dir = "outputs/materials"
mesh_path = "path/to/your/mesh.obj"
renders_metadata = model.render_mesh_views(mesh_path, output_dir, num_views=150)
voxel_centers = model._voxelize_mesh(mesh_path, output_dir)
coords, features = model._extract_dino_features(output_dir, voxel_centers, renders_metadata, save_features=True)
results = model.predict_materials(coords, features)
save_materials(results, "materials.npz")Use the same rendering and voxelization stages as meshes. USD assets come in many formats sodataset="common" selects the general USD material loader; use "simready" for SimReady assets.
from vomp.inference.utils import save_materials
output_dir = "outputs/materials"
usd_path = "path/to/your/model.usd"
renders_metadata = model.render_mesh_views(usd_path, output_dir, dataset="common")
voxel_centers = model._voxelize_mesh(usd_path, output_dir)
coords, features = model._extract_dino_features(
output_dir, voxel_centers, renders_metadata, save_features=True
)
results = model.predict_materials(coords, features)
save_materials(results, "materials.npz")Bring your own 3D representation with custom render/voxelize functions (see Loading the Model section first):
from vomp.inference.utils import save_materials
def my_render_func(obj, output_dir, num_views, image_size, **kwargs):
# Your rendering code here - save images to output_dir/renders/
frames_metadata = []
for i in range(num_views):
# Your custom rendering logic
frames_metadata.append({
"file_path": f"frame_{i:04d}.png",
"transform_matrix": camera_matrix.tolist(), # 4x4 matrix
"camera_angle_x": fov_radians
})
# Save RGBA images and their camera manifest for feature-cache validation.
import json
from pathlib import Path
render_dir = Path(output_dir) / "renders"
render_dir.mkdir(parents=True, exist_ok=True)
(render_dir / "transforms.json").write_text(json.dumps({"frames": frames_metadata}))
return frames_metadata
def my_voxelize_func(obj, output_dir, **kwargs):
# Your voxelization code here
# Return unique 64^3 cell centers: (integer_indices + 0.5) / 64 - 0.5.
voxel_centers = your_voxelization_method(obj) # (N, 3) array
return voxel_centers
# Use with any 3D representation
coords, features = model.get_features(
obj_3d=your_mesh,
render_func=my_render_func,
voxelize_func=my_voxelize_func,
num_views=150
)
# Get materials
results = model.predict_materials(coords, features)
save_materials(results, "materials.npz")The high-level splat API (get_splat_materials) automatically returns materials interpolated to splat centers. However, you may want to upsample materials to other locations like higher resolution grids or custom query points. We provide a utility class for these cases (see Loading the Model section first).
import numpy as np
from vomp.inference.utils import MaterialUpsampler
from vomp.representations.gaussian import Gaussian
# Get voxel-level materials (needed for upsampling to custom locations)
voxel_results = model.get_splat_materials(
"path/to/your/gaussian_splat.ply",
query_points="voxel_centers", return_original_scale=False,
)
# OR for meshes
# voxel_results = model.get_mesh_materials("path/to/your/mesh.obj", query_points="voxel_centers", return_original_scale=False)
# Create upsampler from voxel-level results
upsampler = MaterialUpsampler(
voxel_coords=voxel_results["voxel_coords_world"],
voxel_materials=np.column_stack([
voxel_results["youngs_modulus"],
voxel_results["poisson_ratio"],
voxel_results["density"]
])
)
# Example 1: Interpolate to Gaussian centers manually (usually not needed - high-level API does this)
gaussian = Gaussian(sh_degree=3, aabb=[-1,-1,-1,2,2,2], device="cuda")
gaussian.load_ply("path/to/your/gaussian_splat.ply", transform=None)
gaussian = model._normalize_gaussian(gaussian)
gaussian_materials, gaussian_distances = upsampler.interpolate_to_gaussians(gaussian)
# Example 2: Interpolate to higher resolution grid (128x128x128) - main use case for manual upsampling
x = np.linspace(-0.5, 0.5, 128)
xx, yy, zz = np.meshgrid(x, x, x)
high_res_points = np.column_stack([xx.ravel(), yy.ravel(), zz.ravel()])
high_res_materials, high_res_distances = upsampler.interpolate(high_res_points)
# Example 3: Interpolate to custom query points - another main use case for manual upsampling
query_points = np.random.uniform(-0.4, 0.4, size=(1000, 3))
query_materials, query_distances = upsampler.interpolate(query_points)
# Save results
upsampler.save_results(gaussian.get_xyz.detach().cpu().numpy(), gaussian_materials,
gaussian_distances, "gaussian_materials.npz")
upsampler.save_results(high_res_points, high_res_materials, high_res_distances, "high_res_materials.npz")
upsampler.save_results(query_points, query_materials, query_distances, "custom_materials.npz")Note
Due to licenses we are unable to make the vegetation subset of the dataset public. Thus, when you compare outputs to the paper make sure to compare them to the listed results on the "public dataset" (Table 2 and Table 3).
We provide a dataset and a benchmark with fine-grained volumetric mechanical properties (65.9 GB) at 
Since our dataset is quite large, we provide a way to download only the test set by running the following command:
huggingface-cli download nvidia/PhysicalAI-Robotics-PhysicalAssets-VoMP-Eval --repo-type dataset --local-dir datasets/simready_benchmarkWe can now run VoMP on the test set:
python scripts/evaluate_geometry_encoder.py \
--config configs/materials/v1/inference.json \
--checkpoint_dir weights/vomp_1_0/ft/geometry_transformer \
--data_dir datasets/simready_benchmark \
--ckpt latestThis script requires loading the model in the "Using Directories" method.
This prints out many detailed metrics. Particularly, you can also make sure you can reproduce the main results from the paper by comparing Table 2 and Appendix Table 3 from the paper with the outputs from Section 5 (SUMMARY TABLES) of the results printed out.
To build on top of our benchmark, you can replace the load_model and evaluate_model functions in the scripts/evaluate_geometry_encoder.py script with your own model and evaluation code.
Our properties are compatible with all simulators. We provide instructions to run a few kinds of simulations with the properties.
For the large-scale simulations that we perform with Simplicits, refer to the Simplicits documentation.
We provide a way to run FEM simulations using warp.fem.
cd simulation/warp.fem
PYTHONPATH=./ python drop_tetmesh.py --mesh assets/cube_res20.msh --materials assets/cube_materials_two_halves.npz This simple example has an artificially constructed NPZ file which can be used in warp.fem. This requires installing warp and meshio.
We provide a way to run FEM simulations using libuipc. These simulations use the config files in the configs/sim/ directory and they can be run as,
python vomp/sim/main.py configs/sim/falling_oranges.jsonThis config runs a simulation of falling oranges (Figure 5 from the paper) with the NPZ files we generated from the VoMP model.
These simulations require a .npz file with the estimated mechanical properties of the object. This requires installing the Python version of libuipc using the instructions in the libuipc repository. The command above will run the simulation, show it in a GUI, and save framewise surface meshes in the outputs/simulation_output/falling_oranges directory. The config also specifies a visual textured surface mesh so the per frame visualizations will use the high resolution visual mesh and also have textures.
We provide a way to run Newton simulations. Run an example simulation of a soft body cube with the NPZ files we generated from the VoMP model by running the following command:
cd simulation/newton
python mesh_falling_sim.py --grid_dim 16 --materials cube_high_E.npz
python mesh_falling_sim.py --grid_dim 16 --materials cube_low_E.npzThis simple example has two artificially constructed NPZ files which can be used in Newton. Observe the difference in simulation showing all Young's modulus, Poisson's ratio, and density values were properly applied. This requires installing newton and meshio.
Note
Our properties are also compatible with PhysX and rigid-body simulators. We plan to release some example code to do so at a later date. Until then, if you want to use our properties in PhysX, we recommend clustering the properties we produce, split the underlying meshes based on the clusters, and then add the averaged property for each such "connected part".
We are also grateful to several other open-source repositories that we drew inspiration from or built upon during the development of our pipeline:
If you find VoMP helpful, please consider citing:
@inproceedings{dagli2026vomp,
title={Vo{MP}: Predicting Volumetric Mechanical Property Fields},
author={Rishit Dagli and Donglai Xiang and Vismay Modi and Charles Loop and Clement Fuji Tsang and Anka He Chen and Anita Hu and Gavriel State and David Levin I.W. and Maria Shugrina},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=aTP1IM6alo}
}This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
VoMP source code is released under the Apache 2 License.
VoMP models are released under the NVIDIA Open Model License. For a custom license, please visit our website and submit the form: NVIDIA Research Licensing.