Paper | Project Page | Demo | Poster
DIMO: Diverse 3D Motion Generation for Arbitrary Objects
Linzhan Mou, Jiahui Lei, Chen Wang, Lingjie Liu, Kostas Daniilidis
University of Pennsylvania
ICCV 2025 (Highlight)
- [2026-01-04] Code and data are pre-released!
- [2025-07-24] DIMO is selected as Highlight Paper!
- [2025-06-26] DIMO is accepted by ICCV 2025! 🎉 We will release code in this repo.
We use Python 3.10 with PyTorch 2.1.1 and CUDA 11.8. The environment and packages can be installed as follows:
git clone --recursive https://github.com/Friedrich-M/DIMO.git && cd DIMO
conda create -y -n dimo -c nvidia/label/cuda-11.8.0 -c defaults cuda-toolkit=11.8 cuda-compiler=11.8 cudnn=8 python=3.10
conda activate dimo
pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt --no-build-isolation
pip install --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py310_cu118_pyt211/download.html
pip install submodules/diff-gauss submodules/diff-gaussian-rasterization --no-build-isolationTwo rasterizers are needed: diff-gauss renders depth and normals (used when add_normal=True, the default) and diff-gaussian-rasterization is the plain fallback. Nearest-neighbour queries use PyTorch3D.
Build and offline-cache notes (clusters, air-gapped compute nodes)
requirements.txt pins setuptools<70 because the extension builder of PyTorch 2.1.1 still imports pkg_resources. When compiling the rasterizers on a machine without a GPU (e.g. a cluster login node), set the target architectures explicitly, e.g. TORCH_CUDA_ARCH_LIST="8.0;9.0" for A100/H100.
Three model weights are downloaded on first use, which fails on compute nodes without outbound internet access. Fetch them once from a machine that has it, then point the code at the cache:
python -c "import lpips; lpips.LPIPS(net='vgg')" # LPIPS VGG, into the package dir
python -c "from rembg import new_session; new_session('u2net')" # rembg matting, into ~/.u2net
python -c "from transformers import AutoModel, AutoTokenizer; [c.from_pretrained('bert-base-cased', cache_dir='ckpts/hf_cache') for c in (AutoTokenizer, AutoModel)]"Then pass bert_cache_dir=ckpts/hf_cache to test.py mode=language and projector.bert_cache_dir=ckpts/hf_cache to train_text_projector.py. Also export HF_HUB_OFFLINE=1 on the compute node: with a cache directory set, transformers still contacts the hub first and spends about half a minute in retries before falling back to it. rembg is pinned to u2net; override with $DIMO_REMBG_MODEL if you cached a different one.
train.py / test.py entry points (config = configs/default.yaml + key=value overrides)
configs/default.yaml every option, grouped by stage and commented
scripts/ example train / test commands
dimo/
config.py YAML + `key=value` overrides (unknown keys are rejected)
data.py multi-view video loading, mask caching (rembg / white background)
cameras.py orbit cameras and the rasterizer camera
trainer.py two-stage training loop
tester.py rendering, trajectories and the latent-space applications
models/
deform_net.py latent-conditioned motion decoder (position + rotation)
embedding.py positional encoding for the decoder inputs
latent.py per-motion latent codes (plain or Gaussian / VAE)
gaussian_model.py canonical Gaussians, key points, densification, checkpoints
renderer.py stage-1 / stage-2 deformation and rasterisation
text_encoder.py BERT -> latent projector for language-guided generation
losses/ image losses, depth / normal smoothness, ARAP, KL, Chamfer
utils/ I/O, trajectory visualisation, math helpers
data_generation/ single image -> training data (see its own README)
Intuition: distill rich motion priors from video models as diverse motion capture.
data_generation/ implements the full pipeline of Sec. 3.1 of the paper: from a single image it writes the exact folder layout train.py expects, in five resumable stages (captions, image-to-video, filtering, multi-view lifting, dataset assembly).
cd data_generation && bash scripts/run_pipeline.sh trump /path/trump.pngSee data_generation/README.md for the stages, the environments, the checkpoints to download and all options.
You can skip this step and download our processed example data (51 Trump motions) from Google Drive:
mkdir data && cd data && gdown 1b0_2t_KKhOyKlJsYncUcQm6URecAS6M6 && tar -zxvf data_trump_n51_step20.tar.gz && cd ..A data folder contains one sub-folder per motion with view_XX/FF.png frames, plus an info.json listing the view azimuths, elevations and motion names. info.json may be omitted if you pass input_videos=<comma-separated motions> and the config's num_views / num_frames match the folder, in which case the azimuths are assumed uniform; the datasets built by data_generation/ always include it.
Foreground masks, and how to make them 30x faster
Masks are computed on first use and cached next to the frames as FF_mask.npy, so only the first run pays for them; the loader spreads the work over num_workers processes.
mask_method chooses how. The default rembg runs the u2net matting network, which is general but costs about 0.2 s per frame — roughly 27 minutes for a 51 x 9 x 21 dataset on one core, or under 2 minutes on 16. Since every frame this pipeline produces sits on a pure white background, mask_method=white_bg instead derives the alpha from the distance to white, which is about 30x faster (a full dataset in under a minute on one core) and keeps white parts of the object, because it only removes white that is connected to the image border. It agrees with rembg to a median IoU of 0.97; the difference is at soft edges, where white_bg keeps a slightly wider matte. Use it for renders on white, and keep rembg for photographs or any other background. If you precompute masks with data_generation/build_dataset.py, set the matching dataset.mask_method there so the cached masks are the ones training would have produced.
Intuition: jointly model diverse 3D motions in a shared latent space. To train DIMO, simply run:
bash scripts/train.sh # or: python train.py input_folder=data/trump_n51_step20 save_path=outputs/trump_n51_step20 ...- NOTE: All hyperparameters live in
configs/default.yamland can be overridden on the command line askey=value;scripts/train.shshows the settings used for the example data. - NOTE: Set
vae_latent=Trueto enforce a Gaussian distribution on the motion latent code, which also enables the KL divergence loss during training. - NOTE: Training runs in two stages (
s1: motion pre-training on key points,s2: joint refinement). Passload_stage=s1to resume from a saved stage-1 checkpoint. Checkpoints and tensorboard logs are written tosave_path.
You can also skip training and download our pre-trained model from Google Drive for testing:
mkdir ckpts && cd ckpts && gdown 1-a9JxXvoGRV_qy5ontRShc4mgDgVkrsd && tar -zxvf ckpt_trump_n51_step20.tar.gz && cd ..Once trained, you can perform 4d rendering and visualize key point trajectories by running:
bash scripts/test.sh # or: python test.py mode=render input_folder=... save_path=... video_save_dir=...- NOTE: Choose which motions to render with
render_videos=11-walk(comma-separated list); passrender_videos=nullto render all motions (it may take some time). For every motion you get the reference-view video (*_ref.mp4), an orbit video (*_orbit.mp4), the key-point trajectories overlaid on the render (*_blend.mp4), a 3D trajectory animation (*_traj_3d.mp4) and trajectory images;render_views=Trueadditionally renders all training views.
The rendered key point trajectories will look like this (Trump is walking):
key_point_3d_trajectory.mp4
The 4d rendering results should look like this (reference, fixed view, orbit views):
4d_rendering.mp4
- NOTE: Since the video models we use for motion prior distillation were not perfect at that time, the generated videos may contain artifacts. We will update the code and models with more advanced video models like Veo3 and SV4D2.0 in the future.
If you have any questions, please feel free to open an issue or email at linzhan@princeton.edu.
With the learned motion latent space, the same test.py entry point exposes the following applications through the mode option (bash scripts/test.sh <mode> [key=value ...]). We also provide some visualization results below.
They all render through the key-point motion model, so they need a stage-2 checkpoint (test_stage=s2, the default in scripts/test.sh).
- Latent Space Motion Interpolation
mode=interpolation interp_videos=[04-032041,11-raise] interp_alpha=0.5
Blends the two motions' latent codes and renders the result. interp_alpha also takes a list, and the default [0.0, 0.5, 1.0] renders both endpoints and the midpoint and additionally writes them side by side as intp_<a>_<b>_sweep_frames.mp4, so the path through the latent space can be read at a glance. Values outside [0, 1] extrapolate past an endpoint.
interpolation.mp4
- Language-Guided Motion Generation
mode=language test_text_prompt="Trump is walking"
Maps a phrase to a latent code with a small BERT-to-latent projector. The released checkpoint does not include a projector, so train one first — a couple of minutes on CPU:
python data_generation/caption_dataset.py dataset_dir=data/trump_n51_step20 # only if captions.json is missing
python data_generation/train_text_projector.py --config data_generation/configs/default.yaml \
object.name=trump_n51_step20 dataset.output_dir=data \
projector.checkpoint_dir=ckpts/trump_n51_step20/s2That writes ckpts/trump_n51_step20/s2/mlp_encoder.pth, which is exactly where mode=language looks by default (<save_path>/<test_stage>/mlp_encoder.pth), so no extra argument is needed; set text_encoder_ckpt= to point elsewhere.
The projector pairs each motion's short phrase from captions.json with that motion's latent code, using the motion_order.json the training run wrote so the pairing cannot silently shift. The first command above captions a dataset that arrived without them (such as the released Trump data) by showing a strip of frames per motion to a vision-language model.
language.mp4
- Test Motion Reconstruction
mode=fit_motion test_motion_data=<folder with view_XX/FF.png of the new motion> fits a new latent code to an unseen multi-view video, keeping the geometry and the motion decoder fixed, so it recovers the motion only insofar as the learned latent space already spans it.
mode=fit_unaligned_motion handles a new motion whose key-point layout does not match the training one, in two phases: first the latent code and the translation head are fitted on a key-point-only model, then the latent code and the whole motion decoder are refined on the full model.
test_motion.mp4
Differences from the pre-release scripts
The losses, schedules, deformation math and checkpoint format are unchanged: the released checkpoints load and render bit-identically. Four behavioural differences are worth knowing if you compare against the original scripts:
- Key-point downsampling was fixed. The original farthest-point step passed indices where a boolean mask was expected, and
~on an integer tensor is a bitwise NOT, so it kept the mirror of the selected set. It now keeps the points it selects. Camera.projectwas fixed. Normalised device coordinates must scale by width in x and height in y; the original used height and width the other way round. Inert for the square renders used throughout, wrong for non-square ones.mode=fit_unaligned_motionphase 1 uses the trained key points. The original re-initialised the key-point cloud to 512 random points just before fitting against it, so this mode's output will not match the original's.- Rendering is deterministic and runs under
no_grad. Withvae_latent=Truethe original sampled the latent at test time as well as during training; it now returns the mean. Themode=benchmarkFPS is therefore also higher than the original's, which built the autograd graph every iteration, so the two numbers are not comparable.
Cached ground-truth frames and masks are stored as uint8 rather than float32, which is what lets the 51 x 9 x 21 released dataset stay in host memory; the resulting per-pixel error on the loss targets is at most 1/510.
The code in this repository is released under the MIT License. Note that the Gaussian rasterizers in submodules/ derive from gaussian-splatting, which Inria and MPII license for non-commercial research and evaluation only, and that the video and matting models each carry their own terms. See LICENSE for the full list and check them before any commercial use.
Our code is built on top of DreamGaussian, CogVideoX, SV4D. Many thanks to the authors for sharing their code. We also greatly appreciate the help from Yiming Xie.
If you find this paper useful for your research, please consider citing:
@inproceedings{mou2025dimo,
title={DIMO: Diverse 3D Motion Generation for Arbitrary Objects},
author={Mou, Linzhan and Lei, Jiahui and Wang, Chen and Liu, Lingjie and Daniilidis, Kostas},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={14357--14368},
year={2025}
}
