Inference & Reproducing Results

By default, all checkpoints and logs are logged to the cloud automatically. Your training will produce a wandb link that leads to the logs. The link ends with a wandb run id - to load or resume your trained model, simply append load={the_wandb_run_id} or resume={the_wandb_run_id} to the training / inference command. You can also load a local checkpoint, simply replace the the_wandb_run_id with a path to your checkpoint file. The option load will create a fresh wandb run with your checkpoint weights while resume will resume the existing wandb log associated with the id.

Note: We provide inference weights by default for all models, which only contain the EMA weights. For DFoT models, we also provide full weights, which additionally contain the optimizer states and non-EMA weights—these are useful for fine-tuning or resuming training.

All pretrained models are available from the Hugging Face Model Hub, and can be automatically loaded using:

# for inference weights:
... load=pretrained:MODEL_NAME.ckpt
# for full weights:
... load=full:MODEL_NAME.ckpt

Setup: FVD on 50k videos, 5 History frames & 11 Predicted frames, (Typically) 4xGPUs

Setup: FVD, VBench (Quality, Consistency, Dynamics) on 1024 videos, 5 History frames & 59 Predicted frames, (Typically) 4xGPUs

Setup: LPIPS on 32 videos, 4 History frames & 4 Predicted frames, (Typically) 1xGPUs

Append dataset.rotation_angle.min={min_angle} dataset.rotation_angle.max={max_angle} to evaluate for a specific rotation angle range [min_angle, max_angle] (e.g. [140, 150], default: [170, 180])

Setup: FVD on 768 videos, 25 History frames & 125 Predicted frames, (Typically) 8xGPUs