Repository navigation
Training
Boyuan Chen edited this page Feb 17, 2025
·
2 revisions
Your training will produce a wandb link that leads to the logs. The link ends with a wandb run id - to load or resume your trained model, simply append load={the_wandb_run_id} or resume={the_wandb_run_id} to the training / inference command. You can also load a local checkpoint, simply replace the the_wandb_run_id with a path to your checkpoint file. The option load will create a fresh wandb run with your checkpoint weights while resume will resume the existing wandb log associated with the id.
| Model | GPUs | Steps | Command |
|---|---|---|---|
| DFoT | 3x4xH100 | 500k | python -m main +name=DFoT_RE10k dataset=realestate10k algorithm=dfot_video_pose experiment=video_generation @diffusion/continuous |
| SD | 3x4xH100 | 500k | python -m main +name=SD_RE10k dataset=realestate10k algorithm=dfot_video_pose experiment=video_generation @diffusion/continuous @baseline/SD |
| FS | 3x4xH100 | 500k | python -m main +name=FS_RE10k dataset=realestate10k algorithm=dfot_video_pose experiment=video_generation @diffusion/continuous @baseline/FS |
| BD | 3x4xH100 | 500k | python -m main +name=BD_RE10k dataset=realestate10k algorithm=dfot_video_pose experiment=video_generation @diffusion/continuous @baseline/BD |
| Model | GPUs | Steps | Command |
|---|---|---|---|
| DFoT (scratch) | 3x4xH100 | 640k | python -m main +name=DFoT_K600 dataset=kinetics_600 algorithm=dfot_video experiment=video_generation @DiT/XL |
| DFoT (fine-tuned) | 3x4xH100 | 80k | python -m main +name=DFoT_K600 dataset=kinetics_600 algorithm=dfot_video experiment=video_generation @DiT/XL load=full:FS_K600.ckpt experiment.training.max_epochs=36 |
| SD | 3x4xH100 | 640k | python -m main +name=SD_K600 dataset=kinetics_600 algorithm=dfot_video experiment=video_generation @DiT/XL @baseline/SD algorithm.diffusion.loss_weighting.strategy=min_snr |
| FS | 3x4xH100 | 640k | python -m main +name=FS_K600 dataset=kinetics_600 algorithm=dfot_video experiment=video_generation @DiT/XL @baseline/FS algorithm.diffusion.loss_weighting.strategy=min_snr |
| BD | 3x4xH100 | 640k | python -m main +name=BD_K600 dataset=kinetics_600 algorithm=dfot_video experiment=video_generation @DiT/XL algorithm.noise_level=random_uniform @baseline/FS algorithm.diffusion.loss_weighting.strategy=min_snr |
| Model | GPUs | Steps | Command |
|---|---|---|---|
| DFoT | 3x4xH100 | 200k | python -m main +name=DFoT_MCRAFT dataset=minecraft algorithm=dfot_video experiment=video_generation @diffusion/continuous @DiT/B |
| Model | GPUs | Steps | Command |
|---|---|---|---|
| VideoVAE | 3x4xH100 | 600k | python -m main +name=VideoVAE_K600 dataset=kinetics_600 algorithm=video_vae experiment=video_latent_learning |
| Model | GPUs | Steps | Command |
|---|---|---|---|
| ImageVAE | 8xH100 | 50k | python -m main +name=ImageVAE_MCRAFT dataset=minecraft algorithm=image_vae experiment=video_latent_learning |
python -m main +name=Preprocessing_MCRAFT dataset=minecraft algorithm=image_vae_preprocessor experiment=video_latent_preprocessing