Alexander Swerdlow

tl;dr: By generating a set of dynamic scenes in Blender, we can get ground-truth 3D point tracks and train a dust3r-style model (typically used for static 3d-reconstruction) to take in two consecutive frames, and predict a static depth prediction in one frame and a set of 3d deltas in the second frame, enabling tracking and reconstruction of dynamic scenes.

tl;dr: Given an unlabeled image collection, we can condition a diffusion model on a set of segmented region features taken from a off-the-shelf encoder & segmentation model. By augmenting the denoising target and using contrastive losses, these compressed input features become a bottleneck that allows for positional and semantic control for individual objects in a given image.