Unsupervised object-centric video generation and decomposition in 3D
arXiv:2007.06705
Abstract
A natural approach to generative modeling of videos is to represent them as a composition of moving objects. Recent works model a set of 2D sprites over a slowly-varying background, but without considering the underlying 3D scene that gives rise to them. We instead propose to model a video as the view seen while moving through a scene with multiple 3D objects and a 3D background. Our model is trained from monocular videos without any supervision, yet learns to generate coherent 3D scenes containing several moving objects. We conduct detailed experiments on two datasets, going beyond the visual complexity supported by state-of-the-art generative approaches. We evaluate our method on depth-prediction and 3D object detection -- tasks which cannot be addressed by those earlier works -- and show it out-performs them even on 2D instance segmentation and tracking.
arXiv commentAppeared at NeurIPS 2020. Project page: http://pmh47.net/o3v/
References in corpus (16)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- Multi-Object Representation Learning with Iterative Variational Inference
- MONet: Unsupervised Scene Decomposition and Representation
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
- SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
- Unsupervised Generative 3D Shape Learning from Natural Images
- Investigating Object Compositionality in Generative Adversarial Networks
- SCALOR: Generative World Models with Scalable Object Representations
- Unsupervised Discovery of Parts, Structure, and Dynamics
- Towards causal generative scene models via competition of experts
- R-SQAIR: Relational Sequential Attend, Infer, Repeat
- Object-Centric Image Generation with Factored Depths, Locations, and Appearances