Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
arXiv:2006.14727
Abstract
Unsupervised multi-object scene decomposition is a fast-emerging problem in representation learning. Despite significant progress in static scenes, such models are unable to leverage important dynamic cues present in video. We propose a novel spatio-temporal iterative inference framework that is powerful enough to jointly model complex multi-object representations and explicit temporal dependencies between latent variables across frames. This is achieved by leveraging 2D-LSTM, temporally conditioned inference and generation within the iterative amortized inference for posterior refinement. Our method improves the overall quality of decompositions, encodes information about the objects' dynamics, and can be used to predict trajectories of each object separately. Additionally, we show that our model has a high accuracy even without color information. We demonstrate the decomposition, segmentation, and prediction capabilities of our model and show that it outperforms the state-of-the-art on several benchmark datasets, one of which was curated for this work and will be made publicly available.
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Natural Scales in Geographical Patterns
- YOLOv3: An Incremental Improvement
- A Recurrent Latent Variable Model for Sequential Data
- Hierarchical Multiscale Recurrent Neural Networks
- Multi-Object Representation Learning with Iterative Variational Inference
- MONet: Unsupervised Scene Decomposition and Representation
- VideoBERT: A Joint Model for Video and Language Representation Learning
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
- Inference Suboptimality in Variational Autoencoders
- Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
- Iterative Amortized Inference
- SCALOR: Generative World Models with Scalable Object Representations
- Reinforced active learning for image segmentation