SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
arXiv:2106.03849
Abstract
To help agents reason about scenes in terms of their building blocks, we wish to extract the compositional structure of any given scene (in particular, the configuration and characteristics of objects comprising the scene). This problem is especially difficult when scene structure needs to be inferred while also estimating the agent's location/viewpoint, as the two variables jointly give rise to the agent's observations. We present an unsupervised variational approach to this problem. Leveraging the shared structure that exists across different scenes, our model learns to infer two sets of latent representations from RGB video input alone: a set of "object" latents, corresponding to the time-invariant, object-level contents of the scene, as well as a set of "frame" latents, corresponding to global time-varying elements such as viewpoint. This factorization of latents allows our model, SIMONe, to represent object attributes in an allocentric manner which does not depend on viewpoint. Moreover, it allows us to disentangle object dynamics and summarize their trajectories as time-abstracted, view-invariant, per-object properties. We demonstrate these capabilities, as well as the model's performance in terms of view synthesis and instance segmentation, across three procedurally generated video datasets.
arXiv commentAnimated figures are available at https://sites.google.com/view/simone-scene-understanding/
References in corpus (24)
- An Introduction to Variational Autoencoders
- Is Space-Time Attention All You Need for Video Understanding?
- Decomposing Motion and Content for Natural Video Sequence Prediction
- NeRF--: Neural Radiance Fields Without Known Camera Parameters
- Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
- Multi-Object Representation Learning with Iterative Variational Inference
- MONet: Unsupervised Scene Decomposition and Representation
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
- Disentangled Sequential Autoencoder
- Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
- End-to-End Video Instance Segmentation with Transformers
- Hierarchically Learned View-Invariant Representations for Cross-View Action Recognition
- pixelNeRF: Neural Radiance Fields from One or Few Images
- Imitating Interactive Intelligence
- Grounded Language Learning Fast and Slow
- Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
- Entity Abstraction in Visual Model-Based Reinforcement Learning
- NeRF-VAE: A Geometry Aware 3D Scene Generative Model
- View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose
- Neural Radiance Flow for 4D View Synthesis and Video Processing
- Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
- Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction
- BARF: Bundle-Adjusting Neural Radiance Fields