SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
arXiv:2106.03849
Abstract
To help agents reason about scenes in terms of their building blocks, we wish to extract the compositional structure of any given scene (in particular, the configuration and characteristics of objects comprising the scene). This problem is especially difficult when scene structure needs to be inferred while also estimating the agent's location/viewpoint, as the two variables jointly give rise to the agent's observations. We present an unsupervised variational approach to this problem. Leveraging the shared structure that exists across different scenes, our model learns to infer two sets of latent representations from RGB video input alone: a set of "object" latents, corresponding to the time-invariant, object-level contents of the scene, as well as a set of "frame" latents, corresponding to global time-varying elements such as viewpoint. This factorization of latents allows our model, SIMONe, to represent object attributes in an allocentric manner which does not depend on viewpoint. Moreover, it allows us to disentangle object dynamics and summarize their trajectories as time-abstracted, view-invariant, per-object properties. We demonstrate these capabilities, as well as the model's performance in terms of view synthesis and instance segmentation, across three procedurally generated video datasets.
Animated figures are available at https://sites.google.com/view/simone-scene-understanding/
References in corpus (15)
- Is Space-Time Attention All You Need for Video Understanding?
- Decomposing Motion and Content for Natural Video Sequence Prediction
- NeRF--: Neural Radiance Fields Without Known Camera Parameters
- MONet: Unsupervised Scene Decomposition and Representation
- End-to-End Video Instance Segmentation with Transformers
- pixelNeRF: Neural Radiance Fields from One or Few Images
- Imitating Interactive Intelligence
- Grounded Language Learning Fast and Slow
- Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
- NeRF-VAE: A Geometry Aware 3D Scene Generative Model
- View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose
- Neural Radiance Flow for 4D View Synthesis and Video Processing
- Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
- Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction
- BARF: Bundle-Adjusting Neural Radiance Fields