Benchmarking Unsupervised Object Representations for Video Sequences

Abstract

Perceiving the world in terms of objects and tracking them through time is a crucial prerequisite for reasoning and scene understanding. Recently, several methods have been proposed for unsupervised learning of object-centric representations. However, since these models were evaluated on different downstream tasks, it remains unclear how they compare in terms of basic perceptual abilities such as detection, figure-ground segmentation and tracking of objects. To close this gap, we design a benchmark with four data sets of varying complexity and seven additional test sets featuring challenging tracking scenarios relevant for natural videos. Using this benchmark, we compare the perceptual abilities of four object-centric approaches: ViMON, a video-extension of MONet, based on recurrent spatial attention, OP3, which exploits clustering via spatial mixture models, as well as TBA and SCALOR, which use explicit factorization via spatial transformers. Our results suggest that the architectures with unconstrained latent representations learn more powerful representations in terms of object detection, segmentation and tracking than the spatial transformer based architectures. We also observe that none of the methods are able to gracefully handle the most challenging tracking scenarios despite their synthetic nature, suggesting that our benchmark may provide fruitful guidance towards learning more robust object-centric video representations.

References in corpus (14)

• Categorical Reparameterization with Gumbel-Softmax
• Recurrent Models of Visual Attention
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• On the Binding Problem in Artificial Neural Networks
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• SCALOR: Generative World Models with Scalable Object Representations
• Unsupervised Discovery of Parts, Structure, and Dynamics
• Towards causal generative scene models via competition of experts