Learning Object-Centric Video Models by Contrasting Sets

Abstract

Contrastive, self-supervised learning of object representations recently emerged as an attractive alternative to reconstruction-based training. Prior approaches focus on contrasting individual object representations (slots) against one another. However, a fundamental problem with this approach is that the overall contrastive loss is the same for (i) representing a different object in each slot, as it is for (ii) (re-)representing the same object in all slots. Thus, this objective does not inherently push towards the emergence of object-centric representations in the slots. We address this problem by introducing a global, set-based contrastive loss: instead of contrasting individual slot representations against one another, we aggregate the representations and contrast the joined sets against one another. Additionally, we introduce attention-based encoders to this contrastive setup which simplifies training and provides interpretable object masks. Our results on two synthetic video datasets suggest that this approach compares favorably against previous contrastive methods in terms of reconstruction, future prediction and object separation performance.

NeurIPS 2020 Workshop on Object Representations for Learning and Reasoning

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• A Simple Framework for Contrastive Learning of Visual Representations
• Natural Scales in Geographical Patterns
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Object-Centric Learning with Slot Attention
• MONet: Unsupervised Scene Decomposition and Representation
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Contrastive Learning of Structured World Models
• CLEVRER: CoLlision Events for Video REpresentation and Reasoning
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Stochastic Prediction of Multi-Agent Interactions from Partial Observations
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning

Cited by in corpus (4)

• A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning
• Generalization and Robustness Implications in Object-Centric Learning
• Reasoning-Modulated Representations
• The Impact of Negative Sampling on Contrastive Structured World Models