Language-Mediated, Object-Centric Representation Learning
arXiv:2012.15814
Abstract
We present Language-mediated, Object-centric Representation Learning (LORL), a paradigm for learning disentangled, object-centric scene representations from vision and language. LORL builds upon recent advances in unsupervised object discovery and segmentation, notably MONet and Slot Attention. While these algorithms learn an object-centric representation just by reconstructing the input image, LORL enables them to further learn to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input. These object-centric concepts derived from language facilitate the learning of object-centric representations. LORL can be integrated with various unsupervised object discovery algorithms that are language-agnostic. Experiments show that the integration of LORL consistently improves the performance of unsupervised object discovery methods on two datasets via the help of language. We also show that concepts learned by LORL, in conjunction with object discovery methods, aid downstream tasks such as referring expression comprehension.
arXiv commentACL 2021 Findings. First two authors contributed equally; last two authors contributed equally. Project page: https://lang-orl.github.io/
References in corpus (11)
- Natural Scales in Geographical Patterns
- Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
- Multi-Object Representation Learning with Iterative Variational Inference
- MONet: Unsupervised Scene Decomposition and Representation
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
- SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
- Unsupervised Video Object Segmentation for Deep Reinforcement Learning
- Learning Physical Graph Representations from Visual Scenes
- Learning Representations from Audio-Visual Spatial Alignment
- Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems