Better Set Representations For Relational Reasoning

Abstract

Incorporating relational reasoning into neural networks has greatly expanded their capabilities and scope. One defining trait of relational reasoning is that it operates on a set of entities, as opposed to standard vector representations. Existing end-to-end approaches typically extract entities from inputs by directly interpreting the latent feature representations as a set. We show that these approaches do not respect set permutational invariance and thus have fundamental representational limitations. To resolve this limitation, we propose a simple and general network module called a Set Refiner Network (SRN). We first use synthetic image experiments to demonstrate how our approach effectively decomposes objects without explicit supervision. Then, we insert our module into existing relational reasoning models and show that respecting set invariance leads to substantial gains in prediction performance and robustness on several relational reasoning tasks.

arXiv commentPreprint, 17 pages

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• Order Matters: Sequence to sequence for sets
• Relational Deep Reinforcement Learning
• Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
• Deep Set Prediction Networks
• FSPool: Learning Set Representations with Featurewise Sort Pooling
• VideoGraph: Recognizing Minutes-Long Human Activities in Videos
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Iterative Amortized Inference
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks

Cited by in corpus (3)

• Object-Centric Learning with Slot Attention
• Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks
• A Universal Model for Cross Modality Mapping by Relational Reasoning