On the Binding Problem in Artificial Neural Networks

Abstract

Contemporary neural networks still fall short of human-level generalization, which extends far beyond our direct experiences. In this paper, we argue that the underlying cause for this shortcoming is their inability to dynamically and flexibly bind information that is distributed throughout the network. This binding problem affects their capacity to acquire a compositional understanding of the world in terms of symbol-like entities (like objects), which is crucial for generalizing in predictable and systematic ways. To address this issue, we propose a unifying framework that revolves around forming meaningful entities from unstructured sensory inputs (segregation), maintaining this separation of information at a representational level (representation), and using these entities to construct new inferences, predictions, and behaviors (composition). Our analysis draws inspiration from a wealth of research in neuroscience and cognitive psychology, and surveys relevant mechanisms from the machine learning literature, to help identify a combination of inductive biases that allow symbolic information processing to emerge naturally in neural networks. We believe that a compositional approach to AI, in terms of grounded symbol-like representations, is of fundamental importance for realizing human-level generalization, and we hope that this paper may contribute towards that goal as a reference and inspiration.

References in corpus (63)

• Efficient Estimation of Word Representations in Vector Space
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Neural Message Passing for Quantum Chemistry
• Geometric deep learning: going beyond Euclidean data
• How transferable are features in deep neural networks?
• Axiomatic Attribution for Deep Networks
• Deep Convolutional Networks on Graph-Structured Data
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Dota 2 with Large Scale Deep Reinforcement Learning
• Recurrent Models of Visual Attention
• Visualizing and Understanding Recurrent Networks
• Image Segmentation in Video Sequences: A Probabilistic Approach
• Multiple Object Recognition with Visual Attention
• End-To-End Memory Networks
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Interaction Networks for Learning about Objects, Relations and Physics
• Towards a Definition of Disentangled Representations
• FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation
• A Study on Overfitting in Deep Reinforcement Learning
• MONet: Unsupervised Scene Decomposition and Representation
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• Efficient Graph Generation with Graph Recurrent Attention Networks
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Using Fast Weights to Attend to the Recent Past
• Tracking the World State with Recurrent Entity Networks
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
• Measuring abstract reasoning in neural networks
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Causality for Machine Learning
• The Consciousness Prior
• Structured Disentangled Representations
• Unsupervised Learning of Object Landmarks through Conditional Image Generation
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Learning Generative Models with Visual Attention
• Learning visual groups from co-occurrences in space and time
• Recurrent Independent Mechanisms
• Relational inductive bias for physical construction in humans and machines
• Dimensions of Neural-symbolic Integration - A Structured Survey
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• Learning to generalize to new compositions in image understanding
• Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs
• Independently Controllable Factors
• Object Discovery with a Copy-Pasting GAN
• Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
• Investigating Object Compositionality in Generative Adversarial Networks
• Stochastic Prediction of Multi-Agent Interactions from Partial Observations
• SCALOR: Generative World Models with Scalable Object Representations
• Learning to Reason with Third-Order Tensor Products
• Towards causal generative scene models via competition of experts
• Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views
• Neural Random-Access Machines
• Concept Learning with Energy-Based Models
• Disentangled Representations in Neural Models
• Convolutional Bipartite Attractor Networks
• End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning
• Continuous Graph Flow
• State-Denoised Recurrent Neural Networks

Cited by in corpus (22)

• Zero-Shot Text-to-Image Generation
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Understanding the computational demands underlying visual reasoning
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
• Computing on Functions Using Randomized Vector Representations
• Generalization and Robustness Implications in Object-Centric Learning
• Benchmarking Unsupervised Object Representations for Video Sequences
• The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
• SLASH: Embracing Probabilistic Circuits into Neural Answer Set Programming
• Unsupervised Object Keypoint Learning using Local Spatial Predictability
• Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Modular Object-Oriented Games: A Task Framework for Reinforcement Learning, Psychology, and Neuroscience
• The Role of General Intelligence in Mathematical Reasoning
• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition
• The Impact of Negative Sampling on Contrastive Structured World Models
• Learning with Holographic Reduced Representations
• Continual learning under domain transfer with sparse synaptic bursting
• Conditional Object-Centric Learning from Video
• Illiterate DALL-E Learns to Compose