Coordination Among Neural Modules Through a Shared Global Workspace

Abstract

Deep learning has seen a movement away from representing examples with a monolithic hidden state towards a richly structured state. For example, Transformers segment by position, and object-centric architectures decompose images into entities. In all these architectures, interactions between different elements are modeled via pairwise interactions: Transformers make use of self-attention to incorporate information from other positions; object-centric architectures make use of graph neural networks to model interactions among entities. However, pairwise interactions may not achieve global coordination or a coherent, integrated representation that can be used for downstream tasks. In cognitive science, a global workspace architecture has been proposed in which functionally specialized components share information through a common, bandwidth-limited communication channel. We explore the use of such a communication channel in the context of deep learning for modeling the structure of complex environments. The proposed method includes a shared workspace through which communication among different specialist modules takes place but due to limits on the communication bandwidth, specialist modules must compete for access. We show that capacity limitations have a rational basis in that (1) they encourage specialization and compositionality and (2) they facilitate the synchronization of otherwise independent specialists.

ICLR'22 accepted paper

References in corpus (10)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• Linformer: Self-Attention with Linear Complexity
• StarCraft II: A New Challenge for Reinforcement Learning
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Generating Long Sequences with Sparse Transformers
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Routing Networks and the Challenges of Modular and Compositional Computation
• S2RMs: Spatially Structured Recurrent Modules
• Transformers with Competitive Ensembles of Independent Mechanisms

Cited by in corpus (10)

• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Perceiver: General Perception with Iterative Attention
• A survey of multimodal deep generative models
• Luna: Linear Unified Nested Attention
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• Discrete-Valued Neural Communication
• Attention over learned object embeddings enables complex visual reasoning
• The Tensor Brain: A Unified Theory of Perception, Memory and Semantic Decoding
• Compositional Attention: Disentangling Search and Retrieval
• Reconciling the Discrete-Continuous Divide: Towards a Mathematical Theory of Sparse Communication