Recurrent Independent Mechanisms

Abstract

Learning modular structures which reflect the dynamics of the environment can lead to better generalization and robustness to changes which only affect a few of the underlying causes. We propose Recurrent Independent Mechanisms (RIMs), a new recurrent architecture in which multiple groups of recurrent cells operate with nearly independent transition dynamics, communicate only sparingly through the bottleneck of attention, and are only updated at time steps where they are most relevant. We show that this leads to specialization amongst the RIMs, which in turn allows for dramatically improved generalization on tasks where some factors of variation differ systematically between training and evaluation.

References in corpus (22)

• Neural Message Passing for Quantum Chemistry
• Geometric deep learning: going beyond Euclidean data
• A Recurrent Latent Variable Model for Sequential Data
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Learning Latent Dynamics for Planning from Pixels
• Stochastic Video Generation with a Learned Prior
• A Clockwork RNN
• Understanding disentangling in $\beta$-VAE
• Hierarchical Multiscale Recurrent Neural Networks
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Tracking the World State with Recurrent Entity Networks
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• The Consciousness Prior
• Learning Independent Causal Mechanisms
• Discovering objects and their relations from entangled scene representations
• Relational Forward Models for Multi-Agent Learning
• Routing Networks and the Challenges of Modular and Compositional Computation
• Modular Networks: Learning to Decompose Neural Computation
• Variable Computation in Recurrent Neural Networks
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Learning Powerful Policies by Using Consistent Dynamics Model

Cited by in corpus (43)

• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Object-Centric Learning with Slot Attention
• A Practical Survey on Faster and Lighter Transformers
• Causality for Machine Learning
• Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data
• Is Attention Better Than Matrix Decomposition?
• On the Binding Problem in Artificial Neural Networks
• Causal Discovery in Physical Systems from Videos
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Neural Networks with Recurrent Generative Feedback
• Towards causal generative scene models via competition of experts
• Multi-Task Reinforcement Learning with Context-based Representations
• Coordination Among Neural Modules Through a Shared Global Workspace
• Generative Adversarial Transformers
• Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems
• Discrete-Valued Neural Communication
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Dynamic Inference with Neural Interpreters
• Transformers with Competitive Ensembles of Independent Mechanisms
• S2RMs: Spatially Structured Recurrent Modules
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Untangling tradeoffs between recurrence and self-attention in neural networks
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Product Kanerva Machines: Factorized Bayesian Memory
• Generalization and Robustness Implications in Object-Centric Learning
• Amortized learning of neural causal representations
• Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks
• Piecewise-constant Neural ODEs
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
• Modular Action Concept Grounding in Semantic Video Prediction
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Reconciling the Discrete-Continuous Divide: Towards a Mathematical Theory of Sparse Communication
• Compositional Attention: Disentangling Search and Retrieval
• Working Memory Graphs
• Robustifying Sequential Neural Processes
• SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks
• Zero-shot generalization using cascaded system-representations
• Concepts, Properties and an Approach for Compositional Generalization
• GFlowNet Foundations
• Interpretable UAV Collision Avoidance using Deep Reinforcement Learning
• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition
• Boxhead: A Dataset for Learning Hierarchical Representations
• Towards intervention-centric causal reasoning in learning agents