Dynamic Inference with Neural Interpreters

Abstract

Modern neural network architectures can leverage large amounts of data to generalize well within the training distribution. However, they are less capable of systematic generalization to data drawn from unseen but related distributions, a feat that is hypothesized to require compositional reasoning and reuse of knowledge. In this work, we present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules, which we call \emph{functions}. Inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. The proposed architecture can flexibly compose computation along width and depth, and lends itself well to capacity extension after training. To demonstrate the versatility of Neural Interpreters, we evaluate it in two distinct settings: image classification and visual abstract reasoning on Raven Progressive Matrices. In the former, we show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner. In the latter, we find that Neural Interpreters are competitive with respect to the state-of-the-art in terms of systematic generalization

arXiv commentNeurIPS 2021

References in corpus (21)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• The Loss Surfaces of Multilayer Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• On the Variance of the Adaptive Learning Rate and Beyond
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Learning Independent Causal Mechanisms
• Recurrent Independent Mechanisms
• CLEVRER: CoLlision Events for Video REpresentation and Reasoning
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• Automatically Composing Representation Transformations as a Means for Generalization
• Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations
• Routing Networks and the Challenges of Modular and Compositional Computation
• Modular Networks: Learning to Decompose Neural Computation
• The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning
• Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems
• Transformers with Competitive Ensembles of Independent Mechanisms
• Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference
• Few-shot Sequence Learning with Transformers

Cited by in corpus (5)

• Assaying Out-Of-Distribution Generalization in Transfer Learning
• A Review of Emerging Research Directions in Abstract Visual Reasoning
• Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices
• Modularity in Deep Learning: A Survey
• Learning Visual Abstract Reasoning through Dual-Stream Networks