Memorize or generalize? Searching for a compositional RNN in a haystack

Abstract

Neural networks are very powerful learning systems, but they do not readily generalize from one task to the other. This is partly due to the fact that they do not learn in a compositional way, that is, by discovering skills that are shared by different tasks, and recombining them to solve new problems. In this paper, we explore the compositional generalization capabilities of recurrent neural networks (RNNs). We first propose the lookup table composition domain as a simple setup to test compositional behaviour and show that it is theoretically possible for a standard RNN to learn to behave compositionally in this domain when trained with standard gradient descent and provided with additional supervision. We then remove this additional supervision and perform a search over a large number of model initializations to investigate the proportion of RNNs that can still converge to a compositional solution. We discover that a small but non-negligible proportion of RNNs do reach partial compositional solutions even without special architectural constraints. This suggests that a combination of gradient descent and evolutionary strategies directly favouring the minority models that developed more compositional approaches might suffice to lead standard RNNs towards compositional solutions.

arXiv commentAEGAP Workshop (ICML 2018)

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Modular Multitask Reinforcement Learning with Policy Sketches
• Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks
• Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples
• Learning to Compose Neural Networks for Question Answering
• Learning to Compose Skills

Cited by in corpus (22)

• Measuring Systematic Generalization in Neural Proof Generation with Transformers
• Learning compositionally through attentive guidance
• Compositionality decomposed: how do neural networks generalise?
• Compositional Generalization for Primitive Substitutions
• On the proper role of linguistically-oriented deep net analysis in linguistic theorizing
• The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
• Assessing Phrasal Representation and Composition in Transformers
• The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
• Weight Priors for Learning Identity Relations
• On the Interplay Between Fine-tuning and Composition in Transformers
• What Meaning-Form Correlation Has to Compose With
• Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks
• On (Emergent) Systematic Generalisation and Compositionality in Visual Referential Games with Straight-Through Gumbel-Softmax Estimator
• Compositional Attention: Disentangling Search and Retrieval
• Word Interdependence Exposes How LSTMs Compose Representations
• Improving Compositional Generalization in Classification Tasks via Structure Annotations
• Self-play for Data Efficient Language Acquisition
• LSTMs Compose (and Learn) Bottom-Up
• How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction
• Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling
• On learning an interpreted language with recurrent models
• Iterative Decoding for Compositional Generalization in Transformers