What they do when in doubt: a study of inductive biases in seq2seq learners

Abstract

Sequence-to-sequence (seq2seq) learners are widely used, but we still have only limited knowledge about what inductive biases shape the way they generalize. We address that by investigating how popular seq2seq learners generalize in tasks that have high ambiguity in the training data. We use SCAN and three new tasks to study learners' preferences for memorization, arithmetic, hierarchical, and compositional reasoning. Further, we connect to Solomonoff's theory of induction and propose to use description length as a principled and sensitive measure of inductive biases. In our experimental study, we find that LSTM-based learners can learn to perform counting, addition, and multiplication by a constant from a single training example. Furthermore, Transformer and LSTM-based learners show a bias toward the hierarchical induction over the linear one, while CNN-based learners prefer the opposite. On the SCAN dataset, we find that CNN-based, and, to a lesser degree, Transformer- and LSTM-based learners have a preference for compositional generalization over memorization. Finally, across all our experiments, description length proved to be a sensitive measure of inductive biases.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Language Models are Few-Shot Learners
• Convolutional Sequence to Sequence Learning
• A Closer Look at Memorization in Deep Networks
• Towards a Human-like Open-Domain Chatbot
• A tutorial introduction to the minimum description length principle
• Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• The Description Length of Deep Learning Models
• Human few-shot learning of compositional instructions
• Learning Inductive Biases with Simple Neural Networks
• Joint Source-Target Self Attention with Locality Constraints
• Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

Cited by in corpus (4)

• Discrete and continuous representations and processing in deep learning: Looking forward
• How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction
• How BPE Affects Memorization in Transformers
• How Can Self-Attention Networks Recognize Dyck-n Languages?