LSTMs Compose (and Learn) Bottom-Up
arXiv:2010.04650
Abstract
Recent work in NLP shows that LSTM language models capture hierarchical structure in language data. In contrast to existing work, we consider the \textit{learning} process that leads to their compositional behavior. For a closer look at how an LSTM's sequential representations are composed hierarchically, we present a related measure of Decompositional Interdependence (DI) between word meanings in an LSTM, based on their gate interactions. We connect this measure to syntax with experiments on English language data, where DI is higher on pairs of words with lower syntactic distance. To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data. These synthetic experiments support a specific hypothesis about how hierarchical structures are discovered over the course of training: that LSTM constituent representations are learned bottom-up, relying on effective representations of their shorter children, rather than learning the longer-range relations independently from children.
arXiv commentPublished in EMNLP Findings 2020. arXiv admin note: substantial text overlap with arXiv:2004.13195
References in corpus (11)
- Transition-Based Dependency Parsing with Stack Long Short-Term Memory
- Pointer Sentinel Mixture Models
- A mathematical theory of semantic development in deep neural networks
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- An Empirical Exploration of Curriculum Learning for Neural Machine Translation
- Memorize or generalize? Searching for a compositional RNN in a haystack
- Information-theoretic analysis of multivariate single - cell signaling responses using SLEMI
- Boosting Neural Machine Translation
- Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
- Transferring Inductive Biases through Knowledge Distillation
- h-detach: Modifying the LSTM Gradient Towards Better Optimization