Word Interdependence Exposes How LSTMs Compose Representations

Abstract

Recent work in NLP shows that LSTM language models capture compositional structure in language data. For a closer look at how these representations are composed hierarchically, we present a novel measure of interdependence between word meanings in an LSTM, based on their interactions at the internal gates. To explore how compositional representations arise over training, we conduct simple experiments on synthetic data, which illustrate our measure by showing how high interdependence can hurt generalization. These synthetic experiments also illustrate a specific hypothesis about how hierarchical structures are discovered over the course of training: that parent constituents rely on effective representations of their children, rather than on learning long-range relations independently. We further support this measure with experiments on English language data, where interdependence is higher for more closely syntactically linked word pairs.

References in corpus (10)

• Pointer Sentinel Mixture Models
• A mathematical theory of semantic development in deep neural networks
• An Empirical Exploration of Curriculum Learning for Neural Machine Translation
• Memorize or generalize? Searching for a compositional RNN in a haystack
• Insights on representational similarity in neural networks with canonical correlation
• The emergence of number and syntax units in LSTM language models
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
• Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
• Compositionality decomposed: how do neural networks generalise?
• h-detach: Modifying the LSTM Gradient Towards Better Optimization