The emergence of number and syntax units in LSTM language models

Abstract

Recent work has shown that LSTMs trained on a generic language modeling objective capture syntax-sensitive generalizations such as long-distance number agreement. We have however no mechanistic understanding of how they accomplish this remarkable feat. Some have conjectured it depends on heuristics that do not truly take hierarchical structure into account. We present here a detailed study of the inner mechanics of number tracking in LSTMs at the single neuron level. We discover that long-distance number information is largely managed by two `number units'. Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure. We conclude that LSTMs are, to some extent, implementing genuinely syntactic processing mechanisms, paving the way to a more general understanding of grammatical encoding in LSTMs.

arXiv commentTo appear in Proceedings of NAACL, Minneapolis, MN, 2019

References in corpus (5)

• Visualizing and Understanding Recurrent Networks
• Learning to Generate Reviews and Discovering Sentiment
• Understanding intermediate layers using linear classifier probes
• Distinct patterns of syntactic agreement errors in recurrent networks and humans
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning

Cited by in corpus (9)

• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans
• Probing for the Usage of Grammatical Number
• Compositionality decomposed: how do neural networks generalise?
• Can RNNs learn Recursive Nested Subject-Verb Agreements?
• Learning to Perform Role-Filler Binding with Schematic Knowledge
• Word Interdependence Exposes How LSTMs Compose Representations
• Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models
• On the Pitfalls of Analyzing Individual Neurons in Language Models
• Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?