A Systematic Assessment of Syntactic Generalization in Neural Language Models

Abstract

While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures. Factorially manipulating model architecture and training dataset size (1M--40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments. Our results also reveal a dissociation between perplexity and syntactic generalization performance.

arXiv commentTo appear in the Proceedings of the Association for Computational Linguistics (ACL 2020)

References in corpus (5)

• Exploring the Limits of Language Modeling
• Assessing BERT's Syntactic Abilities
• What do Neural Machine Translation Models Learn about Morphology?
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency

Cited by in corpus (6)

• Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale
• StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
• Discovering Useful Sentence Representations from Large Pretrained Language Models
• Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
• Do Large Language Models know who did what to whom?
• Recurrent babbling: evaluating the acquisition of grammar from limited input data