Scalable Syntax-Aware Language Models Using Knowledge Distillation

Abstract

Prior work has shown that, on small amounts of training data, syntactic neural language models learn structurally sensitive generalisations more successfully than sequential language models. However, their computational complexity renders scaling difficult, and it remains an open question whether structural biases are still necessary when sequential models have access to ever larger amounts of training data. To answer this question, we introduce an efficient knowledge distillation (KD) technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM language model, hence enabling the LSTM to develop a more structurally sensitive representation of the larger training data it learns from. On targeted syntactic evaluations, we find that, while sequential LSTMs perform much better than previously reported, our proposed technique substantially improves on this baseline, yielding a new state of the art. Our findings and analysis affirm the importance of structural biases, even in models that learn from large amounts of data.

ACL 2019

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• Universal Transformers
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• DyNet: The Dynamic Neural Network Toolkit
• Assessing BERT's Syntactic Abilities
• What do Neural Machine Translation Models Learn about Morphology?
• A Bayesian Model for Generative Transition-based Dependency Parsing

Cited by in corpus (2)

• Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?
• Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach