Distilling Large Language Models into Tiny and Effective Students using pQRNN

Abstract

Large pre-trained multilingual models like mBERT, XLM-R achieve state of the art results on language understanding tasks. However, they are not well suited for latency critical applications on both servers and edge devices. It's important to reduce the memory and compute resources required by these models. To this end, we propose pQRNN, a projection-based embedding-free neural encoder that is tiny and effective for natural language processing tasks. Without pre-training, pQRNNs significantly outperform LSTM models with pre-trained embeddings despite being 140x smaller. With the same number of parameters, they outperform transformer baselines thereby showcasing their parameter efficiency. Additionally, we show that pQRNNs are effective student architectures for distilling large pre-trained language models. We perform careful ablations which study the effect of pQRNN parameters, data augmentation, and distillation settings. On MTOP, a challenging multilingual semantic parsing dataset, pQRNN students achieve 95.9\% of the performance of an mBERT teacher while being 350x smaller. On mATIS, a popular parsing task, pQRNN students on average are able to get to 97.1\% of the teacher while again being 350x smaller. Our strong results suggest that our approach is great for latency-sensitive applications while being able to leverage large mBERT-like models.

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Compressing Deep Convolutional Networks using Vector Quantization
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Revisiting Distributed Synchronous SGD
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Quasi-Recurrent Neural Networks
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Small and Practical BERT Models for Sequence Labeling
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Rethinking embedding coupling in pre-trained language models
• Making Neural Machine Reading Comprehension Faster

Cited by in corpus (4)

• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Transfer Learning for Multi-lingual Tasks -- a Survey
• MergeDistill: Merging Pre-trained Language Models using Distillation
• Tiny Neural Models for Seq2Seq