WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
arXiv:1912.06638
Abstract
Transformer based Very Large Language Models (VLLMs) like BERT, XLNet and RoBERTa, have recently shown tremendous performance on a large variety of Natural Language Understanding (NLU) tasks. However, due to their size, these VLLMs are extremely resource intensive and cumbersome to deploy at production time. Several recent publications have looked into various ways to distil knowledge from a transformer based VLLM (most commonly BERT-Base) into a smaller model which can run much faster at inference time. Here, we propose a novel set of techniques which together produce a task-specific hybrid convolutional and transformer model, WaLDORf, that achieves state-of-the-art inference speed while still being more accurate than previous distilled models.
arXiv commentAdded Figure, minor edits for clarity
References in corpus (23)
- Distilling the Knowledge in a Neural Network
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Effective Approaches to Attention-based Neural Machine Translation
- Fixed Point Quantization of Deep Convolutional Networks
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Long Short-Term Memory-Networks for Machine Reading
- Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
- ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Massive Exploration of Neural Machine Translation Architectures
- Large Memory Layers with Product Keys
- Single Headed Attention RNN: Stop Thinking With Your Head
- Are Sixteen Heads Really Better than One?
- Probing Neural Network Comprehension of Natural Language Arguments
- What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
- AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
- Retraining-Based Iterative Weight Quantization for Deep Neural Networks
- Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering