WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
arXiv:1912.06638
Abstract
Transformer based Very Large Language Models (VLLMs) like BERT, XLNet and RoBERTa, have recently shown tremendous performance on a large variety of Natural Language Understanding (NLU) tasks. However, due to their size, these VLLMs are extremely resource intensive and cumbersome to deploy at production time. Several recent publications have looked into various ways to distil knowledge from a transformer based VLLM (most commonly BERT-Base) into a smaller model which can run much faster at inference time. Here, we propose a novel set of techniques which together produce a task-specific hybrid convolutional and transformer model, WaLDORf, that achieves state-of-the-art inference speed while still being more accurate than previous distilled models.
Added Figure, minor edits for clarity
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Massive Exploration of Neural Machine Translation Architectures
- Single Headed Attention RNN: Stop Thinking With Your Head
- What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
- Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering