DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
arXiv:2005.00697
Abstract
Transformer-based QA models use input-wide self-attention -- i.e. across both the question and the input passage -- at all layers, causing them to be slow and memory-intensive. It turns out that we can get by without input-wide self-attention at all layers, especially in the lower layers. We introduce DeFormer, a decomposed transformer, which substitutes the full self-attention with question-wide and passage-wide self-attentions in the lower layers. This allows for question-independent processing of the input text representations, which in turn enables pre-computing passage representations reducing runtime compute drastically. Furthermore, because DeFormer is largely similar to the original model, we can initialize DeFormer with the pre-training weights of a standard transformer, and directly fine-tune on the target QA dataset. We show DeFormer versions of BERT and XLNet can be used to speed up QA by over 4.3x and with simple distillation-based losses they incur only a 1% drop in accuracy. We open source the code at https://github.com/StonyBrookNLP/deformer.
ACL 2020 camera ready
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
- Convolutional neural networks with low-rank regularization
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- TinyBERT: Distilling BERT for Natural Language Understanding
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- A Tensorized Transformer for Language Modeling
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- Are Sixteen Heads Really Better than One?
- Visualizing and Understanding the Effectiveness of BERT