DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
arXiv:2004.12993
Abstract
Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. However, they are also notorious for being slow in inference, which makes them difficult to deploy in real-time applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Our approach allows samples to exit earlier without passing through the entire model. Experiments show that DeeBERT is able to save up to ~40% inference time with minimal degradation in model quality. Further analyses show different behaviors in the BERT transformer layers and also reveal their redundancy. Our work provides new ideas to efficiently apply deep transformer-based models to downstream tasks. Code is available at https://github.com/castorini/DeeBERT.
Accepted at ACL 2020
References in corpus (6)
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Reducing Transformer Depth on Demand with Structured Dropout
- TinyBERT: Distilling BERT for Natural Language Understanding
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT