BERT Loses Patience: Fast and Robust Inference with Early Exit

Abstract

In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM). To achieve this, our approach couples an internal-classifier with each layer of a PLM and dynamically stops inference when the intermediate predictions of the internal classifiers remain unchanged for a pre-defined number of steps. Our approach improves inference efficiency as it allows the model to make a prediction with fewer layers. Meanwhile, experimental results with an ALBERT model show that our method can improve the accuracy and robustness of the model by preventing it from overthinking and exploiting multiple classifiers for prediction, yielding a better accuracy-speed trade-off compared to existing early exit methods.

NeurIPS 2020

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Reducing Transformer Depth on Demand with Structured Dropout
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference

Cited by in corpus (9)

• Pre-trained Models for Natural Language Processing: A Survey
• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• Mobile Foundation Model as Firmware
• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
• Towards Practical Few-shot Federated NLP
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning