BERT Loses Patience: Fast and Robust Inference with Early Exit

Abstract

In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM). To achieve this, our approach couples an internal-classifier with each layer of a PLM and dynamically stops inference when the intermediate predictions of the internal classifiers remain unchanged for a pre-defined number of steps. Our approach improves inference efficiency as it allows the model to make a prediction with fewer layers. Meanwhile, experimental results with an ALBERT model show that our method can improve the accuracy and robustness of the model by preventing it from overthinking and exploiting multiple classifiers for prediction, yielding a better accuracy-speed trade-off compared to existing early exit methods.

arXiv commentNeurIPS 2020

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Ensemble Adversarial Training: Attacks and Defenses
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Universal Transformers
• Adaptive Computation Time for Recurrent Neural Networks
• To Trust Or Not To Trust A Classifier
• Reducing Transformer Depth on Demand with Structured Dropout
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference

Cited by in corpus (9)

• Pre-trained Models for Natural Language Processing: A Survey
• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• Mobile Foundation Model as Firmware
• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
• Towards Practical Few-shot Federated NLP
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning