DynaBERT: Dynamic BERT with Adaptive Width and Depth

Abstract

The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive. To alleviate this problem, one approach is to compress them for specific tasks before deployment. However, recent works on BERT compression usually compress the large BERT model to a fixed smaller size. They can not fully satisfy the requirements of different edge devices with various hardware performances. In this paper, we propose a novel dynamic BERT model (abbreviated as DynaBERT), which can flexibly adjust the size and latency by selecting adaptive width and depth. The training process of DynaBERT includes first training a width-adaptive BERT and then allowing both adaptive width and depth, by distilling knowledge from the full-sized model to small sub-networks. Network rewiring is also used to keep the more important attention heads and neurons shared by more sub-networks. Comprehensive experiments under various efficiency constraints demonstrate that our proposed dynamic BERT (or RoBERTa) at its largest size has comparable performance as BERT-base (or RoBERTa-base), while at smaller widths and depths consistently outperforms existing BERT compression methods. Code is available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/DynaBERT.

NeurIPS-2020 (Spotlight)

References in corpus (14)

• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Exploring the Limits of Language Modeling
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Q8BERT: Quantized 8Bit BERT
• Universal Transformers
• Reducing Transformer Depth on Demand with Structured Dropout
• Deep Equilibrium Models
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Training with Quantization Noise for Extreme Model Compression
• A Tensorized Transformer for Language Modeling
• Structured Pruning of a BERT-based Question Answering Model
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model

Cited by in corpus (9)

• Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• BEBERT: Efficient and Robust Binary Ensemble BERT
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Utility is in the Eye of the User: A Critique of NLP Leaderboards
• One Person, One Model--Learning Compound Router for Sequential Recommendation
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding
• Tailored-LLaMA: Optimizing Few-Shot Learning in Pruned LLaMA Models with Task-Specific Prompts