AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search

Abstract

Large pre-trained language models such as BERT have shown their effectiveness in various natural language processing tasks. However, the huge parameter size makes them difficult to be deployed in real-time applications that require quick inference with limited resources. Existing methods compress BERT into small models while such compression is task-independent, i.e., the same compressed BERT for all different downstream tasks. Motivated by the necessity and benefits of task-oriented BERT compression, we propose a novel compression method, AdaBERT, that leverages differentiable Neural Architecture Search to automatically compress BERT into task-adaptive small models for specific tasks. We incorporate a task-oriented knowledge distillation loss to provide search hints and an efficiency-aware loss as search constraints, which enables a good trade-off between efficiency and effectiveness for task-adaptive BERT compression. We evaluate AdaBERT on several NLP tasks, and the results demonstrate that those task-adaptive compressed models are 12.7x to 29.3x faster than BERT in inference time and 11.5x to 17.0x smaller in terms of parameter size, while comparable performance is maintained.

arXiv commentaccepted by IJCAI 2020

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DARTS: Differentiable Architecture Search
• A guide to convolution arithmetic for deep learning
• Convolutional Neural Networks for Sentence Classification
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Q8BERT: Quantized 8Bit BERT
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• A Tensorized Transformer for Language Modeling
• Soft Weight-Sharing for Neural Network Compression
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Cited by in corpus (12)

• Training with Quantization Noise for Extreme Model Compression
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads
• LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• Differentiable NAS Framework and Application to Ads CTR Prediction
• AutoADR: Automatic Model Design for Ad Relevance
• Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation
• Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search
• GiBERT: Introducing Linguistic Knowledge into BERT through a Lightweight Gated Injection Method