KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
arXiv:2101.05938
Abstract
Recently, transformer-based language models such as BERT have shown tremendous performance improvement for a range of natural language processing tasks. However, these language models usually are computation expensive and memory intensive during inference. As a result, it is difficult to deploy them on resource-restricted devices. To improve the inference performance, as well as reduce the model size while maintaining the model accuracy, we propose a novel quantization method named KDLSQ-BERT that combines knowledge distillation (KD) with learned step size quantization (LSQ) for language model quantization. The main idea of our method is that the KD technique is leveraged to transfer the knowledge from a "teacher" model to a "student" model when exploiting LSQ to quantize that "student" model during the quantization training process. Extensive experiment results on GLUE benchmark and SQuAD demonstrate that our proposed KDLSQ-BERT not only performs effectively when doing different bit (e.g. 2-bit 8-bit) quantization, but also outperforms the existing BERT quantization methods, and even achieves comparable performance as the full-precision base-line model while obtaining 14.9x compression ratio. Our code will be public available.
References in corpus (23)
- Distilling the Knowledge in a Neural Network
- FitNets: Hints for Thin Deep Nets
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Quantizing deep convolutional networks for efficient inference: A whitepaper
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- Q8BERT: Quantized 8Bit BERT
- Learned Step Size Quantization
- Reducing Transformer Depth on Demand with Structured Dropout
- Model compression via distillation and quantization
- DynaBERT: Dynamic BERT with Adaptive Width and Depth
- Training with Quantization Noise for Extreme Model Compression
- Sequence-Level Knowledge Distillation
- A Tensorized Transformer for Language Modeling
- Structured Pruning of a BERT-based Question Answering Model
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- And the Bit Goes Down: Revisiting the Quantization of Neural Networks
- Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
- QKD: Quantization-aware Knowledge Distillation
- LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
- Attention-Guided Answer Distillation for Machine Reading Comprehension
- GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference