TernaryBERT: Distillation-aware Ultra-low Bit BERT
arXiv:2009.12812
Abstract
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices. In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the full-precision model while being 14.9x smaller.
Accepted by EMNLP 2020
References in corpus (14)
- Distilling the Knowledge in a Neural Network
- Decoupled Weight Decay Regularization
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Ternary Weight Networks
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- Q8BERT: Quantized 8Bit BERT
- Reducing Transformer Depth on Demand with Structured Dropout
- Loss-aware Binarization of Deep Networks
- Training with Quantization Noise for Extreme Model Compression
- A Tensorized Transformer for Language Modeling
- Loss-aware Weight Quantization of Deep Networks
- Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
- QKD: Quantization-aware Knowledge Distillation
- LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression