BinaryBERT: Pushing the Limit of BERT Quantization

Abstract

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks.

References in corpus (24)

• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• DARTS: Differentiable Architecture Search
• Binarized Neural Networks
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Ternary Weight Networks
• Q8BERT: Quantized 8Bit BERT
• Universal Transformers
• Learned Step Size Quantization
• Reducing Transformer Depth on Demand with Structured Dropout
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Loss-aware Binarization of Deep Networks
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Training with Quantization Noise for Extreme Model Compression
• A Tensorized Transformer for Language Modeling
• Loss-aware Weight Quantization of Deep Networks
• ProxQuant: Quantized Neural Networks via Proximal Operators
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Splitting Steepest Descent for Growing Neural Architectures
• Few Shot Network Compression via Cross Distillation
• Efficient Bitwidth Search for Practical Mixed Precision Neural Network
• GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

Cited by in corpus (11)

• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• A comprehensive review of Binary Neural Network
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• I-BERT: Integer-only BERT Quantization
• Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
• NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
• Mesa: A Memory-saving Training Framework for Transformers
• Post-Training Quantization for Vision Transformer
• On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
• BERMo: What can BERT learn from ELMo?
• A Survey on Green Deep Learning