Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Abstract

Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use a Hessian based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most $2.3\%$ performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to $13\times$ compression of the model parameters, and up to $4\times$ compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.

References in corpus (21)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Do Deep Nets Really Need to be Deep?
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Ternary Weight Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Pruning Filters for Efficient ConvNets
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• A Tensorized Transformer for Language Modeling
• Alternating Multi-bit Quantization for Recurrent Neural Networks
• Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
• Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

Cited by in corpus (21)

• TinyBERT: Distilling BERT for Natural Language Understanding
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• On the Effect of Dropping Layers of Pre-trained Transformer Models
• HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Reweighted Proximal Pruning for Large-Scale Language Representation
• PyHessian: Neural Networks Through the Lens of the Hessian
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• ZeroQ: A Novel Zero Shot Quantization Framework
• DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• Learned Low Precision Graph Neural Networks
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
• BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• Attention improves concentration when learning node embeddings
• Emergent Properties of Finetuned Language Representation Models
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• CoDeNet: Efficient Deployment of Input-Adaptive Object Detection on Embedded FPGAs