Q8BERT: Quantized 8Bit BERT

Abstract

Recently, pre-trained Transformer based language models such as BERT and GPT, have shown great improvement in many Natural Language Processing (NLP) tasks. However, these models contain a large amount of parameters. The emergence of even larger and more accurate models such as GPT2 and Megatron, suggest a trend of large pre-trained Transformer models. However, using these large models in production environments is a complex task requiring a large amount of compute, memory and power resources. In this work we show how to perform quantization-aware training during the fine-tuning phase of BERT in order to compress BERT by $4\times$ with minimal accuracy loss. Furthermore, the produced quantized model can accelerate inference speed if it is optimized for 8bit Integer supporting hardware.

arXiv comment5 Pages, Accepted at the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Cross-lingual Language Model Pretraining
• Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model

Cited by in corpus (56)

• A Survey on Visual Transformer
• Pre-trained Models for Natural Language Processing: A Survey
• Attention Mechanism in Neural Networks: Where it Comes and Where it Goes
• VLP: A Survey on Vision-Language Pre-training
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• A Practical Survey on Faster and Lighter Transformers
• Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
• TinyBERT: Distilling BERT for Natural Language Understanding
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• DynaBERT: Dynamic BERT with Adaptive Width and Depth
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• On the Effect of Dropping Layers of Pre-trained Transformer Models
• Structured Pruning of a BERT-based Question Answering Model
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Vision Transformer Pruning
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• BinaryBERT: Pushing the Limit of BERT Quantization
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• Efficient Quantized Sparse Matrix Operations on Tensor Cores
• STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
• A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• I-BERT: Integer-only BERT Quantization
• BEBERT: Efficient and Robust Binary Ensemble BERT
• Composite Re-Ranking for Efficient Document Search with BERT
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
• NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding
• EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference
• Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm
• Mesa: A Memory-saving Training Framework for Transformers
• Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
• An Investigation on Different Underlying Quantization Schemes for Pre-trained Language Models
• CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade
• Post-Training Quantization for Vision Transformer
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
• Teaching a Massive Open Online Course on Natural Language Processing
• Adversarial Self-Supervised Data-Free Distillation for Text Classification
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
• Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
• Exploring Low-Cost Transformer Model Compression for Large-Scale Commercial Reply Suggestions
• Deep Learning Meets Projective Clustering
• HBert + BiasCorp -- Fighting Racism on the Web
• Elbert: Fast Albert with Confidence-Window Based Early Exit
• Empirical Evaluation of Deep Learning Model Compression Techniques on the WaveNet Vocoder
• Reliable Text-to-SQL in Resource-Constrained Settings with Small LLMs via Iterative Retry Workflows