Training with Quantization Noise for Extreme Model Compression

Abstract

We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator. In this paper, we extend this approach to work beyond int8 fixed-point quantization with extreme compression methods where the approximations introduced by STE are severe, such as Product Quantization. Our proposal is to only quantize a different random subset of weights during each forward, allowing for unbiased gradients to flow through the other weights. Controlling the amount of noise and its form allows for extreme compression rates while maintaining the performance of the original model. As a result we establish new state-of-the-art compromises between accuracy and model size both in natural language processing and image classification. For example, applying our method to state-of-the-art Transformer and ConvNet architectures, we can achieve 82.5% accuracy on MNLI by compressing RoBERTa to 14MB and 80.0 top-1 accuracy on ImageNet by compressing an EfficientNet-B3 to 3.3MB.

References in corpus (28)

• Distilling the Knowledge in a Neural Network
• Deep Residual Learning for Image Recognition
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Cross-lingual Language Model Pretraining
• Compressing Deep Convolutional Networks using Vector Quantization
• Rethinking the Value of Network Pruning
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Pruning Filters for Efficient ConvNets
• Pointer Sentinel Mixture Models
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Variational Dropout Sparsifies Deep Neural Networks
• Quasi-Recurrent Neural Networks
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Reducing Transformer Depth on Demand with Structured Dropout
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Learning Sparse Neural Networks through $L_0$ Regularization
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• A Tensorized Transformer for Language Modeling
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• Adaptive Input Representations for Neural Language Modeling
• Compressive Transformers for Long-Range Sequence Modelling
• Augmenting Self-attention with Persistent Memory
• Training wide residual networks for deployment using a single bit for each weight
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• Accelerating Neural Transformer via an Average Attention Network

Cited by in corpus (29)

• Linformer: Self-Attention with Linear Complexity
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• A Practical Survey on Faster and Lighter Transformers
• DynaBERT: Dynamic BERT with Adaptive Width and Depth
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• TransReID: Transformer-based Object Re-Identification
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• BinaryBERT: Pushing the Limit of BERT Quantization
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• Towards Lossless ANN-SNN Conversion under Ultra-Low Latency with Dual-Phase Optimization
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• I-BERT: Integer-only BERT Quantization
• Mixed-Precision Embedding Using a Cache
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
• Immersive Video Compression using Implicit Neural Representations
• Adapting by Pruning: A Case Study on BERT
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• AnalogNets: ML-HW Co-Design of Noise-robust TinyML Models and Always-On Analog Compute-in-Memory Accelerator
• Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision
• Quantized Non-Volatile Nanomagnetic Synapse based Autoencoder for Efficient Unsupervised Network Anomaly Detection
• Post-Training Quantization for Vision Transformer
• FrostNet: Towards Quantization-Aware Network Architecture Search
• General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
• Training of mixed-signal optical convolutional neural network with reduced quantization level
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
• Filter Pre-Pruning for Improved Fine-tuning of Quantized Deep Neural Networks