Differentiable Model Compression via Pseudo Quantization Noise
arXiv:2104.09987
Abstract
We propose DiffQ a differentiable method for model compression for quantizing model parameters without gradient approximations (e.g., Straight Through Estimator). We suggest adding independent pseudo quantization noise to model parameters during training to approximate the effect of a quantization operator. DiffQ is differentiable both with respect to the unquantized weights and the number of bits used. Given a single hyper-parameter balancing between the quantized model size and accuracy, DiffQ optimizes the number of bits used per individual weight or groups of weights, in end-to-end training. We experimentally verify that our method is competitive with STE based quantization techniques on several benchmarks and architectures for image classification, language modeling, and audio source separation. For instance, on the ImageNet dataset, DiffQ compresses a 12 layers transformer-based model by more than a factor of 8, (lower than 4 bits precision per weight on average), with a loss of 0.3% in model accuracy. Code is available at github.com/facebookresearch/diffq.
arXiv commentfinal TMLR version
References in corpus (18)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Decoupled Weight Decay Regularization
- Categorical Reparameterization with Gumbel-Softmax
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Wide Residual Networks
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- Mixed Precision Training
- Quantizing deep convolutional networks for efficient inference: A whitepaper
- Ternary Weight Networks
- Training and Inference with Integers in Deep Neural Networks
- Reducing Transformer Depth on Demand with Structured Dropout
- Music Source Separation in the Waveform Domain
- WRPN: Wide Reduced-Precision Networks
- Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
- Mixed Precision DNNs: All you need is a good parametrization
- NICE: Noise Injection and Clamping Estimation for Neural Network Quantization