Gradient ${\ell }_1$ Regularization for Quantization Robustness

Abstract

We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-demand to different bit-widths as energy and memory requirements of the application change. Unlike quantization-aware training using the straight-through estimator that only targets a specific bit-width and requires access to training data and pipeline, our regularization-based method paves the way for "on the fly'' post-training quantization to various bit-widths. We show that by modeling quantization as a ${\ell }_{\infty }$-bounded perturbation, the first-order term in the loss expansion can be regularized using the ${\ell }_1$-norm of gradients. We experimentally validate the effectiveness of our regularization scheme on different architectures on CIFAR-10 and ImageNet datasets.

ICLR 2020

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Post-training 4-bit quantization of convolution networks for rapid-deployment
• Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
• Sparse DNNs with Improved Adversarial Robustness
• Robust Learning with Jacobian Regularization
• Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

Cited by in corpus (4)

• Degree-Quant: Quantization-Aware Training for Graph Neural Networks
• BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Do All MobileNets Quantize Poorly? Gaining Insights into the Effect of Quantization on Depthwise Separable Convolutional Networks Through the Eyes of Multi-scale Distributional Dynamics