ProxQuant: Quantized Neural Networks via Proximal Operators

Abstract

To make deep neural networks feasible in resource-constrained environments (such as mobile devices), it is beneficial to quantize models by using low-precision weights. One common technique for quantizing neural networks is the straight-through gradient method, which enables back-propagation through the quantization mapping. Despite its empirical success, little is understood about why the straight-through gradient method works. Building upon a novel observation that the straight-through gradient method is in fact identical to the well-known Nesterov's dual-averaging algorithm on a quantization constrained optimization problem, we propose a more principled alternative approach, called ProxQuant, that formulates quantized network training as a regularized learning problem instead and optimizes it via the prox-gradient method. ProxQuant does back-propagation on the underlying full-precision vector and applies an efficient prox-operator in between stochastic gradient steps to encourage quantizedness. For quantizing ResNets and LSTMs, ProxQuant outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization. For binary quantization, our analysis shows both theoretically and experimentally that ProxQuant is more stable than the straight-through gradient method (i.e. BinaryConnect), challenging the indispensability of the straight-through gradient method and providing a powerful alternative.

References in corpus (7)

• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Ternary Weight Networks
• Trained Ternary Quantization
• Loss-aware Binarization of Deep Networks
• Loss-aware Weight Quantization of Deep Networks
• Alternating Multi-bit Quantization for Recurrent Neural Networks

Cited by in corpus (19)

• A Survey on Visual Transformer
• Binary Neural Networks: A Survey
• Mixed Precision DNNs: All you need is a good parametrization
• BinaryBERT: Pushing the Limit of BERT Quantization
• Learning Interpretable Rules for Scalable Data Representation and Classification
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Few Shot Network Compression via Cross Distillation
• Differentiable Fine-grained Quantization for Deep Neural Network Compression
• Mirror Descent View for Neural Network Quantization
• Efficient Neural Architecture Search via Proximal Iterations
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• Learning by Turning: Neural Architecture Aware Optimisation
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• Training Quantized Neural Networks with a Full-precision Auxiliary Module
• Pruning Ternary Quantization
• Improving Network Slimming with Nonconvex Regularization
• A principled approach for generating adversarial images under non-smooth dissimilarity metrics
• FATNN: Fast and Accurate Ternary Neural Networks