AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks

Abstract

In this paper, we develop a new algorithm, Annealed Skewed SGD - AskewSGD - for training deep neural networks (DNNs) with quantized weights. First, we formulate the training of quantized neural networks (QNNs) as a smoothed sequence of interval-constrained optimization problems. Then, we propose a new first-order stochastic method, AskewSGD, to solve each constrained optimization subproblem. Unlike algorithms with active sets and feasible directions, AskewSGD avoids projections or optimization under the entire feasible set and allows iterates that are infeasible. The numerical complexity of AskewSGD is comparable to existing approaches for training QNNs, such as the straight-through gradient estimator used in BinaryConnect, or other state of the art methods (ProxQuant, LUQ). We establish convergence guarantees for AskewSGD (under general assumptions for the objective function). Experimental results show that the AskewSGD algorithm performs better than or on par with state of the art methods in classical benchmarks.

References in corpus (14)

• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Binary Neural Networks: A Survey
• WRPN: Wide Reduced-Precision Networks
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• Training Quantized Nets: A Deeper Understanding
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• Rotated Binary Neural Network
• Techniques for Learning Binary Stochastic Feedforward Neural Networks
• Model compression as constrained optimization, with application to neural nets. Part II: quantization
• Training Binary Neural Networks using the Bayesian Learning Rule
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• Mirror Descent View for Neural Network Quantization
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats