BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights

Abstract

We propose BinaryRelax, a simple two-phase algorithm, for training deep neural networks with quantized weights. The set constraint that characterizes the quantization of weights is not imposed until the late stage of training, and a sequence of \emph{pseudo} quantized weights is maintained. Specifically, we relax the hard constraint into a continuous regularizer via Moreau envelope, which turns out to be the squared Euclidean distance to the set of quantized weights. The pseudo quantized weights are obtained by linearly interpolating between the float weights and their quantizations. A continuation strategy is adopted to push the weights towards the quantized state by gradually increasing the regularization parameter. In the second phase, exact quantization scheme with a small learning rate is invoked to guarantee fully quantized weights. We test BinaryRelax on the benchmark CIFAR and ImageNet color image datasets to demonstrate the superiority of the relaxed quantization approach and the improved accuracy over the state-of-the-art training methods. Finally, we prove the convergence of BinaryRelax under an approximate orthogonality condition.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Ternary Weight Networks
• Trained Ternary Quantization
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Model compression as constrained optimization, with application to neural nets. Part II: quantization
• SEP-Nets: Small and Effective Pattern Networks
• Stochastic Layer-Wise Precision in Deep Neural Networks

Cited by in corpus (8)

• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• SYMOG: learning symmetric mixture of Gaussian modes for improved fixed-point quantization
• DNQ: Dynamic Network Quantization
• Median Binary-Connect Method and a Binary Convolutional Neural Nework for Word Recognition
• DAC: Data-free Automatic Acceleration of Convolutional Networks