Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization

Abstract

Low-bit deep neural networks (DNNs) become critical for embedded applications due to their low storage requirement and computing efficiency. However, they suffer much from the non-negligible accuracy drop. This paper proposes the stochastic quantization (SQ) algorithm for learning accurate low-bit DNNs. The motivation is due to the following observation. Existing training algorithms approximate the real-valued elements/filters with low-bit representation all together in each iteration. The quantization errors may be small for some elements/filters, while are remarkable for others, which lead to inappropriate gradient direction during training, and thus bring notable accuracy drop. Instead, SQ quantizes a portion of elements/filters to low-bit with a stochastic probability inversely proportional to the quantization error, while keeping the other portion unchanged with full-precision. The quantized and full-precision portions are updated with corresponding gradients separately in each iteration. The SQ ratio is gradually increased until the whole network is quantized. This procedure can greatly compensate the quantization error and thus yield better accuracy for low-bit DNNs. Experiments show that SQ can consistently and significantly improve the accuracy for different low-bit DNNs on various datasets and various network structures.

BMVC 2017 Oral

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Fully Convolutional Networks for Semantic Segmentation
• Ternary Weight Networks
• Trained Ternary Quantization
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Convolutional Neural Networks using Logarithmic Data Representation
• Deep Networks with Stochastic Depth
• Accelerating Deep Convolutional Networks using low-precision and sparsity

Cited by in corpus (11)

• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• NICE: Noise Injection and Clamping Estimation for Neural Network Quantization
• Accurate and Compact Convolutional Neural Networks with Trained Binarization
• PyHessian: Neural Networks Through the Lens of the Hessian
• Degree-Quant: Quantization-Aware Training for Graph Neural Networks
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Composite Binary Decomposition Networks
• $S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
• A Survey on Green Deep Learning
• Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations