Towards Effective Low-bitwidth Convolutional Neural Networks

Abstract

This paper tackles the problem of training a deep convolutional neural network with both low-precision weights and low-bitwidth activations. Optimizing a low-precision network is very challenging since the training process can easily get trapped in a poor local minima, which results in substantial accuracy loss. To mitigate this problem, we propose three simple-yet-effective approaches to improve the network training. First, we propose to use a two-stage optimization strategy to progressively find good local minima. Specifically, we propose to first optimize a net with quantized weights and then quantized activations. This is in contrast to the traditional methods which optimize them simultaneously. Second, following a similar spirit of the first method, we propose another progressive optimization approach which progressively decreases the bit-width from high-precision to low-precision during the course of training. Third, we adopt a novel learning scheme to jointly train a full-precision model alongside the low-precision one. By doing so, the full-precision model provides hints to guide the low-precision model training. Extensive experiments on various datasets ( i.e., CIFAR-100 and ImageNet) show the effectiveness of the proposed methods. To highlight, using our methods to train a 4-bit precision network leads to no performance decrease in comparison with its full-precision counterpart with standard network architectures ( i.e., AlexNet and ResNet-50).

arXiv comment11 pages. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2018

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Binarized Neural Networks
• Trained Ternary Quantization
• Tensorizing Neural Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks
• Convolutional neural networks with low-rank regularization
• Xception: Deep Learning with Depthwise Separable Convolutions
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Deep Mutual Learning

Cited by in corpus (5)

• Quantization for Rapid Deployment of Deep Neural Networks
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Joint Pruning & Quantization for Extremely Sparse Neural Networks
• Efficient Deep Neural Networks
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation