Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations

Abstract

This paper tackles the problem of training a deep convolutional neural network of both low-bitwidth weights and activations. Optimizing a low-precision network is very challenging due to the non-differentiability of the quantizer, which may result in substantial accuracy loss. To address this, we propose three practical approaches, including (i) progressive quantization; (ii) stochastic precision; and (iii) joint knowledge distillation to improve the network training. First, for progressive quantization, we propose two schemes to progressively find good local minima. Specifically, we propose to first optimize a net with quantized weights and subsequently quantize activations. This is in contrast to the traditional methods which optimize them simultaneously. Furthermore, we propose a second progressive quantization scheme which gradually decreases the bit-width from high-precision to low-precision during training. Second, to alleviate the excessive training burden due to the multi-round training stages, we further propose a one-stage stochastic precision strategy to randomly sample and quantize sub-networks while keeping other parts in full-precision. Finally, we adopt a novel learning scheme to jointly train a full-precision model alongside the low-precision one. By doing so, the full-precision model provides hints to guide the low-precision model training and significantly improves the performance of the low-precision network. Extensive experiments on various datasets (e.g., CIFAR-100, ImageNet) show the effectiveness of the proposed methods.

arXiv commentAccepted to IEEE T. Pattern Analysis and Machine Intelligence (TPAMI). Extended version of arXiv:1711.00205 (CVPR 2018)

References in corpus (29)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• DARTS: Differentiable Architecture Search
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Binarized Neural Networks
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Tensorizing Neural Networks
• Convolutional neural networks with low-rank regularization
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Dynamic Channel Pruning: Feature Boosting and Suppression
• Loss-aware Binarization of Deep Networks
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Loss-aware Weight Quantization of Deep Networks
• ProxQuant: Quantized Neural Networks via Proximal Operators
• Training Competitive Binary Neural Networks from Scratch
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
• Knowledge Adaptation for Efficient Semantic Segmentation
• Learning to Train a Binary Neural Network
• Precision Highway for Ultra Low-Precision Quantization

Cited by in corpus (5)

• Review: Deep Learning in Electron Microscopy
• Towards Efficient Training for Neural Network Quantization
• Generative Low-bitwidth Data Free Quantization
• Training Quantized Neural Networks with a Full-precision Auxiliary Module
• Knowledge Distillation-aided End-to-End Learning for Linear Precoding in Multiuser MIMO Downlink Systems with Finite-Rate Feedback