Efficient Bitwidth Search for Practical Mixed Precision Neural Network

Abstract

Network quantization has rapidly become one of the most widely used methods to compress and accelerate deep neural networks. Recent efforts propose to quantize weights and activations from different layers with different precision to improve the overall performance. However, it is challenging to find the optimal bitwidth (i.e., precision) for weights and activations of each layer efficiently. Meanwhile, it is yet unclear how to perform convolution for weights and activations of different precision efficiently on generic hardware platforms. To resolve these two issues, in this paper, we first propose an Efficient Bitwidth Search (EBS) algorithm, which reuses the meta weights for different quantization bitwidth and thus the strength for each candidate precision can be optimized directly w.r.t the objective without superfluous copies, reducing both the memory and computational cost significantly. Second, we propose a binary decomposition algorithm that converts weights and activations of different precision into binary matrices to make the mixed precision convolution efficient and practical. Experiment results on CIFAR10 and ImageNet datasets demonstrate our mixed precision QNN outperforms the handcrafted uniform bitwidth counterparts and other mixed precision techniques.

arXiv comment21 pages, 7 figures

References in corpus (14)

• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Ternary Weight Networks
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Efficient Neural Architecture Search via Parameter Sharing
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• WRPN: Wide Reduced-Precision Networks
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision

Cited by in corpus (3)

• BinaryBERT: Pushing the Limit of BERT Quantization
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks