Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
arXiv:2008.13128
Abstract
Quantized Neural Networks (QNNs) use low bit-width fixed-point numbers for representing weight parameters and activations, and are often used in real-world applications due to their saving of computation resources and reproducibility of results. Batch Normalization (BN) poses a challenge for QNNs for requiring floating points in reciprocal operations, and previous QNNs either require computing BN at high precision or revise BN to some variants in heuristic ways. In this work, we propose a novel method to quantize BN by converting an affine transformation of two floating points to a fixed-point operation with shared quantized scale, which is friendly for hardware acceleration and model deployment. We confirm that our method maintains same outputs through rigorous theoretical analysis and numerical analysis. Accuracy and efficiency of our quantization method are verified by experiments at layer level on CIFAR and ImageNet datasets. We also believe that our method is potentially useful in other problems involving quantization.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Machine Translation by Jointly Learning to Align and Translate
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Instance Normalization: The Missing Ingredient for Fast Stylization
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Quantizing deep convolutional networks for efficient inference: A whitepaper
- Fixed Point Quantization of Deep Convolutional Networks
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Training and Inference with Integers in Deep Neural Networks
- WRPN: Wide Reduced-Precision Networks
- Neural Networks with Few Multiplications
- Norm matters: efficient and accurate normalization schemes in deep networks