Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines

Abstract

Deep learning as a means to inferencing has proliferated thanks to its versatility and ability to approach or exceed human-level accuracy. These computational models have seemingly insatiable appetites for computational resources not only while training, but also when deployed at scales ranging from data centers all the way down to embedded devices. As such, increasing consideration is being made to maximize the computational efficiency given limited hardware and energy resources and, as a result, inferencing with reduced precision has emerged as a viable alternative to the IEEE 754 Standard for Floating-Point Arithmetic. We propose a quantization scheme that allows inferencing to be carried out using arithmetic that is fundamentally more efficient when compared to even half-precision floating-point. Our quantization procedure is significant in that we determine our quantization scheme parameters by calibrating against its reference floating-point model using a single inference batch rather than (re)training and achieve end-to-end post quantization accuracies comparable to the reference model.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Deep Residual Learning for Image Recognition
• Going Deeper with Convolutions
• Training deep neural networks with low precision multiplications
• Hardware-oriented Approximation of Convolutional Neural Networks

Cited by in corpus (7)

• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Fighting Quantization Bias With Bias
• Rethinking Machine Learning Development and Deployment for Edge Devices
• Phoenix: A Low-Precision Floating-Point Quantization Oriented Architecture for Convolutional Neural Networks
• Composite Binary Decomposition Networks
• Low Precision Floating-point Arithmetic for High Performance FPGA-based CNN Acceleration