Deep Learning with Low Precision by Half-wave Gaussian Quantization

Abstract

The problem of quantizing the activations of a deep neural network is considered. An examination of the popular binary quantization approach shows that this consists of approximating a classical non-linearity, the hyperbolic tangent, by two functions: a piecewise constant sign function, which is used in feedforward network computations, and a piecewise linear hard tanh function, used in the backpropagation step during network learning. The problem of approximating the ReLU non-linearity, widely used in the recent deep learning literature, is then considered. An half-wave Gaussian quantizer (HWGQ) is proposed for forward approximation and shown to have efficient implementation, by exploiting the statistics of of network activations and batch normalization operations commonly used in the literature. To overcome the problem of gradient mismatch, due to the use of different forward and backward approximations, several piece-wise backward approximators are then investigated. The implementation of the resulting quantized network, denoted as HWGQ-Net, is shown to achieve much closer performance to full precision networks, such as AlexNet, ResNet, GoogLeNet and VGG-Net, than previously available low-precision networks, with 1-bit binary weights and 2-bit quantized activations.

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• On the difficulty of training Recurrent Neural Networks
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Deeply-Supervised Nets
• Compressing Deep Convolutional Networks using Vector Quantization
• Ternary Weight Networks
• Predicting Parameters in Deep Learning
• Fixed Point Quantization of Deep Convolutional Networks
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Neural Networks with Few Multiplications
• Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks

Cited by in corpus (36)

• A Survey on Methods and Theories of Quantized Neural Networks
• Rethinking floating point for deep learning
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• Training Binary Neural Networks through Learning with Noisy Supervision
• FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks
• Streamlined Deployment for Quantized Neural Networks
• MARS: Multi-macro Architecture SRAM CIM-Based Accelerator with Co-designed Compressed Neural Networks
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Towards Unified INT8 Training for Convolutional Neural Network
• Relaxed Quantization for Discretized Neural Networks
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• SEP-Nets: Small and Effective Pattern Networks
• Full deep neural network training on a pruned weight budget
• BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights
• CodeX: Bit-Flexible Encoding for Streaming-based FPGA Acceleration of DNNs
• Automating Generation of Low Precision Deep Learning Operators
• Condensation-Net: Memory-Efficient Network Architecture with Cross-Channel Pooling Layers and Virtual Feature Maps
• QuTiBench: Benchmarking Neural Networks on Heterogeneous Hardware
• Knowledge distillation for optimization of quantized deep neural networks
• Progressive Learning of Low-Precision Networks
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Efficient Crowd Counting via Structured Knowledge Transfer
• Balanced Binary Neural Networks with Gated Residual
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• Efficient and Robust Machine Learning for Real-World Systems
• Scaling Neural Network Performance through Customized Hardware Architectures on Reconfigurable Logic
• BABO: Background Activation Black-Out for Efficient Object Detection
• PBGen: Partial Binarization of Deconvolution-Based Generators for Edge Intelligence
• Full-stack Optimization for Accelerating CNNs with FPGA Validation
• Modulated binary cliquenet
• Accuracy to Throughput Trade-offs for Reduced Precision Neural Networks on Reconfigurable Logic
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks
• BitSplit-Net: Multi-bit Deep Neural Network with Bitwise Activation Function