Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights

Abstract

This paper presents incremental network quantization (INQ), a novel method, targeting to efficiently convert any pre-trained full-precision convolutional neural network (CNN) model into a low-precision version whose weights are constrained to be either powers of two or zero. Unlike existing methods which are struggled in noticeable accuracy loss, our INQ has the potential to resolve this issue, as benefiting from two innovations. On one hand, we introduce three interdependent operations, namely weight partition, group-wise quantization and re-training. A well-proven measure is employed to divide the weights in each layer of a pre-trained CNN model into two disjoint groups. The weights in the first group are responsible to form a low-precision base, thus they are quantized by a variable-length encoding method. The weights in the other group are responsible to compensate for the accuracy loss from the quantization, thus they are the ones to be re-trained. On the other hand, these three operations are repeated on the latest re-trained group in an iterative manner until all the weights are converted into low-precision ones, acting as an incremental network quantization and accuracy enhancement procedure. Extensive experiments on the ImageNet classification task using almost all known deep CNN architectures including AlexNet, VGG-16, GoogleNet and ResNets well testify the efficacy of the proposed method. Specifically, at 5-bit quantization, our models have improved accuracy than the 32-bit floating-point references. Taking ResNet-18 as an example, we further show that our quantized models with 4-bit, 3-bit and 2-bit ternary weights have improved or very similar accuracy against its 32-bit floating-point baseline. Besides, impressive results with the combination of network pruning and INQ are also reported. The code is available at https://github.com/Zhouaojun/Incremental-Network-Quantization.

arXiv commentPublished by ICLR 2017, and the code is available at https://github.com/Zhouaojun/Incremental-Network-Quantization

Cited by in corpus (155)

• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Binary Neural Networks: A Survey
• Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
• Training and Inference with Integers in Deep Neural Networks
• Real-Time Apple Detection System Using Embedded Systems With Hardware Accelerators: An Edge AI Application
• A Survey on Methods and Theories of Quantized Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Bringing AI To Edge: From Deep Learning's Perspective
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Discrimination-aware Network Pruning for Deep Model Compression
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference
• Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• Layer-specific Optimization for Mixed Data Flow with Mixed Precision in FPGA Design for CNN-based Object Detectors
• Applications and Techniques for Fast Machine Learning in Science
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• Optimizing Speech Recognition For The Edge
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition
• An Experimental Study of Reduced-Voltage Operation in Modern FPGAs for Neural Network Acceleration
• Mixed Precision DNNs: All you need is a good parametrization
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Distilling Object Detectors with Task Adaptive Regularization
• A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM
• Improved training of binary networks for human pose estimation and image recognition
• From Hashing to CNNs: Training BinaryWeight Networks via Hashing
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• NICE: Noise Injection and Clamping Estimation for Neural Network Quantization
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks
• Compressing Word Embeddings via Deep Compositional Code Learning
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
• Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM
• Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
• Learned Threshold Pruning
• Self-Binarizing Networks
• Towards Effective Low-bitwidth Convolutional Neural Networks
• Towards Unified INT8 Training for Convolutional Neural Network
• Fighting Quantization Bias With Bias
• Anchor Pruning for Object Detection
• On the Adversarial Robustness of Quantized Neural Networks
• LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Relaxed Quantization for Discretized Neural Networks
• 26ms Inference Time for ResNet-50: Towards Real-Time Execution of all DNNs on Smartphone
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• KTAN: Knowledge Transfer Adversarial Network
• BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• Additive Noise Annealing and Approximation Properties of Quantized Neural Networks
• Knowledge Distillation for End-to-End Person Search
• SBNet: Sparse Blocks Network for Fast Inference
• Toward Extremely Low Bit and Lossless Accuracy in DNNs with Progressive ADMM
• Trained Rank Pruning for Efficient Deep Neural Networks
• Knowledge Squeezed Adversarial Network Compression
• Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network
• Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
• Bit Efficient Quantization for Deep Neural Networks
• Joint Pruning & Quantization for Extremely Sparse Neural Networks
• Sparse Weight Activation Training
• On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method
• BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights
• Accelerating Deep Unsupervised Domain Adaptation with Transfer Channel Pruning
• Channel-wise Hessian Aware trace-Weighted Quantization of Neural Networks
• SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
• DNNVM : End-to-End Compiler Leveraging Heterogeneous Optimizations on FPGA-based CNN Accelerators
• SYMOG: learning symmetric mixture of Gaussian modes for improved fixed-point quantization
• Term Revealing: Furthering Quantization at Run Time on Quantized DNNs
• Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks
• MWQ: Multiscale Wavelet Quantized Neural Networks
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Learning Filter Basis for Convolutional Neural Network Compression
• Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
• Ternary Residual Networks
• End-to-End Learned Image Compression with Quantized Weights and Activations
• DARC: Differentiable ARchitecture Compression
• Learning compositional functions via multiplicative weight updates
• A Pre-defined Sparse Kernel Based Convolution for Deep CNNs
• Dynamic Runtime Feature Map Pruning
• Pufferfish: Communication-efficient Models At No Extra Cost
• RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• GBCNs: Genetic Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Auto-tuning Neural Network Quantization Framework for Collaborative Inference Between the Cloud and Edge
• A Unified DNN Weight Compression Framework Using Reweighted Optimization Methods
• Quantization of Deep Neural Networks for Accurate Edge Computing
• AUSN: Approximately Uniform Quantization by Adaptively Superimposing Non-uniform Distribution for Deep Neural Networks
• BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization
• Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA
• AnalogNet: Convolutional Neural Network Inference on Analog Focal Plane Sensor Processors
• n-hot: Efficient bit-level sparsity for powers-of-two neural network quantization
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR
• One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment
• Light-weight Calibrator: a Separable Component for Unsupervised Domain Adaptation
• OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators
• DNQ: Dynamic Network Quantization
• Brain-inspired reverse adversarial examples
• Table-Based Neural Units: Fully Quantizing Networks for Multiply-Free Inference
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
• Generative Design of Hardware-aware DNNs
• HAO: Hardware-aware neural Architecture Optimization for Efficient Inference
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
• Learning Multimodal Fixed-Point Weights using Gradient Descent
• Localization-aware Channel Pruning for Object Detection
• GDRQ: Group-based Distribution Reshaping for Quantization
• A Very Compact Embedded CNN Processor Design Based on Logarithmic Computing
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• CoDeNet: Efficient Deployment of Input-Adaptive Object Detection on Embedded FPGAs
• Pruning Ternary Quantization
• Gator: Customizable Channel Pruning of Neural Networks with Gating
• Quantization Mimic: Towards Very Tiny CNN for Object Detection
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• End-to-end Learned Image Compression with Fixed Point Weight Quantization
• PBGen: Partial Binarization of Deconvolution-Based Generators for Edge Intelligence
• Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
• Addressing Limited Weight Resolution in a Fully Optical Neuromorphic Reservoir Computing Readout
• Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
• CompConv: A Compact Convolution Module for Efficient Feature Learning
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks
• Neural Network Activation Quantization with Bitwise Information Bottlenecks
• Hybrid and Non-Uniform quantization methods using retro synthesis data for efficient inference
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• A study on speech enhancement using exponent-only floating point quantized neural network (EOFP-QNN)
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression
• ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition
• DNN Quantization with Attention
• Approximate Random Dropout
• Conditional Neural Architecture Search
• $S^2$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning
• Neural Network Training with Approximate Logarithmic Computations
• Full-stack Optimization for Accelerating CNNs with FPGA Validation
• $S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
• Penetrating the Fog: the Path to Efficient CNN Models
• A Targeted Acceleration and Compression Framework for Low bit Neural Networks
• Smoothed Differential Privacy
• MoRS: An Approximate Fault Modelling Framework for Reduced-Voltage SRAMs
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
• Kernel Quantization for Efficient Network Compression
• Joint Coreset Construction and Quantization for Distributed Machine Learning