Pruning and Quantization for Deep Neural Network Acceleration: A Survey
arXiv:2101.09671
Abstract
Deep neural networks have been applied in many applications exhibiting extraordinary abilities in the field of computer vision. However, complex network architectures challenge efficient real-time deployment and require significant computation resources and energy costs. These challenges can be overcome through optimizations such as network compression. Network compression can often be realized with little loss of accuracy. In some cases accuracy may even improve. This paper provides a survey on two types of network compression: pruning and quantization. Pruning can be categorized as static if it is performed offline or dynamic if it is performed at run-time. We compare pruning techniques and describe criteria used to remove redundant computations. We discuss trade-offs in element-wise, channel-wise, shape-wise, filter-wise, layer-wise and even network-wise pruning. Quantization reduces computations by reducing the precision of the datatype. Weights, biases, and activations may be quantized typically to 8-bit integers although lower bit width implementations are also discussed including binary neural networks. Both pruning and quantization can be used independently or combined. We compare current techniques, analyze their strengths and weaknesses, present compressed network accuracy results on a number of frameworks, and provide practical guidance for compressing networks.
References in corpus (44)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Improving neural networks by preventing co-adaptation of feature detectors
- Language Models are Few-Shot Learners
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- Compressing Deep Convolutional Networks using Vector Quantization
- Mixed Precision Training
- Rethinking the Value of Network Pruning
- Benchmark Analysis of Representative Deep Neural Network Architectures
- Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
- Quantizing deep convolutional networks for efficient inference: A whitepaper
- Trained Ternary Quantization
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- Compressing Neural Networks with the Hashing Trick
- Binary Neural Networks: A Survey
- The State of Sparsity in Deep Neural Networks
- Adaptive Computation Time for Recurrent Neural Networks
- Training and Inference with Integers in Deep Neural Networks
- Training deep neural networks with low precision multiplications
- Multi-Scale Dense Networks for Resource Efficient Image Classification
- Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
- Training Deep Neural Networks with 8-bit Floating Point Numbers
- A Survey on Methods and Theories of Quantized Neural Networks
- Scalable Methods for 8-bit Training of Neural Networks
- L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
- DSD: Dense-Sparse-Dense Training for Deep Neural Networks
- A Survey of FPGA-Based Neural Network Accelerator
- Mixed Precision Training of Convolutional Neural Networks using Integer Operations
- Highway and Residual Networks learn Unrolled Iterative Estimation
- Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
- Dissecting the Graphcore IPU Architecture via Microbenchmarking
- Estimating or Propagating Gradients Through Stochastic Neurons
- Ternary Neural Networks with Fine-Grained Quantization
- Diversity Networks: Neural Network Compression Using Determinantal Point Processes
- Rounding Methods for Neural Networks with Low Resolution Synaptic Weights
- Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation
- From Hashing to CNNs: Training BinaryWeight Networks via Hashing
- Cnvlutin2: Ineffectual-Activation-and-Weight-Free Deep Neural Network Computing
- Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines
- Low-Precision Batch-Normalized Activations
- The State of Knowledge Distillation for Classification
- Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks