Network Automatic Pruning: Start NAP and Take a Nap

Abstract

Network pruning can significantly reduce the computation and memory footprint of large neural networks. To achieve a good trade-off between model size and performance, popular pruning techniques usually rely on hand-crafted heuristics and require manually setting the compression ratio for each layer. This process is typically time-consuming and requires expert knowledge to achieve good results. In this paper, we propose NAP, a unified and automatic pruning framework for both fine-grained and structured pruning. It can find out unimportant components of a network and automatically decide appropriate compression ratios for different layers, based on a theoretically sound criterion. Towards this goal, NAP uses an efficient approximation of the Hessian for evaluating the importances of components, based on a Kronecker-factored Approximate Curvature method. Despite its simpleness to use, NAP outperforms previous pruning methods by large margins. For fine-grained pruning, NAP can compress AlexNet and VGG16 by 25x, and ResNet-50 by 6.7x without loss in accuracy on ImageNet. For structured pruning (e.g. channel pruning), it can reduce flops of VGG16 by 5.4x and ResNet-50 by 2.3x with only 1% accuracy drop. More importantly, this method is almost free from hyper-parameter tuning and requires no expert knowledge. You can start NAP and then take a nap!

arXiv commentAn updated version of 'MLPrune: Multi-Layer Pruning for Automated Neural Network Compression'

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Playing Atari with Deep Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Compressing Deep Convolutional Networks using Vector Quantization
• Trained Ternary Quantization
• Pruning Filters for Efficient ConvNets
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference
• Noisy Natural Gradient as Variational Inference