AutoPruning for Deep Neural Network with Dynamic Channel Masking

Abstract

Modern deep neural network models are large and computationally intensive. One typical solution to this issue is model pruning. However, most current pruning algorithms depend on hand crafted rules or domain expertise. To overcome this problem, we propose a learning based auto pruning algorithm for deep neural network, which is inspired by recent automatic machine learning(AutoML). A two objectives' problem that aims for the the weights and the best channels for each layer is first formulated. An alternative optimization approach is then proposed to derive the optimal channel numbers and weights simultaneously. In the process of pruning, we utilize a searchable hyperparameter, remaining ratio, to denote the number of channels in each convolution layer, and then a dynamic masking process is proposed to describe the corresponding channel evolution. To control the trade-off between the accuracy of a model and the pruning ratio of floating point operations, a novel loss function is further introduced. Preliminary experimental results on benchmark datasets demonstrate that our scheme achieves competitive results for neural network pruning.

arXiv comment9 pages

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Benchmark Analysis of Representative Deep Neural Network Architectures
• Pruning Filters for Efficient ConvNets
• Efficient Neural Architecture Search via Parameter Sharing
• Channel Pruning for Accelerating Very Deep Neural Networks
• Learning Efficient Convolutional Networks through Network Slimming
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Dynamic Network Surgery for Efficient DNNs
• Forward and Reverse Gradient-Based Hyperparameter Optimization
• Efficient Residual Dense Block Search for Image Super-Resolution
• A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications