Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework

Abstract

Most neural network pruning methods, such as filter-level and layer-level prunings, prune the network model along one dimension (depth, width, or resolution) solely to meet a computational budget. However, such a pruning policy often leads to excessive reduction of that dimension, thus inducing a huge accuracy loss. To alleviate this issue, we argue that pruning should be conducted along three dimensions comprehensively. For this purpose, our pruning framework formulates pruning as an optimization problem. Specifically, it first casts the relationships between a certain model's accuracy and depth/width/resolution into a polynomial regression and then maximizes the polynomial to acquire the optimal values for the three dimensions. Finally, the model is pruned along the three optimal dimensions accordingly. In this framework, since collecting too much data for training the regression is very time-costly, we propose two approaches to lower the cost: 1) specializing the polynomial to ensure an accurate regression even with less training data; 2) employing iterative pruning and fine-tuning to collect the data faster. Extensive experiments show that our proposed algorithm surpasses state-of-the-art pruning algorithms and even neural architecture search-based algorithms.

arXiv comment11 pages, 5 figures. Accepted by ICML 2021

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Do Deep Nets Really Need to be Deep?
• DARTS: Differentiable Architecture Search
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Learning Efficient Convolutional Networks through Network Slimming
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• Operation-Aware Soft Channel Pruning using Differentiable Masks
• Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
• DBP: Discrimination Based Block-Level Pruning for Deep Model Acceleration
• Accelerating CNN Training by Pruning Activation Gradients

Cited by in corpus (2)

• Dynamic Resolution Network
• RGP: Neural Network Pruning through Its Regular Graph Structure