AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates

Abstract

Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference. An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters. This work proposes AutoCompress, an automatic structured pruning framework with the following key performance improvements: (i) effectively incorporate the combination of structured pruning schemes in the automatic process; (ii) adopt the state-of-art ADMM-based structured weight pruning as the core algorithm, and propose an innovative additional purification step for further weight reduction without accuracy loss; and (iii) develop effective heuristic search method enhanced by experience-based guided search, replacing the prior deep reinforcement learning technique which has underlying incompatibility with the target pruning problem. Extensive experiments on CIFAR-10 and ImageNet datasets demonstrate that AutoCompress is the key to achieve ultra-high pruning rates on the number of weights and FLOPs that cannot be achieved before. As an example, AutoCompress outperforms the prior work on automatic model compression by up to 33x in pruning rate (120x reduction in the actual parameter count) under the same accuracy. Significant inference speedup has been observed from the AutoCompress framework on actual measurements on smartphone. We release all models of this work at anonymous link: http://bit.ly/2VZ63dS.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Neural Architecture Search with Reinforcement Learning
• Understanding deep learning requires rethinking generalization
• Rethinking the Value of Network Pruning
• Large-Scale Evolution of Image Classifiers
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Learning Structured Sparsity in Deep Neural Networks
• Deep Exploration via Bootstrapped DQN
• Deep Reinforcement Learning that Matters
• An Entropy-based Pruning Method for CNN Compression
• Dynamic Network Surgery for Efficient DNNs
• Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• Progressive Weight Pruning of Deep Neural Networks using ADMM

Cited by in corpus (6)

• Physics-informed learning of governing equations from scarce data
• A Programmable Approach to Neural Network Compression
• EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning
• DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation
• RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices
• Towards Real-Time DNN Inference on Mobile Platforms with Model Pruning and Compiler Optimization