Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers

Abstract

We present a systematic weight pruning framework of deep neural networks (DNNs) using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a constrained nonconvex optimization problem, and then adopt the ADMM framework for systematic weight pruning. We show that ADMM is highly suitable for weight pruning due to the computational efficiency it offers. We achieve a much higher compression ratio compared with prior work while maintaining the same test accuracy, together with a faster convergence rate. Our models are released at https://github.com/KaiqiZhang/admm-pruning

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Learning Structured Sparsity in Deep Neural Networks
• Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks

Cited by in corpus (8)

• SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency
• RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition
• NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration
• Achieving Real-Time LiDAR 3D Object Detection on a Mobile Device
• Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Search
• RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices
• Channel-wise pruning of neural networks with tapering resource constraint
• A Privacy-Preserving-Oriented DNN Pruning and Mobile Acceleration Framework