A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers

Abstract

Weight pruning methods for deep neural networks (DNNs) have been investigated recently, but prior work in this area is mainly heuristic, iterative pruning, thereby lacking guarantees on the weight reduction ratio and convergence time. To mitigate these limitations, we present a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a nonconvex optimization problem with combinatorial constraints specifying the sparsity requirements, and then adopt the ADMM framework for systematic weight pruning. By using ADMM, the original nonconvex optimization problem is decomposed into two subproblems that are solved iteratively. One of these subproblems can be solved using stochastic gradient descent, the other can be solved analytically. Besides, our method achieves a fast convergence rate. The weight pruning results are very promising and consistently outperform the prior work. On the LeNet-5 model for the MNIST data set, we achieve 71.2 times weight reduction without accuracy loss. On the AlexNet model for the ImageNet data set, we achieve 21 times weight reduction without accuracy loss. When we focus on the convolutional layer pruning for computation reductions, we can reduce the total computation by five times compared with the prior work (achieving a total of 13.4 times weight reduction in convolutional layers). Our models and codes are released at https://github.com/KaiqiZhang/admm-pruning

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Playing Atari with Deep Reinforcement Learning
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Predicting Parameters in Deep Learning
• Learning both Weights and Connections for Efficient Neural Networks
• Compressing Neural Networks with the Hashing Trick
• Learning Structured Sparsity in Deep Neural Networks
• Dynamic Network Surgery for Efficient DNNs
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks

Cited by in corpus (83)

• Bringing AI To Edge: From Deep Learning's Perspective
• Filter Pruning by Switching to Neighboring CNNs with Good Attributes
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• HYDRA: Pruning Adversarially Robust Neural Networks
• StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs
• Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?
• Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM
• YOLObile: Real-Time Object Detection on Mobile Devices via Compression-Compilation Co-Design
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices
• Learned Threshold Pruning
• Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
• YOLoC: DeploY Large-Scale Neural Network by ROM-based Computing-in-Memory using ResiduaL Branch on a Chip
• ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning
• 26ms Inference Time for ResNet-50: Towards Real-Time Execution of all DNNs on Smartphone
• Grow and Prune Compact, Fast, and Accurate LSTMs
• Toward Extremely Low Bit and Lossless Accuracy in DNNs with Progressive ADMM
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Convolutional Neural Network Pruning with Structural Redundancy Reduction
• Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization
• DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures
• Pipeline Parallelism for Inference on Heterogeneous Edge Computing
• Energy-Efficient Transformer Inference: Optimization Strategies for Time Series Classification
• Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression
• ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers
• BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
• Rethinking Class-Discrimination Based CNN Channel Pruning
• FORMS: Fine-grained Polarized ReRAM-based In-situ Computation for Mixed-signal DNN Accelerator
• A SOT-MRAM-based Processing-In-Memory Engine for Highly Compressed DNN Implementation
• Improving Neural Network with Uniform Sparse Connectivity
• Structured Convolutions for Efficient Neural Network Design
• Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation
• ESMFL: Efficient and Secure Models for Federated Learning
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Weight Pruning via Adaptive Sparsity Loss
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Effective Model Sparsification by Scheduled Grow-and-Prune Methods
• Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via ${\ell }_1,{\ell }_0$, and transformed-${\ell }_1$ Penalties
• Transform Quantization for CNN (Convolutional Neural Network) Compression
• Filter Sketch for Network Pruning
• Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware
• Approximations in Deep Learning
• Single-shot Channel Pruning Based on Alternating Direction Method of Multipliers
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• A Unified DNN Weight Compression Framework Using Reweighted Optimization Methods
• Binary Complex Neural Network Acceleration on FPGA
• Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
• AdapMTL: Adaptive Pruning Framework for Multitask Learning Model
• High Area/Energy Efficiency RRAM CNN Accelerator with Kernel-Reordering Weight Mapping Scheme Based on Pattern Pruning
• Alternating Direction Method of Multipliers for Quantization
• Learn-Prune-Share for Lifelong Learning
• Light-weight Calibrator: a Separable Component for Unsupervised Domain Adaptation
• Real-Time Decentralized knowledge Transfer at the Edge
• SPEC2: SPECtral SParsE CNN Accelerator on FPGAs
• Content-Aware GAN Compression
• AutoScale: Optimizing Energy Efficiency of End-to-End Edge Inference under Stochastic Variance
• Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding
• E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAs
• The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics
• Autodidactic Neurosurgeon: Collaborative Deep Inference for Mobile Edge Intelligence via Online Learning
• Channel-wise pruning of neural networks with tapering resource constraint
• Evolving Transferable Neural Pruning Functions
• SCoTTi: Save Computation at Training Time with an adaptive framework
• Towards Efficient Neural Networks On-a-chip: Joint Hardware-Algorithm Approaches
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression
• Tetra-AML: Automatic Machine Learning via Tensor Networks
• RGP: Neural Network Pruning through Its Regular Graph Structure
• DSXplore: Optimizing Convolutional Neural Networks via Sliding-Channel Convolutions
• Channel Pruning Guided by Spatial and Channel Attention for DNNs in Intelligent Edge Computing
• Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning
• Joint Regularization on Activations and Weights for Efficient Neural Network Pruning
• In-Place Zero-Space Memory Protection for CNN
• Against Membership Inference Attack: Pruning is All You Need
• Improving DNN Fault Tolerance using Weight Pruning and Differential Crossbar Mapping for ReRAM-based Edge AI
• Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework
• GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity
• Unified Signal Compression Using a GAN with Iterative Latent Representation Optimization
• Weight Evolution: Improving Deep Neural Networks Training through Evolving Inferior Weight Values
• Scalable Adversarial Attack on Graph Neural Networks with Alternating Direction Method of Multipliers