Global Sparse Momentum SGD for Pruning Very Deep Neural Networks

Abstract

Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices. DNN pruning is an approach for deep model compression, which aims at eliminating some parameters with tolerable performance degradation. In this paper, we propose a novel momentum-SGD-based optimization method to reduce the network complexity by on-the-fly pruning. Concretely, given a global compression ratio, we categorize all the parameters into two parts at each training iteration which are updated using different rules. In this way, we gradually zero out the redundant parameters, as we update them using only the ordinary weight decay but no gradients derived from the objective function. As a departure from prior methods that require heavy human works to tune the layer-wise sparsity ratios, prune by solving complicated non-differentiable problems or finetune the model after pruning, our method is characterized by 1) global compression that automatically finds the appropriate per-layer sparsity ratios; 2) end-to-end training; 3) no need for a time-consuming re-training process after pruning; and 4) superior capability to find better winning tickets which have won the initialization lottery.

arXiv commentAccepted by NeurIPS 2019

Cited by in corpus (33)

• Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors
• Rotated Binary Neural Network
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Loss-Aware Automatic Selection of Structured Pruning Criteria for Deep Neural Network Acceleration
• Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval
• A Gradient Flow Framework For Analyzing Network Pruning
• Knapsack Pruning with Inner Distillation
• On the Compression of Neural Networks Using ${\ell }_0$-Norm Regularization and Weight Pruning
• FreezeNet: Full Performance by Reduced Storage Costs
• Real-time Universal Style Transfer on High-resolution Images via Zero-channel Pruning
• Taxonomy of Saliency Metrics for Channel Pruning
• Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
• Revisiting Loss Modelling for Unstructured Pruning
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• CPOT: Channel Pruning via Optimal Transport
• Neural Pruning via Growing Regularization
• Network Pruning using Adaptive Exemplar Filters
• A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs
• SASL: Saliency-Adaptive Sparsity Learning for Neural Network Acceleration
• ReCU: Reviving the Dead Weights in Binary Neural Networks
• Joint Channel and Weight Pruning for Model Acceleration on Moblie Devices
• Training Sparse Neural Networks using Compressed Sensing
• Learning to Prune in Training via Dynamic Channel Propagation
• NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks
• The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics
• Architecture Aware Latency Constrained Sparse Neural Networks
• Multi-Scale Aligned Distillation for Low-Resolution Detection
• CNN with large memory layers
• Dep-$L_0$: Improving $L_0$-based Network Sparsification via Dependency Modeling
• Weight Evolution: Improving Deep Neural Networks Training through Evolving Inferior Weight Values