Dynamic Model Pruning with Feedback

Abstract

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.

appearing at ICLR 2020

References in corpus (15)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• Rethinking the Value of Network Pruning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• The State of Sparsity in Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Sparsified SGD with Memory
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Stabilizing the Lottery Ticket Hypothesis
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication

Cited by in corpus (32)

• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers
• Layer-adaptive sparsity for the Magnitude-based Pruning
• Progressive Skeletonization: Trimming more fat from a network at initialization
• WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• Provable Filter Pruning for Efficient Neural Networks
• Network Pruning That Matters: A Case Study on Retraining Variants
• Dynamic Hard Pruning of Neural Networks at the Edge of the Internet
• SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• Consistent Sparse Deep Learning: Theory and Computation
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion
• Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration
• Effective Model Sparsification by Scheduled Grow-and-Prune Methods
• Lottery Jackpots Exist in Pre-trained Models
• Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Neural Pruning via Growing Regularization
• Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model via Refined Gradients for Pruned Weights
• The Elastic Lottery Ticket Hypothesis
• Distributed Sparse SGD with Majority Voting
• How Well Do Sparse Imagenet Models Transfer?
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?
• On the Existence of Universal Lottery Tickets
• Masked Training of Neural Networks with Partial Gradients
• Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach
• RGP: Neural Network Pruning through Its Regular Graph Structure
• Architecture Aware Latency Constrained Sparse Neural Networks