Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers

Abstract

We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly find the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds. These thresholds can have fine-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same number of training epochs as dense models. Dynamic Sparse Training achieves the state of the art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence for the effectiveness and efficiency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures.

ICLR 2020, camera ready version

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Binarized Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Deep Rewiring: Training very sparse deep networks
• Exploring Sparsity in Recurrent Neural Networks
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization

Cited by in corpus (10)

• Soft Threshold Weight Reparameterization for Learnable Sparsity
• SpaceNet: Make Free Space For Continual Learning
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Sparse Weight Activation Training
• Training Sparse Neural Networks using Compressed Sensing
• Training highly effective connectivities within neural networks with randomly initialized, fixed weights
• Search Spaces for Neural Model Training
• RGP: Neural Network Pruning through Its Regular Graph Structure