Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization

Abstract

Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive. Recent work suggests that initialization is the key. However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate. In this paper, we take a broader view of training sparse networks and consider the role of regularization, optimization, and architecture choices on sparse models. We propose a simple experimental framework, Same Capacity Sparse vs Dense Comparison (SC-SDC), that allows for a fair comparison of sparse and dense networks. Furthermore, we propose a new measure of gradient flow, Effective Gradient Flow (EGF), that better correlates to performance in sparse networks. Using top-line metrics, SC-SDC and EGF, we show that default choices of optimizers, activation functions and regularizers used for dense networks can disadvantage sparse networks. Based upon these findings, we show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime. Our work suggests that initialization is only one piece of the puzzle and taking a wider view of tailoring optimization to sparse networks yields promising results.

References in corpus (28)

• Decoupled Weight Decay Regularization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• On the difficulty of training Recurrent Neural Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• SpArch: Efficient Architecture for Sparse Matrix Multiplication
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Deep Rewiring: Training very sparse deep networks
• Fantastic Generalization Measures and Where to Find Them
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Stabilizing the Lottery Ticket Hypothesis
• Exploring Sparsity in Recurrent Neural Networks
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Deep Learning with S-shaped Rectified Linear Activation Units
• Dive into Deep Learning
• How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• The Difficulty of Training Sparse Neural Networks
• A Gradient Flow Framework For Analyzing Network Pruning
• A Signal Propagation Perspective for Pruning Neural Networks at Initialization
• Characterizing Well-Behaved vs. Pathological Deep Neural Networks
• Activation function impact on Sparse Neural Networks

Cited by in corpus (5)

• What Do Compressed Deep Neural Networks Forget?
• Randomness In Neural Network Training: Characterizing The Impact of Tooling
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Towards Structured Dynamic Sparse Pre-Training of BERT
• Spending Your Winning Lottery Better After Drawing It