The Difficulty of Training Sparse Neural Networks

Abstract

We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that are significantly worse than those found by pruning. We show that, despite the failure of optimizers, there is a linear path with a monotonically decreasing objective from the initialization to the "good" solution. Additionally, our attempts to find a decreasing objective path from "bad" solutions to the "good" ones in the sparse subspace fail. However, if we allow the path to traverse the dense subspace, then we consistently find a path between two solutions. These findings suggest traversing extra dimensions may be needed to escape stationary points found in the sparse subspace.

arXiv commentsparse networks, pruning, energy landscape, sparsity

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Rethinking the Value of Network Pruning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Qualitatively characterizing neural network optimization problems
• Deep Rewiring: Training very sparse deep networks
• The statistics of critical points of Gaussian fields on large-dimensional spaces
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization

Cited by in corpus (21)

• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Learning Neural Network Subspaces
• Directional Pruning of Deep Neural Networks
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Powerpropagation: A sparsity inducing weight reparameterisation
• GANs Can Play Lottery Tickets Too
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
• Search Spaces for Neural Model Training
• $S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
• Deconstructing the Structure of Sparse Neural Networks