Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training

Abstract

In this paper, we introduce a new perspective on training deep neural networks capable of state-of-the-art performance without the need for the expensive over-parameterization by proposing the concept of In-Time Over-Parameterization (ITOP) in sparse training. By starting from a random sparse network and continuously exploring sparse connectivities during training, we can perform an Over-Parameterization in the space-time manifold, closing the gap in the expressibility between sparse training and dense training. We further use ITOP to understand the underlying mechanism of Dynamic Sparse Training (DST) and indicate that the benefits of DST come from its ability to consider across time all possible parameters when searching for the optimal sparse connectivity. As long as there are sufficient parameters that have been reliably explored during training, DST can outperform the dense neural network by a large margin. We present a series of experiments to support our conjecture and achieve the state-of-the-art sparse training performance with ResNet-50 on ImageNet. More impressively, our method achieves dominant performance over the overparameterization-based sparse methods at extreme sparsity levels. When trained on CIFAR-100, our method can match the performance of the dense model even at an extreme sparsity (98%). Code can be found https://github.com/Shiweiliuiiiiiii/In-Time-Over-Parameterization.

arXiv comment16 pages; 10 figures; Published in Proceedings of the 38th International Conference on Machine Learning. Code can be found https://github.com/Shiweiliuiiiiiii/In-Time-Over-Parameterization

References in corpus (37)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Language Models are Few-Shot Learners
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• A Convergence Theory for Deep Learning via Over-Parameterization
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Don't Decay the Learning Rate, Increase the Batch Size
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Qualitatively characterizing neural network optimization problems
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Deep Rewiring: Training very sparse deep networks
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Exploring Sparsity in Recurrent Neural Networks
• Compressing Neural Networks using the Variational Information Bottleneck
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• A topological insight into restricted Boltzmann machines
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Learning Sparse Networks Using Targeted Dropout
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Winning the Lottery with Continuous Sparsification
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• The Difficulty of Training Sparse Neural Networks
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• Top-KAST: Top-K Always Sparse Training
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Selfish Sparse RNN Training
• Grow and Prune Compact, Fast, and Accurate LSTMs
• Sparse Weight Activation Training
• Leveraging Sparse Linear Layers for Debuggable Deep Networks

Cited by in corpus (5)

• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Towards Structured Dynamic Sparse Pre-Training of BERT
• The Elastic Lottery Ticket Hypothesis