Selfish Sparse RNN Training

Abstract

Sparse neural networks have been widely applied to reduce the computational demands of training and deploying over-parameterized deep neural networks. For inference acceleration, methods that discover a sparse network from a pre-trained dense network (dense-to-sparse training) work effectively. Recently, dynamic sparse training (DST) has been proposed to train sparse neural networks without pre-training a dense model (sparse-to-sparse training), so that the training process can also be accelerated. However, previous sparse-to-sparse methods mainly focus on Multilayer Perceptron Networks (MLPs) and Convolutional Neural Networks (CNNs), failing to match the performance of dense-to-sparse methods in the Recurrent Neural Networks (RNNs) setting. In this paper, we propose an approach to train intrinsically sparse RNNs with a fixed parameter count in one single run, without compromising performance. During training, we allow RNN layers to have a non-uniform redistribution across cell gates for better regularization. Further, we propose SNT-ASGD, a novel variant of the averaged stochastic gradient optimizer, which significantly improves the performance of all sparse training methods for RNNs. Using these strategies, we achieve state-of-the-art sparse training results, better than the dense-to-sparse methods, with various types of RNNs on Penn TreeBank and Wikitext-2 datasets. Our codes are available at https://github.com/Shiweiliuiiiiiii/Selfish-RNN.

arXiv commentPublished in Proceedings of the 38th International Conference on Machine Learning. Code can be found in https://github.com/Shiweiliuiiiiiii/Selfish-RNN

References in corpus (32)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Recurrent Neural Network Regularization
• Training Very Deep Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• The State of Sparsity in Deep Neural Networks
• Bayesian Compression for Deep Learning
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Qualitatively characterizing neural network optimization problems
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Essentially No Barriers in Neural Network Energy Landscape
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Deep Rewiring: Training very sparse deep networks
• Learning Sparse Neural Networks through $L_0$ Regularization
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Convergent Learning: Do different neural networks learn the same representations?
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• A topological insight into restricted Boltzmann machines
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Progressive Skeletonization: Trimming more fat from a network at initialization
• Top-KAST: Top-K Always Sparse Training
• Sparse Weight Activation Training
• Intrinsically Sparse Long Short-Term Memory Networks
• Topological Insights into Sparse Neural Networks

Cited by in corpus (6)

• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Spectral Pruning for Recurrent Neural Networks
• The Elastic Lottery Ticket Hypothesis