Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks

Abstract

(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly train-prune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping and low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 4.7x energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training. Code available at https://github.com/RICE-EIC/Early-Bird-Tickets.

arXiv commentAccepted as ICLR2020 Spotlight

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Rethinking the Value of Network Pruning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Training and Inference with Integers in Deep Neural Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Scalable Methods for 8-bit Training of Neural Networks
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• On the Spectral Bias of Neural Networks
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• The Early Phase of Neural Network Training
• Low-Memory Neural Network Training: A Technical Report
• SWALP : Stochastic Weight Averaging in Low-Precision Training
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers

Cited by in corpus (13)

• Efficient Quantized Sparse Matrix Operations on Tensor Cores
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• L${}^2$-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networks
• Pufferfish: Communication-efficient Models At No Extra Cost
• A Generalized Lottery Ticket Hypothesis
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks
• LNPT: Label-free Network Pruning and Training
• How many winning tickets are there in one DNN?
• BN-NAS: Neural Architecture Search with Batch Normalization
• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?