REX: Revisiting Budgeted Training with an Improved Schedule

Abstract

Deep learning practitioners often operate on a computational and monetary budget. Thus, it is critical to design optimization algorithms that perform well under any budget. The linear learning rate schedule is considered the best budget-aware schedule, as it outperforms most other schedules in the low budget regime. On the other hand, learning rate schedules -- such as the \texttt{30-60-90} step schedule -- are known to achieve high performance when the model can be trained for many epochs. Yet, it is often not known a priori whether one's budget will be large or small; thus, the optimal choice of learning rate schedule is made on a case-by-case basis. In this paper, we frame the learning rate schedule selection problem as a combination of $i)$ selecting a profile (i.e., the continuous function that models the learning rate schedule), and $ii)$ choosing a sampling rate (i.e., how frequently the learning rate is updated/sampled from this profile). We propose a novel profile and sampling rate combination called the Reflected Exponential (REX) schedule, which we evaluate across seven different experimental settings with both SGD and Adam optimizers. REX outperforms the linear schedule in the low budget regime, while matching or exceeding the performance of several state-of-the-art learning rate schedules (linear, step, exponential, cosine, step decay on plateau, and OneCycle) in both high and low budget regimes. Furthermore, REX requires no added computation, storage, or hyperparameters.

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• YOLOv3: An Incremental Improvement
• ADADELTA: An Adaptive Learning Rate Method
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Don't Decay the Learning Rate, Increase the Batch Size
• FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• NVAE: A Deep Hierarchical Variational Autoencoder
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Tackling Over-pruning in Variational Autoencoders
• Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
• Deep Feature Consistent Variational Autoencoder

Cited by in corpus (2)

• Revisiting Tensor Basis Neural Networks for Reynolds stress modeling: application to plane channel and square duct flows
• Better Schedules for Low Precision Training of Deep Neural Networks