A Generalizable Approach to Learning Optimizers
arXiv:2106.00958
Abstract
A core issue with learning to optimize neural networks has been the lack of generalization to real world problems. To address this, we describe a system designed from a generalization-first perspective, learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function. This system outperforms Adam at all neural network tasks including on modalities not seen during training. We achieve 2x speedups on ImageNet, and a 2.5x speedup on a language modeling task using over 5 orders of magnitude more compute than the training tasks.
References in corpus (28)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Language Models are Few-Shot Learners
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Scaling Laws for Neural Language Models
- Dota 2 with Large Scale Deep Reinforcement Learning
- Large Batch Training of Convolutional Networks
- Pointer Sentinel Mixture Models
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Gradient-based Hyperparameter Optimization through Reversible Learning
- Practical recommendations for gradient-based training of deep architectures
- On Empirical Comparisons of Optimizers for Deep Learning
- MLPerf Training Benchmark
- An Empirical Model of Large-Batch Training
- Learned Optimizers that Scale and Generalize
- Stop Wasting My Gradients: Practical SVRG
- Online Learning Rate Adaptation with Hypergradient Descent
- Single Headed Attention RNN: Stop Thinking With Your Head
- Understanding and correcting pathologies in the training of learned optimizers
- Understanding Short-Horizon Bias in Stochastic Meta-Optimization
- Learning an Adaptive Learning Rate Schedule
- Reinforcement Learning for Learning Rate Control
- Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
- The Two Regimes of Deep Network Training
- GradNets: Dynamic Interpolation Between Neural Architectures
- First-Order Preconditioning via Hypergradient Descent