Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

Abstract

Much as replacing hand-designed features with learned functions has revolutionized how we solve perceptual tasks, we believe learned algorithms will transform how we train models. In this work we focus on general-purpose learned optimizers capable of training a wide variety of problems with no user-specified hyperparameters. We introduce a new, neural network parameterized, hierarchical optimizer with access to additional features such as validation loss to enable automatic regularization. Most learned optimizers have been trained on only a single task, or a small number of tasks. We train our optimizers on thousands of tasks, making use of orders of magnitude more compute, resulting in optimizers that generalize better to unseen tasks. The learned optimizers not only perform well, but learn behaviors that are distinct from existing first order optimizers. For instance, they generate update steps that have implicit regularization and adapt as the problem hyperparameters (e.g. batch size) or architecture (e.g. neural network width) change. Finally, these learned optimizers show evidence of being useful for out of distribution tasks such as training themselves from scratch.

References in corpus (30)

• Adam: A Method for Stochastic Optimization
• The NumPy array: a structure for efficient numerical computation
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Dota 2 with Large Scale Deep Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• Deep Knowledge Tracing
• On First-Order Meta-Learning Algorithms
• Neural Optimizer Search with Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• On Empirical Comparisons of Optimizers for Deep Learning
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Learned Optimizers that Scale and Generalize
• Bilevel Programming for Hyperparameter Optimization and Meta-Learning
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Learning to Optimize Neural Nets
• Unbiasing Truncated Backpropagation Through Time
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Structured Evolution with Compact Architectures for Scalable Policy Optimization
• Understanding and correcting pathologies in the training of learned optimizers
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Learning an Adaptive Learning Rate Schedule
• Guided evolutionary strategies: Augmenting random search with surrogate gradients
• Risks from Learned Optimization in Advanced Machine Learning Systems
• Automatic Differentiation of Algorithms for Machine Learning
• Reinforcement Learning for Learning Rate Control
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
• Using learned optimizers to make models robust to input noise

Cited by in corpus (10)

• Learning to Optimize: A Primer and A Benchmark
• Meta Learning Backpropagation And Improving It
• Meta Learning Black-Box Population-Based Optimizers
• Evolving to learn: discovering interpretable plasticity rules for spiking networks
• A Generalizable Approach to Learning Optimizers
• Parameter Prediction for Unseen Deep Architectures
• Meta-Learning Bidirectional Update Rules
• Training Learned Optimizers with Randomly Initialized Learned Optimizers
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Memory Augmented Optimizers for Deep Learning