Understanding and correcting pathologies in the training of learned optimizers

Abstract

Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned optimizers may similarly outperform current hand-designed optimizers, especially for specific problems. However, learned optimizers are notoriously difficult to train and have yet to demonstrate wall-clock speedups over hand-designed optimizers, and thus are rarely used in practice. Typically, learned optimizers are trained by truncated backpropagation through an unrolled optimization process resulting in gradients that are either strongly biased (for short truncations) or have exploding norm (for long truncations). In this work we propose a training scheme which overcomes both of these difficulties, by dynamically weighting two unbiased gradient estimators for a variational loss on optimizer performance, allowing us to train neural networks to perform optimization of a specific task faster than tuned first-order methods. We demonstrate these results on problems where our learned optimizer trains convolutional networks faster in wall-clock time compared to tuned first-order methods and with an improvement in test loss.

Cited by in corpus (32)

• Learning to Optimize: A Primer and A Benchmark
• HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• Accelerating Quadratic Optimization with Reinforcement Learning
• Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Stateless Neural Meta-Learning using Second-Order Gradients
• Parallel Training of Deep Networks with Local Updates
• Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
• A Generalizable Approach to Learning Optimizers
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation
• Improved Adversarial Training via Learned Optimizer
• Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• Online hyperparameter optimization by real-time recurrent learning
• Training Learned Optimizers with Randomly Initialized Learned Optimizers
• Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies
• Meta-Learning Bidirectional Update Rules
• Learning to Remember from a Multi-Task Teacher
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Learning the Step-size Policy for the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm
• Gradients are Not All You Need
• MetricOpt: Learning to Optimize Black-Box Evaluation Metrics
• One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning
• Bootstrapped Meta-Learning
• Gradient-based Hyperparameter Optimization Over Long Horizons
• MTL2L: A Context Aware Neural Optimiser
• ModelPred: A Framework for Predicting Trained Model from Training Data