Reverse engineering learned optimizers reveals known and novel mechanisms
arXiv:2011.02159
Abstract
Learned optimizers are algorithms that can themselves be trained to solve optimization problems. In contrast to baseline optimizers (such as momentum or Adam) that use simple update rules derived from theoretical principles, learned optimizers use flexible, high-dimensional, nonlinear parameterizations. Although this can lead to better performance in certain settings, their inner workings remain a mystery. How is a learned optimizer able to outperform a well tuned baseline? Has it learned a sophisticated combination of existing optimization techniques, or is it implementing completely new behavior? In this work, we address these questions by careful analysis and visualization of learned optimizers. We study learned optimizers trained from scratch on three disparate tasks, and discover that they have learned interpretable mechanisms, including: momentum, gradient clipping, learning rate schedules, and a new form of learning rate adaptation. Moreover, we show how the dynamics of learned optimizers enables these behaviors. Our results help elucidate the previously murky understanding of how learned optimizers work, and establish tools for interpreting future learned optimizers.
arXiv commentThirty-Fifth Conference on Neural Information Processing Systems. 2021
References in corpus (26)
- Adam: A Method for Stochastic Optimization
- Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
- On the difficulty of training Recurrent Neural Networks
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- SGDR: Stochastic Gradient Descent with Warm Restarts
- On the Convergence of Adam and Beyond
- Data-driven discovery of coordinates and governing equations
- Don't Decay the Learning Rate, Increase the Batch Size
- A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- No More Pesky Learning Rates
- Discovering Symbolic Models from Deep Learning with Inductive Biases
- Neural Optimizer Search with Reinforcement Learning
- On Empirical Comparisons of Optimizers for Deep Learning
- Learning to Optimize
- A Lyapunov Analysis of Momentum Methods in Optimization
- Why gradient clipping accelerates training: A theoretical justification for adaptivity
- Acceleration via Symplectic Discretization of High-Resolution Differential Equations
- Learning Gradient Descent: Better Generalization and Longer Horizons
- Understanding and correcting pathologies in the training of learned optimizers
- An Exponential Learning Rate Schedule for Deep Learning
- Understanding Short-Horizon Bias in Stochastic Meta-Optimization
- Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
- How recurrent networks implement contextual processing in sentiment analysis
- Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs