Understanding Short-Horizon Bias in Stochastic Meta-Optimization

Abstract

Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is unrolled. But because the training procedure must be unrolled thousands of times, the meta-objective must be defined with an orders-of-magnitude shorter time horizon than is typical for neural net training. We show that such short-horizon meta-objectives cause a serious bias towards small step sizes, an effect we term short-horizon bias. We introduce a toy problem, a noisy quadratic cost function, on which we analyze short-horizon bias by deriving and comparing the optimal schedules for short and long time horizons. We then run meta-optimization experiments (both offline and online) on standard benchmark datasets, showing that meta-optimization chooses too small a learning rate by multiple orders of magnitude, even when run with a moderately long time horizon (100 steps) typical of work in the area. We believe short-horizon bias is a fundamental problem that needs to be addressed if meta-optimization is to scale to practical neural net training regimes.

17 pages, 8 figures; To appear in ICLR2018

References in corpus (7)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Understanding Black-box Predictions via Influence Functions
• No More Pesky Learning Rates
• Learned Optimizers that Scale and Generalize
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Noisy Natural Gradient as Variational Inference

Cited by in corpus (28)

• Lookahead Optimizer: k steps forward, 1 step back
• An Empirical Model of Large-Batch Training
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Incremental Few-Shot Learning with Attention Attractor Networks
• Meta-Learning for Stochastic Gradient MCMC
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Offline Meta-Reinforcement Learning with Advantage Weighting
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• On Accelerating Distributed Convex Optimizations
• Neural Complexity Measures
• A Generalizable Approach to Learning Optimizers
• Learning Composable Energy Surrogates for PDE Order Reduction
• Improved Adversarial Training via Learned Optimizer
• Population-Based Evolution Optimizes a Meta-Learning Objective
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Stability and Generalization of Bilevel Programming in Hyperparameter Optimization
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
• Gradients are Not All You Need
• Meta-Learning to Improve Pre-Training
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients
• Linear Speedup in Personalized Collaborative Learning