Reducing the variance in online optimization by transporting past gradients

Abstract

Most stochastic optimization methods use gradients once before discarding them. While variance reduction methods have shown that reusing past gradients can be beneficial when there is a finite number of datapoints, they do not easily extend to the online setting. One issue is the staleness due to using past gradients. We propose to correct this staleness using the idea of implicit gradient transport (IGT) which transforms gradients computed at previous iterates into gradients evaluated at the current iterate without using the Hessian explicitly. In addition to reducing the variance and bias of our updates over time, IGT can be used as a drop-in replacement for the gradient estimate in a number of well-understood methods such as heavy ball or Adam. We show experimentally that it achieves state-of-the-art results on a wide range of architectures and benchmarks. Additionally, the IGT gradient estimator yields the optimal asymptotic convergence rate for online stochastic optimization in the restricted setting where the Hessians of all component functions are equal.

Open-source implementation available at: https://github.com/seba-1511/igt.pth

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Measuring the Effects of Data Parallelism on Neural Network Training
• Optimization with First-Order Surrogate Functions
• Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
• Stop Wasting My Gradients: Practical SVRG
• YellowFin and the Art of Momentum Tuning
• Accelerating Stochastic Gradient Descent For Least Squares Regression
• Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chains
• Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods
• Curvature-aided Incremental Aggregated Gradient Method
• Anytime Tail Averaging

Cited by in corpus (5)

• Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip
• High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails
• ErrorCompensatedX: error compensation for variance reduced algorithms
• Scaling transition from momentum stochastic gradient descent to plain stochastic gradient descent
• Momentum Improves Normalized SGD