Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods

Abstract

Our goal is to improve variance reducing stochastic methods through better control variates. We first propose a modification of SVRG which uses the Hessian to track gradients over time, rather than to recondition, increasing the correlation of the control variates and leading to faster theoretical convergence close to the optimum. We then propose accurate and computationally efficient approximations to the Hessian, both using a diagonal and a low-rank matrix. Finally, we demonstrate the effectiveness of our method on a wide range of problems.

17 pages, 2 figures, 1 table

References in corpus (2)

• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• Randomized Quasi-Newton Updates are Linearly Convergent Matrix Inversion Algorithms

Cited by in corpus (5)

• Stochastic Variance-Reduced Cubic Regularized Newton Method
• Reducing the variance in online optimization by transporting past gradients
• Fast Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron
• Sample Efficient Stochastic Variance-Reduced Cubic Regularization Method
• Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering