Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
arXiv:2011.04803
Abstract
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such as the curvature can help de-sensitize common hyperparameters. Based on this intuition, we explore the use of exact per-sample Hessian-vector products and gradients to construct optimizers that are self-tuning and hyperparameter-free. Based on a dynamics model of the gradient, we derive a process which leads to a curvature-corrected, noise-adaptive online gradient estimate. The smoothness of our updates makes it more amenable to simple step size selection schemes, which we also base off of our estimates quantities. We prove that our model-based procedure converges in the noisy quadratic setting. Though we do not see similar gains in deep learning tasks, we can match the performance of well-tuned optimizers and ultimately, this is an interesting step for constructing self-tuning optimizers.
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- ADADELTA: An Adaptive Learning Rate Method
- New insights and perspectives on the natural gradient method
- On Empirical Comparisons of Optimizers for Deep Learning
- A Framework for Self-Tuning Optimization Algorithm
- Online Learning Rate Adaptation with Hypergradient Descent
- Understanding Short-Horizon Bias in Stochastic Meta-Optimization
- Kalman-based Stochastic Gradient Method with Stop Condition and Insensitivity to Conditioning
- Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimization
- DeepOBS: A Deep Learning Optimizer Benchmark Suite
- Auto-Vectorizing TensorFlow Graphs: Jacobians, Auto-Batching And Beyond
- Training Neural Networks for and by Interpolation
- Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods