A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization
arXiv:1706.06569
Abstract
We describe a framework for deriving and analyzing online optimization algorithms that incorporate adaptive, data-dependent regularization, also termed preconditioning. Such algorithms have been proven useful in stochastic optimization by reshaping the gradients according to the geometry of the data. Our framework captures and unifies much of the existing literature on adaptive online methods, including the AdaGrad and Online Newton Step algorithms as well as their diagonal versions. As a result, we obtain new convergence proofs for these algorithms that are substantially simpler than previous analyses. Our framework also exposes the rationale for the different preconditioned updates used in common stochastic optimization methods.
Cited by in corpus (4)
- Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
- Structured Dropout Variational Inference for Bayesian Neural Networks
- Extreme Tensoring for Low-Memory Preconditioning
- Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent