Less Regret via Online Conditioning
arXiv:1002.4862
Abstract
We analyze and evaluate an online gradient descent algorithm with adaptive per-coordinate adjustment of learning rates. Our algorithm can be thought of as an online version of batch gradient descent with a diagonal preconditioner. This approach leads to regret bounds that are stronger than those of standard online gradient descent for general online convex optimization problems. Experimentally, we show that our algorithm is competitive with state-of-the-art algorithms for large scale machine learning problems.
Cited by in corpus (9)
- Adaptive Bound Optimization for Online Convex Optimization
- Online Learning: A Modern Introduction Using Convex Optimization
- Simultaneous Model Selection and Optimization through Parameter-free Stochastic Learning
- A Survey of Algorithms and Analysis for Adaptive Online Learning
- Large-Scale Learning with Less RAM via Randomization
- Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize
- Fast Dimension Independent Private AdaGrad on Publicly Estimated Subspaces
- Scale-Free Algorithms for Online Linear Optimization
- Training Deep Networks without Learning Rates Through Coin Betting