Projection-Free Adaptive Gradients for Large-Scale Optimization
arXiv:2009.14114
Abstract
The complexity in large-scale optimization can lie in both handling the objective function and handling the constraint set. In this respect, stochastic Frank-Wolfe algorithms occupy a unique position as they alleviate both computational burdens, by querying only approximate first-order information from the objective and by maintaining feasibility of the iterates without using projections. In this paper, we improve the quality of their first-order information by blending in adaptive gradients. We derive convergence rates and demonstrate the computational advantage of our method over the state-of-the-art stochastic Frank-Wolfe algorithms on both convex and nonconvex objectives. The experiments further show that our method can improve the performance of adaptive gradient algorithms for constrained optimization.
arXiv comment28 pages, 10 figures
References in corpus (8)
- ADADELTA: An Adaptive Learning Rate Method
- On the Convergence of Adam and Beyond
- Improving Generalization Performance by Switching from Adam to SGD
- SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- Adaptive Bound Optimization for Online Convex Optimization
- Convergence Rate of Frank-Wolfe for Non-Convex Objectives
- Variance-Reduced and Projection-Free Stochastic Optimization