Accelerated Gradient Methods with Memory

Abstract

A set of accelerated first order algorithms with memory are proposed for minimising strongly convex functions. The algorithms are differentiated by their use of the iterate history for the gradient step. The increased convergence rate of the proposed algorithms comes at the cost of robustness, a problem that is resolved by a switching controller based upon adaptive restarting. Several numerical examples highlight the benefits of the proposed approach over the fast gradient method. For example, it is shown that these gradient based methods can minimise the Rosenbrock banana function to $7.58\times 10^{-12}$ in 43 iterations from an initial condition of $(-1,1)$.

arXiv comment10 pages. 10 figures

References in corpus (7)

• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• A Variational Perspective on Accelerated Methods in Optimization
• How to Escape Saddle Points Efficiently
• A Lyapunov Analysis of Momentum Methods in Optimization
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• On Symplectic Optimization
• Dissipativity Theory for Nesterov's Accelerated Method

Cited by in corpus (1)

• Optimization methods for achieving high diffraction efficiency with perfect electric conducting gratings