Integration Methods and Accelerated Optimization Algorithms

Abstract

We show that accelerated optimization methods can be seen as particular instances of multi-step integration schemes from numerical analysis, applied to the gradient flow equation. In comparison with recent advances in this vein, the differential equation considered here is the basic gradient flow and we show that multi-step schemes allow integration of this differential equation using larger step sizes, thus intuitively explaining acceleration results.

References in corpus (1)

• A Lyapunov Analysis of Momentum Methods in Optimization

Cited by in corpus (5)

• The Approximate Duality Gap Technique: A Unified Theory of First-Order Methods
• Direct Runge-Kutta Discretization Achieves Acceleration
• Potential Function-based Framework for Making the Gradients Small in Convex and Min-Max Optimization
• Continuous vs. Discrete Optimization of Deep Neural Networks
• Conjugate Gradients and Accelerated Methods Unified: The Approximate Duality Gap View