A New Perspective of Accelerated Gradient Methods: The Controlled Invariant Manifold Approach

Abstract

Gradient Descent (GD) is a ubiquitous algorithm for finding the optimal solution to an optimization problem. For reduced computational complexity, the optimal solution ${\mathrm{x}}^{\ast }$ of the optimization problem must be attained in a minimum number of iterations. For this objective, the paper proposes a genesis of an accelerated gradient algorithm through the controlled dynamical system perspective. The objective of optimally reaching the optimal solution ${\mathrm{x}}^{\ast }$ where $\nabla f({\mathrm{x}}^{\ast })=0$ with a given initial condition $\mathrm{x}(0)$ is achieved through control.