New Results on the Polyak Stepsize: Tight Convergence Analysis and Universal Function Classes

Chang He, Wenzhi Gao, Bo Jiang +2

In this paper, we revisit a classical adaptive stepsize strategy for gradient descent: the Polyak stepsize (PolyakGD), originally proposed in Polyak (1969). We study the convergenc…