Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness

Yuki Takezawa, Eduard Gorbunov

Normalized gradient descent is a widely studied adaptive optimization method. Most existing analyses focus on the best iterate or a weighted average of the iterates, whereas practi…