A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networks

Xin Liu, Wei Tao, Zhisong Pan

Momentum methods, including heavy-ball~(HB) and Nesterov's accelerated gradient~(NAG), are widely used in training neural networks for their fast convergence. However, there is a l…