Nesterov's method with decreasing learning rate leads to accelerated stochastic gradient descent

Maxime Laborde, Adam M. Oberman

We present a coupled system of ODEs which, when discretized with a constant time step/learning rate, recovers Nesterov's accelerated gradient descent algorithm. The same ODEs, when…