Gradient correlation is a key ingredient to accelerate SGD with momentum

Julien Hermant, Marien Renaud, Jean-François Aujol +2

Empirically, it has been observed that adding momentum to Stochastic Gradient Descent (SGD) accelerates the convergence of the algorithm. However, the literature has been rather pe…