Effects of momentum scaling for SGD

Dmitry A. Pasechnyuk, Alexander Gasnikov, Martin Takáč

The paper studies the properties of stochastic gradient methods with preconditioning. We focus on momentum updated preconditioners with momentum coefficient $\beta$. Seeking to explain…