Optimal Two-Step Stepsize Schedule for Stochastic Gradient Methods

Luwei Bai, Baoyu Zhou

Structured nonconstant large stepsizes can improve the convergence of gradient descent in the deterministic setting. However, in stochastic optimization, aggressive stepsizes can a…