Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods

Junchi Yang, Xiang Li, Ilyas Fatkhullin +1

The classical analysis of Stochastic Gradient Descent (SGD) with polynomially decaying stepsize ${\eta }_t=\eta /\sqrt{t}$ relies on well-tuned $\eta$ depending on problem parameters such as…