8 citations · 23 across the 12 of their papers we have counts for
Showing 2021Show all
3 papers · 1 filter
cs.LG2021★ 1 cited
SGD with a Constant Large Learning Rate Can Converge to Local Maxima
Liu Ziyin, Botao Li, James B. Simon +1
Previous works on stochastic gradient descent (SGD) often focus on its success. In this work, we construct worst-case optimization problems illustrating that, when not in the regim…
cs.LG2021★ 3 cited
Power-law escape rate of SGD
Takashi Mori, Liu Ziyin, Kangqiao Liu +1
Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss. We use this property of SGD noise to derive a stochastic differential equatio…
cs.LG2021★ 8 cited
Strength of Minibatch Noise in SGD
Liu Ziyin, Kangqiao Liu, Takashi Mori +1
The noise in stochastic gradient descent (SGD), caused by minibatch sampling, is poorly understood despite its practical importance in deep learning. This work presents the first s…