518 citations · 3.1k across the 58 of their papers we have counts for
4 papers · 1 filter
Power-law escape rate of SGD
Takashi Mori, Liu Ziyin, Kangqiao Liu +1
Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss. We use this property of SGD noise to derive a stochastic differential equatio…
Strength of Minibatch Noise in SGD
Liu Ziyin, Kangqiao Liu, Takashi Mori +1
The noise in stochastic gradient descent (SGD), caused by minibatch sampling, is poorly understood despite its practical importance in deep learning. This work presents the first s…
Improved generalization by noise enhancement
Takashi Mori, Masahito Ueda
Recent studies have demonstrated that noise in stochastic gradient descent (SGD) is closely related to generalization: A larger SGD noise, if not too large, results in better gener…
Is deeper better? It depends on locality of relevant features
Takashi Mori, Masahito Ueda
It has been recognized that a heavily overparameterized artificial neural network exhibits surprisingly good generalization performance in various machine-learning tasks. Recent th…