8 citations · 23 across the 12 of their papers we have counts for
12 papers
Noise Balance and Stationary Distribution of Stochastic Gradient Descent
Liu Ziyin, Hongchao Li, Masahito Ueda
The stochastic gradient descent (SGD) algorithm is the algorithm we use to train neural networks. However, it remains poorly understood how the SGD navigates the highly nonlinear a…
Pink-noise dynamics in an evolutionary game on a regular graph
Yuki Sakamoto, Masahito Ueda
We consider an iterated multiplayer prisoner's dilemma game on a square lattice and regular graphs based on the pairwise-Fermi update rule, and obtain heat-maps of the fraction of…
Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent
Liu Ziyin, Botao Li, Tomer Galanti +1
Characterizing and understanding the dynamics of stochastic gradient descent (SGD) around saddle points remains an open problem. We first show that saddle points in neural networks…
What shapes the loss landscape of self-supervised learning?
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda +1
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theor…
Exact Phase Transitions in Deep Learning
Liu Ziyin, Masahito Ueda
This work reports deep-learning-unique first-order and second-order phase transitions, whose phenomenology closely follows that in statistical physics. In particular, we prove that…
Universal Thermodynamic Uncertainty Relation in Non-Equilibrium Dynamics
Liu Ziyin, Masahito Ueda
We derive a universal thermodynamic uncertainty relation (TUR) that applies to an arbitrary observable in a general Markovian system. The generality of our result allows us to make…
Interplay between depth of neural networks and locality of target functions
Takashi Mori, Masahito Ueda
It has been recognized that heavily overparameterized deep neural networks (DNNs) exhibit surprisingly good generalization performance in various machine-learning tasks. Although b…
Stochastic Neural Networks with Infinite Width are Deterministic
Liu Ziyin, Hanlin Zhang, Xiangming Meng +3
This work theoretically studies stochastic neural networks, a main type of neural network in use. We prove that as the width of an optimized stochastic neural network tends to infi…
SGD with a Constant Large Learning Rate Can Converge to Local Maxima
Liu Ziyin, Botao Li, James B. Simon +1
Previous works on stochastic gradient descent (SGD) often focus on its success. In this work, we construct worst-case optimization problems illustrating that, when not in the regim…
Power-law escape rate of SGD
Takashi Mori, Liu Ziyin, Kangqiao Liu +1
Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss. We use this property of SGD noise to derive a stochastic differential equatio…
Strength of Minibatch Noise in SGD
Liu Ziyin, Kangqiao Liu, Takashi Mori +1
The noise in stochastic gradient descent (SGD), caused by minibatch sampling, is poorly understood despite its practical importance in deep learning. This work presents the first s…
Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
Kangqiao Liu, Liu Ziyin, Masahito Ueda
In the vanishing learning rate regime, stochastic gradient descent (SGD) is now relatively well understood. In this work, we propose to study the basic properties of SGD and its va…