4 papers
Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theory
Sota Nishiyama, Masaaki Imaizumi
Diagonal linear networks (DLNs) are a tractable model that captures several nontrivial behaviors in neural network training, such as initialization-dependent solutions and incremen…
High-Dimensional Limit of Stochastic Gradient Flow via Dynamical Mean-Field Theory
Sota Nishiyama, Masaaki Imaizumi
Modern machine learning models are typically trained via multi-pass stochastic gradient descent (SGD) with small batch sizes, and understanding their dynamics in high dimensions is…
Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descent
Shota Imai, Sota Nishiyama, Masaaki Imaizumi
The dynamics of gradient-based training in neural networks often exhibit nontrivial structures; hence, understanding them remains a central challenge in theoretical machine learnin…
Solution space and storage capacity of fully connected two-layer neural networks with generic activation functions
Sota Nishiyama, Masayuki Ohzeki
The storage capacity of a binary classification model is the maximum number of random input-output pairs per parameter that the model can learn. It is one of the indicators of the…