574 citations · 2.1k across the 29 of their papers we have counts for
29 papers
Effective Theory of Transformers at Initialization
Emily Dinan, Sho Yaida, Susan Zhang
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks a…
Meta-Principled Family of Hyperparameter Scaling Strategies
Sho Yaida
In this note, we first derive a one-parameter family of hyperparameter scaling strategies that interpolates between the neural-tangent scaling and mean-field/maximal-update scaling…
The Principles of Deep Learning Theory
Daniel A. Roberts, Sho Yaida, Boris Hanin
This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks,…
Non-Gaussian processes and neural networks at finite widths
Sho Yaida
Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processe…
Robust Learning with Jacobian Regularization
Judy Hoffman, Daniel A. Roberts, Sho Yaida
Design of reliable systems must guarantee stability against input perturbations. In machine learning, such guarantee entails preventing overfitting and ensuring robustness of model…
Fluctuation-dissipation relations for stochastic gradient descent
Sho Yaida
The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that…