45 citations · 93 across the 25 of their papers we have counts for
28 papers
Measuring the local shape of learning-rate transfer curves
Satoki Ishikawa, Rio Yokota, Ryo Karakida
Understanding the learning-rate (LR) dependence of trained model performance is essential for better optimization of neural networks, but the appropriate size of the LR depends on…
Gaussian Equivalence for Multi-Head Self-Attention
Tomohiro Hayase, Ryo Karakida
A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for mu…
Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models
Ken Takeda, Masafumi Oizumi, Ryo Karakida
Generative models, including diffusion models, are increasingly used as foundation models and adapted through sequential fine-tuning, making continual learning an essential problem…
A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention
Tomohiro Hayase, Ryo Karakida
Length-dependent logit rescaling is widely used to stabilize long-context self-attention, but existing analyses and methods suggest conflicting inverse-temperature laws for the con…
The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networks
Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae
The success of deep neural networks largely depends on the statistical structure of the training data. While learning dynamics and generalization on isotropic data are well-establi…
Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix
Tomohiro Hayase, Benoît Collins, Ryo Karakida
Self-attention layers have become fundamental building blocks of modern deep neural networks, yet their theoretical understanding remains limited, particularly from the perspective…