Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

Hyunseok Lee, Mihir Basil, Yizhou Liu +1

Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent emerges from learning peaked…