Weight Decay may matter more than muP for Learning Rate Transfer in Practice

Atli Kosson, Jeremy Welborn, Yang Liu +2

Transferring the optimal learning rate from small to large neural networks can enable efficient training at scales where hyperparameter tuning is otherwise prohibitively expensive.…