Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks

Shenxi Wu, Haosong Zhang, Xingjian Ma +4

Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning. Maximal Update Parametrization ($\mu$P) helps explain why man…