Understanding the Mechanisms of Fast Hyperparameter Transfer

Nikhil Ghosh, Denny Wu, Alberto Bietti

The growing scale of deep learning models has rendered standard hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware hyperparame…