Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training

Abraham J. Fetterman, Ellie Kitanidis, Joshua Albrecht +6

Hyperparameter tuning of deep learning models can lead to order-of-magnitude performance gains for the same amount of compute. Despite this, systematic tuning is uncommon, particul…