Optimizer Memory Schedules for Outscaling the Overtraining Axis

Katie Everett, Shikai Qiu

We investigate how optimizers scale across the overtraining axis and show that relative optimizer performance and optimal hyperparameters change substantially with training horizon…