Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling

Shuaipeng Li, Penghao Zhao, Hailin Zhang +10

In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optim…