Anon: Extrapolating Adaptivity Beyond SGD and Adam

Yiheng Zhang, Kaiyan Zhao, Shaowu Wu +5

Adaptive optimizers such as Adam have achieved great success in training large-scale models like large language models and diffusion models. However, they often generalize worse th…