Anon: Extrapolating Adaptivity Beyond SGD and Adam

Yiheng Zhang, Kaiyan Zhao, Shaowu Wu +5

Adaptive optimizers such as Adam and non-adaptive methods like SGD exhibit distinct generalization capabilities across different architectures. Prior tunable optimizers attempt to…