Adam Converges Without Any Modification On Update Rules

Yushun Zhang, Bingran Li, Congliang Chen +2

Adam is the default algorithm for training neural networks, including large language models (LLMs). However, \citet{reddi2019convergence} provided an example that Adam diverges, ra…