A Theory on Adam Instability in Large-Scale Machine Learning

Igor Molybog, Peter Albert, Moya Chen +14

We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant…