Adaptive Gradient Methods at the Edge of Stability

Jeremy M. Cohen, Behrooz Ghorbani, Shankar Krishnan +8

Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning. In this paper, we shed light on the behavior of these algorithms in the fu…