The Rich and the Simple: On the Implicit Bias of Adam and SGD

Bhavya Vasudeva, Jung Whan Lee, Vatsal Sharan +1

Adam is the de facto optimization algorithm for several deep learning applications, but an understanding of its implicit bias and how it differs from other algorithms, particularly…