The Implicit Bias of Adam on Separable Data

Chenyang Zhang, Difan Zou, Yuan Cao

Adam has become one of the most favored optimizers in deep learning problems. Despite its success in practice, numerous mysteries persist regarding its theoretical understanding. I…