Implicit Bias of AdamW: ${\ell }_{\infty }$ Norm Constrained Optimization

Shuo Xie, Zhiyuan Li

Adam with decoupled weight decay, also known as AdamW, is widely acclaimed for its superior performance in language modeling tasks, surpassing Adam with ${\ell }_2$ regularization in…