Decoupled Orthogonal Dynamics: Regularization for Deep Network Optimizers

Hao Chen, Jinghui Yuan, Hanmin Zhang

Is the standard weight decay in AdamW truly optimal? Although AdamW decouples weight decay from adaptive gradient scaling, a fundamental conflict remains: the Radial Tug-of-War. In…