DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule

Euijin Hong, Guannan Qu

Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate. However, the inputs a deep networ…