Adaptive Momentum and Nonlinear Damping for Neural Network Training

Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler +1

Momentum Stochastic Gradient Descent (mSGD) relies on a fixed momentum coefficient shared across all parameters, failing to account for the heterogeneous structure of modern loss l…