2 papers
cs.LG2026
Adaptive Momentum and Nonlinear Damping for Neural Network Training
Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler +1
Momentum Stochastic Gradient Descent (mSGD) relies on a fixed momentum coefficient shared across all parameters, failing to account for the heterogeneous structure of modern loss l…
math.OC2023
Friction-adaptive descent: a family of dynamics-based optimization methods
Aikaterini Karoni, Benedict Leimkuhler, Gabriel Stoltz
We describe a family of descent algorithms which generalizes common existing schemes used in applications such as neural network training and more broadly for optimization of smoot…