30 citations · 30 across the 6 of their papers we have counts for
3 papers · 1 filter
Low-Rank Friction for Memory-Efficient Transformer Pretraining
Rajit Rajpal, Benedict Leimkuhler
iKFAD is a recently proposed optimiser that replaces adaptive learning rates with adaptive friction in the momentum dynamics, yet performs as well as Adam. Its limitation is that t…
Accelerated sampling using SamAdams variable timesteps and position-adaptive Langevin dynamics
Benedict Leimkuhler, Peter A. Whalley
We introduce an accelerated Langevin-based sampling method that is based on two complementary devices: \emph{SamAdams} adaptive timestepping, which automatically shrinks the effect…
Adaptive Momentum and Nonlinear Damping for Neural Network Training
Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler +1
Momentum Stochastic Gradient Descent (mSGD) relies on a fixed momentum coefficient shared across all parameters, failing to account for the heterogeneous structure of modern loss l…