Low-Rank Friction for Memory-Efficient Transformer Pretraining

Rajit Rajpal, Benedict Leimkuhler

iKFAD is a recently proposed optimiser that replaces adaptive learning rates with adaptive friction in the momentum dynamics, yet performs as well as Adam. Its limitation is that t…