Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics

Cong Xu, Wenbin Liang, Mo Yu +7

The rapid scaling of models has led to prohibitively high training and fine-tuning costs. A major factor accounting for memory consumption is the widespread use of stateful optimiz…