Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression

Xuheng Li, Quanquan Gu

Exponential moving average (EMA) has recently gained significant popularity in training modern deep learning models, especially diffusion-based generative models. However, there ha…