Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits

Daniel Morales-Brotons, Thijs Vogels, Hadrien Hendrikx

Weight averaging of Stochastic Gradient Descent (SGD) iterates is a popular method for training deep learning models. While it is often used as part of complex training pipelines t…