How I learned to stop worrying and love StopGrads: Stationarity, Convergence, and a case study on Flow Map Learning

Max W. Shen, Mark Goldstein, Zichu Wang +2

Stopgrads are widely used in training machine learning models, but stopgrads can alter the gradient, stationary points and convergence guarantees of the original objective, which c…