Variance Reduction with Sparse Gradients

Melih Elibol, Lihua Lei, Michael I. Jordan

Variance reduction methods such as SVRG and SpiderBoost use a mixture of large and small batch gradients to reduce the variance of stochastic gradients. Compared to SGD, these meth…