Variance Reduction in Deep Learning: More Momentum is All You Need

Lionel Tondji, Sergii Kashubin, Moustapha Cisse

Variance reduction (VR) techniques have contributed significantly to accelerating learning with massive datasets in the smooth and strongly convex setting (Schmidt et al., 2017; Jo…