Variance Suppression: Balanced Training Process in Deep Learning

Tao Yi, Xingxuan Wang

Stochastic gradient descent updates parameters with summation gradient computed from a random data batch. This summation will lead to unbalanced training process if the data we obt…