The Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descent

Xin Qian, Diego Klabjan

The mini-batch stochastic gradient descent (SGD) algorithm is widely used in training machine learning models, in particular deep learning models. We study SGD dynamics under linea…