A Guide Through the Zoo of Biased SGD

Yury Demidovich, Grigory Malinovsky, Igor Sokolov +1

Stochastic Gradient Descent (SGD) is arguably the most important single algorithm in modern machine learning. Although SGD with unbiased gradient estimators has been studied extens…