Guaranteed Sufficient Decrease for Variance Reduced Stochastic Gradient Descent

Fanhua Shang, Yuanyuan Liu, James Cheng +2

In this paper, we propose a novel sufficient decrease technique for variance reduced stochastic gradient descent methods such as SAG, SVRG and SAGA. In order to make sufficient dec…