Training trajectories, mini-batch losses and the curious role of the learning rate

Mark Sandler, Andrey Zhmoginov, Max Vladymyrov +1

Stochastic gradient descent plays a fundamental role in nearly all applications of deep learning. However its ability to converge to a global minimum remains shrouded in mystery. I…