On the interplay between noise and curvature and its effect on optimization and generalization

Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer +3

The speed at which one can minimize an expected loss using stochastic methods depends on two properties: the curvature of the loss and the variance of the gradients. While most pre…