SGD with Large Step Sizes Learns Sparse Features

Maksym Andriushchenko, Aditya Varre, Loucas Pillaud-Vivien +1

We showcase important features of the dynamics of the Stochastic Gradient Descent (SGD) in the training of neural networks. We present empirical observations that commonly used lar…