SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network

Tomer Galanti, Zachary S. Siegel, Aparna Gupte +1

We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate…