Low Rank Gradients and Where to Find Them

Rishi Sonthalia, Michael Murray, Guido Montúfar

This paper investigates low-rank structure in the gradients of the training loss for two-layer neural networks while relaxing the usual isotropy assumptions on the training data an…