Feature learning as alignment: a structural property of gradient descent in non-linear neural networks

Daniel Beaglehole, Ioannis Mitliagkas, Atish Agarwala

Understanding the mechanisms through which neural networks extract statistics from input-label pairs through feature learning is one of the most important unsolved problems in supe…