Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations

Alec S. Xu, Can Yaras, Matthew Asato +2

Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new resear…