Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity

Arthur Jacot, François Ged, Berfin Şimşek +2

The dynamics of Deep Linear Networks (DLNs) is dramatically affected by the variance ${\sigma }^2$ of the parameters at initialization ${\theta }_0$. For DLNs of width $w$, we show a phase transit…