Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation

Markus Gross, Arne P. Raulf, Christoph Räth

We investigate the stationary (late-time) training regime of single- and two-layer underparameterized linear neural networks within the continuum limit of stochastic gradient desce…