Learning Dynamics of Deep Linear Networks Beyond the Edge of Stability

Avrajit Ghosh, Soo Min Kwon, Rongrong Wang +2

Deep neural networks trained using gradient descent with a fixed learning rate $\hat{I}\cdot$ often operate in the regime of "edge of stability" (EOS), where the largest eigenvalue of the He…