Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network

Antonin Chodron de Courcel

We study the dynamics of gradient descent in the Edge of Stability regime, where the learning rate is large enough to induce persistent oscillations in the loss and the sharpness.…