SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates

Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun +1

Modern deep learning has been shown to operate at the edge of stability, routinely using learning rates far larger than those justified by classical optimization theory. Most prior…