Generalization at the Edge of Stability

Mario Tuci, Caner Korkmaz, Umut Şimşekli +1

Training modern neural networks often relies on large learning rates, operating at the edge of stability, where the optimization dynamics exhibit oscillatory and chaotic behavior.…