5 papers
Convergence of Large-Stepsize SGD for the Cross-entropy Loss via Stochastic Lyapunov Stability
Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun +1
Modern deep learning has been shown to operate using learning rates far larger than those justified by classical optimization theory. Most prior analyses of this phenomenon focus o…
Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squares
Lachlan Ewen MacDonald, Hancheng Min, Leandro Palma +3
Classical optimisation theory guarantees monotonic objective decrease for gradient descent (GD) when employed in a small step size, or ``stable", regime. In contrast, gradient desc…
A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Models
Ziqing Xu, Hancheng Min, Salma Tarmoun +2
Most prior work on the convergence of gradient descent (GD) for overparameterized neural networks relies on strong assumptions on the step size (infinitesimal), the hidden-layer wi…
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization
Ziqing Xu, Hancheng Min, Lachlan Ewen MacDonald +4
Despite the empirical success of Low-Rank Adaptation (LoRA) in fine-tuning pre-trained models, there is little theoretical understanding of how first-order methods with carefully c…
Convergence and Implicit Bias of Gradient Flow on Overparametrized Linear Networks
Hancheng Min, Salma Tarmoun, Rene Vidal +1
Neural networks trained via gradient descent with random initialization and without any regularization enjoy good generalization performance in practice despite being highly overpa…