4 papers · 1 filter
Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squares
Lachlan Ewen MacDonald, Hancheng Min, Leandro Palma +3
Classical optimisation theory guarantees monotonic objective decrease for gradient descent (GD) when employed in a small step size, or ``stable", regime. In contrast, gradient desc…
A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Models
Ziqing Xu, Hancheng Min, Salma Tarmoun +2
Most prior work on the convergence of gradient descent (GD) for overparameterized neural networks relies on strong assumptions on the step size (infinitesimal), the hidden-layer wi…
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization
Ziqing Xu, Hancheng Min, Lachlan Ewen MacDonald +4
Despite the empirical success of Low-Rank Adaptation (LoRA) in fine-tuning pre-trained models, there is little theoretical understanding of how first-order methods with carefully c…
Convergence and Implicit Bias of Gradient Flow on Overparametrized Linear Networks
Hancheng Min, Salma Tarmoun, Rene Vidal +1
Neural networks trained via gradient descent with random initialization and without any regularization enjoy good generalization performance in practice despite being highly overpa…