10 citations · 15 across the 10 of their papers we have counts for
4 papers · 1 filter
Training-Free Transformer Merging via Sequential Local Operator Alignment
Akansh Maurya, Ya-Wei Eileen Lin, Stefanie Jegelka +2
Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging…
Learning When to Adapt
Ali Zindari, Xiaowen Jiang, Rotem Mulayoff +1
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This i…
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
Ali Zindari, Rotem Mulayoff, Sebastian U. Stich
Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory a…
On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials
Rotem Mulayoff, Sebastian U. Stich
The dynamical stability of the iterates during training plays a key role in determining the minima obtained by optimization algorithms. For example, stable solutions of gradient de…