10 citations · 15 across the 10 of their papers we have counts for
9 papers · 1 filter
Training-Free Transformer Merging via Sequential Local Operator Alignment
Akansh Maurya, Ya-Wei Eileen Lin, Stefanie Jegelka +2
Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging…
Learning When to Adapt
Ali Zindari, Xiaowen Jiang, Rotem Mulayoff +1
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This i…
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
Ali Zindari, Rotem Mulayoff, Sebastian U. Stich
Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory a…
On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials
Rotem Mulayoff, Sebastian U. Stich
The dynamical stability of the iterates during training plays a key role in determining the minima obtained by optimization algorithms. For example, stable solutions of gradient de…
The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank
Amitay Bar, Rotem Mulayoff, Tomer Michaeli +1
Langevin dynamics (LD) is widely used for sampling from distributions and for optimization. In this work, we derive a closed-form expression for the expected loss of preconditioned…
The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie +2
We study the type of solutions to which stochastic gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss. Our result…
Exact Mean Square Linear Stability Analysis for SGD
Rotem Mulayoff, Tomer Michaeli
The dynamical stability of optimization methods at the vicinity of minima of the loss has recently attracted significant attention. For gradient descent (GD), stable convergence is…
Spectral Discovery of Jointly Smooth Features for Multimodal Data
Felix Dietrich, Or Yair, Rotem Mulayoff +2
In this paper, we propose a spectral method for deriving functions that are jointly smooth on multiple observed manifolds. This allows us to register measurements of the same pheno…
Unique Properties of Flat Minima in Deep Networks
Rotem Mulayoff, Tomer Michaeli
It is well known that (stochastic) gradient descent has an implicit bias towards flat minima. In deep neural network training, this mechanism serves to screen out minima. However,…