9 papers · 1 filter
From Task Mixtures to Specialized Experts
Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari +1
In collaborative foundation model fine-tuning, client data is rarely homogeneous. Instead, clients typically possess unknown mixtures of distinct data distributions, or tasks. Conv…
Conditional Optimal Bridge for Riemannian Activation Steering
Seyed Arshan Dalili, Ajay Narayanan Sridhar, Vijaykrishnan Narayanan +1
Activation steering offers a lightweight alternative to fine-tuning for controlling large language models at inference time. Many existing steering methods implicitly optimize a lo…
Correlation-Aware Contextual Bandits with Surrogate Rewards for LLM Routing
Ajay Narayanan Sridhar, Ronak Singh, Mehrdad Mahdavi +1
We study contextual bandit problems with correlated arms and access to surrogate reward signals produced by a machine learning model, motivated by applications such as large langua…
Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability
Seyed Arshan Dalili, Mehrdad Mahdavi
Sparse Autoencoders (SAEs) are widely used for mechanistic interpretability in large language models, yet their formulation assigns each latent feature a single decoder direction,…
Merge before Forget: A Single LoRA Continual Learning via Continual Merging
Fuli Qiao, Mehrdad Mahdavi
Parameter-efficient continual learning has emerged as a promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tas…
Model Merging via Multi-Teacher Knowledge Distillation
Seyed Arshan Dalili, Mehrdad Mahdavi
Model merging has emerged as a lightweight alternative to joint multi-task learning (MTL), yet the generalization properties of merged models remain largely unexplored. Establishin…
On the Convergence and Stability of Distributed Sub-model Training
Yuyang Deng, Fuli Qiao, Mehrdad Mahdavi
As learning models continue to grow in size, enabling on-device local training of these models has emerged as a critical challenge in federated learning. A popular solution is sub-…
Harnessing Optimization Dynamics for Curvature-Informed Model Merging
Pouria Mahdavinia, Hamed Mahdavi, Niloofar Mireshghallah +1
Model merging is an effective post-training strategy for composing capabilities in large language models without joint retraining. We study this in the supervised fine-tuning (SFT)…
Low-rank Momentum Factorization for Memory Efficient Training
Pouria Mahdavinia, Mehrdad Mahdavi
Fine-tuning large foundation models presents significant memory challenges due to stateful optimizers like AdamW, often requiring several times more GPU memory than inference. Whil…