6 papers
Merge before Forget: A Single LoRA Continual Learning via Continual Merging
Fuli Qiao, Mehrdad Mahdavi
Parameter-efficient continual learning has emerged as a promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tas…
Model Merging via Multi-Teacher Knowledge Distillation
Seyed Arshan Dalili, Mehrdad Mahdavi
Model merging has emerged as a lightweight alternative to joint multi-task learning (MTL), yet the generalization properties of merged models remain largely unexplored. Establishin…
On the Convergence and Stability of Distributed Sub-model Training
Yuyang Deng, Fuli Qiao, Mehrdad Mahdavi
As learning models continue to grow in size, enabling on-device local training of these models has emerged as a critical challenge in federated learning. A popular solution is sub-…
NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairment
Ajay Narayanan Sridhar, Fuli Qiao, Nelson Daniel Troncoso Aldas +4
People with visual impairments often face significant challenges in locating and retrieving objects in their surroundings. Existing assistive technologies present a trade-off: syst…
Harnessing Optimization Dynamics for Curvature-Informed Model Merging
Pouria Mahdavinia, Hamed Mahdavi, Niloofar Mireshghallah +1
Model merging is an effective post-training strategy for composing capabilities in large language models without joint retraining. We study this in the supervised fine-tuning (SFT)…
Low-rank Momentum Factorization for Memory Efficient Training
Pouria Mahdavinia, Mehrdad Mahdavi
Fine-tuning large foundation models presents significant memory challenges due to stateful optimizers like AdamW, often requiring several times more GPU memory than inference. Whil…