9 citations · 15 across the 13 of their papers we have counts for
7 papers · 1 filter
Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts
Sanae Lotfi, Lucas Caccia, Alessandro Sordoni +2
While large language models (LLMs) fine-tuned with lightweight adapters achieve strong performance across diverse tasks, their performance on individual tasks depends on the fine-t…
Learning to Solve Complex Problems via Dataset Decomposition
Wanru Zhao, Lucas Caccia, Zhengyan Shi +3
Curriculum learning is a class of training strategies that organizes the data being exposed to a model by difficulty, gradually from simpler to more complex examples. This research…
Learning to Extract Context for Context-Aware LLM Inference
Minseon Kim, Lucas Caccia, Zhengyan Shi +4
User prompts to large language models (LLMs) are often ambiguous or under-specified, and subtle contextual cues shaped by user intentions, prior knowledge, and risk factors strongl…
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
Samin Yeasar Arnob, Zhan Su, Minseon Kim +6
Merging parameter-efficient task experts has recently gained growing attention as a way to build modular architectures that can be rapidly adapted on the fly for specific downstrea…
Training Plug-n-Play Knowledge Modules with Deep Context Distillation
Lucas Caccia, Alan Ansell, Edoardo Ponti +2
Dynamically integrating new or rapidly evolving information after (Large) Language Model pre-training remains challenging, particularly in low-data scenarios or when dealing with p…
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
Prateek Yadav, Colin Raffel, Mohammed Muqeeth +6
The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task. Model MoErging methods…
Towards Modular LLMs by Building and Reusing a Library of LoRAs
Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti +5
The growing number of parameter-efficient adaptations of a base large language model (LLM) calls for studying whether we can reuse such trained adapters to improve performance for…