4 papers
MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling
Ning Ding, Fangcheng Liu, Kyungrae Kim +4
Scaling Large Language Models (LLMs) typically relies on increasing the number of parameters or test-time computations to boost performance. However, these strategies are impractic…
Data-driven Clustering and Merging of Adapters for On-device Large Language Models
Ondrej Bohdal, Taha Ceritli, Mete Ozay +4
On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks. While storing all available adapters is impr…
On-device System of Compositional Multi-tasking in Large Language Models
Ondrej Bohdal, Konstantinos Theodosiadis, Asterios Mpatziakas +10
Large language models (LLMs) are commonly adapted for diverse downstream tasks via parameter-efficient fine-tuning techniques such as Low-Rank Adapters (LoRA). While adapters can b…
HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Merging
Taha Ceritli, Ondrej Bohdal, Mete Ozay +4
Large language models (LLMs) often leverage adapters, such as low-rank-based adapters, to achieve strong performance on downstream tasks. However, storing a separate adapter for ea…