FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

Xinyi Wang, Lirong Gao, Haobo Wang +2

Parameter-Efficient Fine-Tuning (PEFT) methods have emerged as a widely adopted strategy for adapting pre-trained Large Language Models (LLMs) to downstream tasks, significantly re…