SiRA: Sparse Mixture of Low Rank Adaptation

Yun Zhu, Nevan Wichers, Chu-Cheng Lin +8

Parameter Efficient Tuning has been an prominent approach to adapt the Large Language Model to downstream tasks. Most previous works considers adding the dense trainable parameters…