Learning Attentional Mixture of LoRAs for Language Model Continual Learning

Jialin Liu, Jianhua Wu, Jie Liu +1

Fine-tuning large language models (LLMs) with Low-Rank adaption (LoRA) is widely acknowledged as an effective approach for continual learning for new tasks. However, it often suffe…