MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning

Shu Yang, Muhammad Asif Ali, Cheng-Long Wang +2

Adapting large language models (LLMs) to new domains/tasks and enabling them to be efficient lifelong learners is a pivotal challenge. In this paper, we propose MoRAL, i.e., Mixtur…