LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

Shihan Dou, Enyu Zhou, Yan Liu +13

Supervised fine-tuning (SFT) is a crucial step for large language models (LLMs), enabling them to align with human instructions and enhance their capabilities in downstream tasks.…