SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture

Jiayi Han, Liang Du, Hongwei Du +4

Although many efforts have been made, it is still a challenge to balance the training budget, downstream performance, and the general capabilities of the LLMs in many applications.…