Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression

Zilun Zhang, Yutao Sun, Tiancheng Zhao +4

Humans can retain old knowledge while learning new information, but Large Language Models (LLMs) often suffer from catastrophic forgetting when post-pretrained or supervised fine-t…