Exploring Forgetting in Large Language Model Pre-Training

Chonghua Liao, Ruobing Xie, Xingwu Sun +2

Catastrophic forgetting remains a formidable obstacle to building an omniscient model in large language models (LLMs). Despite the pioneering research on task-level forgetting in L…