Learning Dynamics in Continual Pre-Training for Large Language Models

Xingjin Wang, Howe Tissue, Lu Wang +2

Continual Pre-Training (CPT) has become a popular and effective method to apply strong foundation models to specific downstream tasks. In this work, we explore the learning dynamic…