Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

Fırat Öncel, Salman Hussain Ali, Mirco Ravanelli +2

Large language models (LLMs) drift out of date the moment their pretraining ends, yet retraining from scratch is prohibitively expensive. Continued pretraining (CPT) is the natural…