Continual Pre-Training of Large Language Models: How to (re)warm your model?

Kshitij Gupta, Benjamin Thérien, Adam Ibrahim +5

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more effici…