Self-Improving Pretraining: using post-trained models to pretrain better models

Ellen Xiaoqing Tan, Jack Lanchantin, Shehzaad Dhuliawala +9

Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates…