Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining

Michael K. Chen, Xikun Zhang, Fan Bai +2

As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, c…