Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

Nicholas Roberts, Ahmed Ahmed, Suhas Kotha +4

Synthetic rewrites, where a language model rewrites a natural document e.g. into a target style, are now a standard ingredient of pretraining corpora. However, critical aspects of…