It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7

Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--for instance, they contain litt…