DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

Zhen Huang, Yikun Wang, Shijie Xia +1

Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the c…