Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods

Wanru Zhao, Yihong Chen, Yuzhi Tang +6

Data curation is a critical yet under-explored area in large language model (LLM) training. Existing methods, such as data selection and mixing, operate in an offline paradigm, det…