Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale

Fan Zhou, Zengzhi Wang, Qian Liu +2

Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. Ho…