Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

Zhenghao Zhao, Gaowen Liu, Zhiling Lan +1

Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it…