Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models
Zhenghao Zhao, Gaowen Liu, Zhiling Lan +1
Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it…
cs.LG2024
Distilling Long-tailed Datasets
Zhenghao Zhao, Haoxuan Wang, Yuzhang Shang +2
Dataset distillation aims to synthesize a small, information-rich dataset from a large one for efficient model training. However, existing dataset distillation methods struggle wit…