3 papers
cs.CL2026
Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
Zijun Wang, Haoqin Tu, Weidong Zhou +7
Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretrainin…
cs.LG2025
TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-training
Yifan Wang, Binbin Liu, Fengze Liu +6
The data mixture used in the pre-training of a language model is a cornerstone of its final performance. However, a static mixing strategy is suboptimal, as the model's learning pr…
cs.CL2025
QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining
Fengze Liu, Weidong Zhou, Binbin Liu +8
Quality and diversity are two critical metrics for the training data of large language models (LLMs), positively impacting performance. Existing studies often optimize these metric…