3 papers
cs.CL2025
FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy
Xuemiao Zhang, Feiyu Duan, Liangyu Xu +5
Large language models (LLMs) have significantly advanced human language understanding and generation, with pretraining data quality and organization being crucial to their performa…
cs.CL2025
FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training
Liangyu Xu, Xuemiao Zhang, Feiyu Duan +4
Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signal…
cs.CL2025
Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
Xuemiao Zhang, Liangyu Xu, Feiyu Duan +5
Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model's capability improves, it is intuitive that…