4 papers
CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
Peiguang Li, Yongwei Zhou, Juncheng Diao +12
Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance. Ho…
HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning
Juncheng Diao, Zhicong Lu, Peiguang Li +6
While Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks, their performance often degrades in multi-turn long-hori…
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
Yongwei Zhou, Juncheng Diao, Junlin Shang +2
The efficacy of continued pre-training for Large Language Models (LLMs) hinges upon hyperparameter configurations, such as learning rate and batch size. However, current practices…
Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
Changyuan Tian, Zhicong Lu, Huaxing Liu +7
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for advancing complex reasoning in large language models, and recent work extends RLVR to…