4 papers · 1 filter
Which Long Documents Matter? A Controlled Study of Data Selection for Context Extension
Junlong Jia, Jiang Zhou, Xing Wu +3
Natural long documents are widely used for context extension, but which selection pipelines are effective remains unclear. We conduct a controlled comparison on Qwen2.5-3B at 128K…
AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
Yunhao Wang, Feng Zhang, Tinghao Yu +2
Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or d…
Deep Research Pretraining via Predictive Navigation
Jiang Zhou, Zhiyuan Fan, Xing Wu +3
Deep research agents are often trained on expensive, environment-grounded tool-use trajectories that require repeated retrieval, document inspection, and report evaluation. We intr…
WRAP++: Web discoveRy Amplified Pretraining
Jiang Zhou, Yunhao Wang, Xing Wu +2
Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operat…