7 papers
Which Long Documents Matter? A Controlled Study of Data Selection for Context Extension
Junlong Jia, Jiang Zhou, Xing Wu +3
Natural long documents are widely used for context extension, but which selection pipelines are effective remains unclear. We conduct a controlled comparison on Qwen2.5-3B at 128K…
AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
Yunhao Wang, Feng Zhang, Tinghao Yu +2
Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or d…
Environment Evolution for Terminal Agents
Zhiyuan Fan, Tinghao Yu, Yuanjun Cai +9
Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less…
Deep Research Pretraining via Predictive Navigation
Jiang Zhou, Zhiyuan Fan, Xing Wu +3
Deep research agents are often trained on expensive, environment-grounded tool-use trajectories that require repeated retrieval, document inspection, and report evaluation. We intr…
Toward Scalable Terminal Task Synthesis via Skill Graphs
Zhiyuan Fan, Tinghao Yu, Yuanjun Cai +8
Terminal agents have demonstrated strong potential for autonomous command-line execution, yet their training remains constrained by the scarcity of high-quality and diverse executi…
WRAP++: Web discoveRy Amplified Pretraining
Jiang Zhou, Yunhao Wang, Xing Wu +2
Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operat…
PolicyLong: Towards On-Policy Context Extension
Junlong Jia, Jiang Zhou, Ziyang Chen +5
Extending LLM context windows is hindered by scarce high-quality long-context data. Recent methods synthesize data with genuine long-range dependencies via information-theoretic ve…