6 papers
Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining
Hai Wang, Chenhao Wang, Qifeng Cai +6
Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challengi…
Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
Jiawen Tao, Miao Peng, Yaoming Li +7
The paper introduces a pipeline that creates synthetic textbooks by clustering source material, planning hierarchical tables of contents, and assembling sections into full books, s…
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
Yuyao Wang, Zhongjian Zhang, Mo Chi +7
Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agents, as it enables them to stor…
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
Yan Wang, Qifan Zhang, Jiachen Yu +12
Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving. In this report, we propose \textbf{Lookahead…
Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping
Miao Peng, Weizhou Shen, Nuo Chen +3
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing LLMs short-context reasoning, but its performance degrades in long-context scenarios that re…
QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management
Weizhou Shen, Ziyi Yang, Chenliang Li +11
We introduce QwenLong-L1.5, a model that achieves superior long-context reasoning capabilities through systematic post-training innovations. The key technical breakthroughs of Qwen…