14 papers
Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Haodong Wen, Yiran Zhang, Yingfa Chen +1
While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an…
The Power of Power Law: Asymmetry Enables Compositional Reasoning
Zixuan Wang, Xingyu Dang, Jason D. Lee +1
Natural language data follows a power-law distribution, with most knowledge and skills appearing at very low frequency. While a common intuition suggests that reweighting or curati…
Building Multi-Task Agentic LLMs via Two-Phase Distillation
Huaijie Wang, Shusheng Xu, Yi Wu +1
A key step toward artificial general intelligence is to train models that can perform multiple tasks. In this paper, we study how to build such models by first training separate RL…
Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
Jinhan Li, Kexian Tang, Yihan Xu +2
To achieve deeper safety alignment for large language models (LLMs), recent efforts have studied how to push safety interventions earlier into the pretraining stage, primarily by f…
Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming
Tingqiang Xu, Hangrui Zhou, Tianle Cai +2
Despite strong performance in competitive programming, the role of Large Language Models (LLMs) in supporting human learning in the same setting remains largely unexplored. In this…
How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
Kairong Luo, Zhenbo Sun, Haodong Wen +5
Due to the scarcity of high-quality data, large language models (LLMs) are often trained on mixtures of data with varying quality levels, even after sophisticated data curation. A…