7 papers
BackPlay: Head-Only Look-Back Self-Correction for Diffusion Language Models
Liming Liu, Binxuan Huang, Zixuan Zhang +3
Diffusion Language Models (DLMs) decode multiple tokens in parallel, but aggressive multi-token decoding amplifies cross-token dependency errors and can sharply degrade generation…
Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
Zichong Li, Chen Liang, Liliang Ren +3
Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding, such as retrieval-augmented generation and multi-document reasoning.…
Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning
Siyuan Xu, Shiyang Li, Xin Liu +9
Existing synthetic tool-use corpora are primarily designed for offline supervised fine-tuning, yet reinforcement learning (RL) requires executable environments that support reward-…
COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs
Liming Liu, Zhenghao Xu, Zixuan Zhang +5
Large Language Models (LLMs) have demonstrated remarkable success across various domains, yet their optimization remains a significant challenge due to the complex and high-dimensi…
NorMuon: Making Muon more efficient and scalable
Zichong Li, Liming Liu, Chen Liang +2
The choice of optimizer significantly impacts the training efficiency and computational costs of large language models (LLMs). Recently, the Muon optimizer has demonstrated promisi…
LLMs Can Generate a Better Answer by Aggregating Their Own Responses
Zichong Li, Xinyu Feng, Yuheng Cai +6
Large Language Models (LLMs) have shown remarkable capabilities across tasks, yet they often require additional prompting techniques when facing complex problems. While approaches…