4 papers · 1 filter
BackPlay: Head-Only Look-Back Self-Correction for Diffusion Language Models
Liming Liu, Binxuan Huang, Zixuan Zhang +3
Diffusion Language Models (DLMs) decode multiple tokens in parallel, but aggressive multi-token decoding amplifies cross-token dependency errors and can sharply degrade generation…
COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs
Liming Liu, Zhenghao Xu, Zixuan Zhang +5
Large Language Models (LLMs) have demonstrated remarkable success across various domains, yet their optimization remains a significant challenge due to the complex and high-dimensi…
NorMuon: Making Muon more efficient and scalable
Zichong Li, Liming Liu, Chen Liang +2
The choice of optimizer significantly impacts the training efficiency and computational costs of large language models (LLMs). Recently, the Muon optimizer has demonstrated promisi…
A Minimalist Example of Edge-of-Stability and Progressive Sharpening
Liming Liu, Zixuan Zhang, Simon Du +1
Recent advances in deep learning optimization have unveiled two intriguing phenomena under large learning rates: Edge of Stability (EoS) and Progressive Sharpening (PS), challengin…