12 papers
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7
Pre-training large language models on massive GPU clusters has made hardware faults routine rather than rare, driving the need for resilient training systems. Yet existing framewor…
MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training
Ruijie Zhang, Yequan Zhao, Ziyue Liu +4
Muon has recently emerged as a strong optimizer for large language model pre-training, orthogonalizing the momentum matrix via Newton--Schulz polar iterations. A natural intuition…
When Emotion Becomes Trigger: Emotion-style dynamic Backdoor Attack Parasitising Large Language Models
Ziyu Liu, Tao Li, Tianjie Ni +5
Backdoor vulnerabilities widely exist in the fine-tuning of large language models(LLMs). Most backdoor poisoning methods operate mainly at the token level and lack deeper semantic…
MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
Yupeng Su, Ruijie Zhang, Ziyue Liu +2
The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization…
BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
Zhengyang Wang, Ziyue Liu, Ruijie Zhang +5
The scale of transformer model pre-training is constrained by the increasing computation and communication cost. Low-rank bottleneck architectures offer a promising solution to sig…
Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks
Yequan Zhao, Xinling Yu, Xian Xiao +6
Physics intelligence and digital twins often require rapid and repeated performance evaluation of various engineering systems (e.g. robots, autonomous vehicles, semiconductor chips…