3 papers
cs.LG2026
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Sifan Yang, Hongyi Tao, Dingzhi Yu +5
Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization performance. Previous algorithm…
cs.LG2026
Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2
AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying…
cs.LG2026
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on -norm Lower Bounds
Hongyi Tao, Dingzhi Yu, Lijun Zhang
Sign-based optimization algorithms, such as SignSGD and Muon, have garnered significant attention for their remarkable performance in training large foundation models. Despite this…