4 papers · 1 filter
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Zhuanghua Liu, Menglian Wang, Luo Luo
Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and A…
Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2
While adaptive gradient methods are the workhorse of modern machine learning, sign-based optimization algorithms such as Lion and Muon have recently demonstrated superior empirical…
Optimal Asynchronous Stochastic Nonconvex Optimization under Heavy-Tailed Noise
Yidong Wu, Luo Luo
This paper considers the problem of asynchronous stochastic nonconvex optimization with heavy-tailed gradient noise and arbitrarily heterogeneous computation times across workers.…
Near-Optimal Decentralized Stochastic Nonconvex Optimization with Heavy-Tailed Noise
Menglian Wang, Zhuanghua Liu, Luo Luo
This paper studies decentralized stochastic nonconvex optimization problem over row-stochastic networks. We consider the heavy-tailed gradient noise which is empirically observed i…