9 papers
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Zhuanghua Liu, Menglian Wang, Luo Luo
Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and A…
Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2
While adaptive gradient methods are the workhorse of modern machine learning, sign-based optimization algorithms such as Lion and Muon have recently demonstrated superior empirical…
Optimal Asynchronous Stochastic Nonconvex Optimization under Heavy-Tailed Noise
Yidong Wu, Luo Luo
This paper considers the problem of asynchronous stochastic nonconvex optimization with heavy-tailed gradient noise and arbitrarily heterogeneous computation times across workers.…
Near-Optimal Decentralized Stochastic Nonconvex Optimization with Heavy-Tailed Noise
Menglian Wang, Zhuanghua Liu, Luo Luo
This paper studies decentralized stochastic nonconvex optimization problem over row-stochastic networks. We consider the heavy-tailed gradient noise which is empirically observed i…
Accelerated Evolving Set Processes for Local PageRank Computation
Binbin Huang, Luo Luo, Yanghua Xiao +2
This work proposes a novel framework based on nested evolving set processes to accelerate Personalized PageRank (PPR) computation. At each stage of the process, we employ a localiz…
Explicit Global Convergence Rates of BFGS without Line Search
Jianjiang Yu, Weiguo Gao, Luo Luo
This paper studies the convergence rates of the Broyden--Fletcher--Goldfarb--Shanno~(BFGS) method without line search. We show that the BFGS method with an adaptive step size [Gao…
A Parameter-Free and Near-Optimal Zeroth-Order Algorithm for Stochastic Convex Optimization
Kunjie Ren, Luo Luo
This paper considers zeroth-order optimization for stochastic convex minimization problem. We propose a parameter-free stochastic zeroth-order method (POEM) by introducing a step-s…
Optimizing over Multiple Distributions under Generalized Quasar-Convexity Condition
Shihong Ding, Long Yang, Luo Luo +1
We study a typical optimization model where the optimization variable is composed of multiple probability distributions. Though the model appears frequently in practice, such as fo…
Decentralized Gradient-Free Methods for Stochastic Non-Smooth Non-Convex Optimization
Zhenwei Lin, Jingfan Xia, Qi Deng +1
We consider decentralized gradient-free optimization of minimizing Lipschitz continuous functions that satisfy neither smoothness nor convexity assumption. We propose two novel gra…