5 papers · 1 filter
Schattor: Schatten-family methods for deep learning optimization
Bohao Ma, Junyu Zhang, Chuan He
Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm de…
DeMuon: A Decentralized Muon for Matrix Optimization over Graphs
Chuan He, Shuyi Ren, Jingwei Mao +1
In this paper, we propose DeMuon, a method for decentralized matrix optimization over a given communication topology. DeMuon incorporates matrix orthogonalization via Newton-Schulz…
Stochastic interior-point methods for smooth conic optimization with applications
Chuan He, Zhanwang Deng
Conic optimization plays a crucial role in many machine learning (ML) problems. However, practical algorithms for conic constrained ML problems with large datasets are often limite…
Faster stochastic cubic regularized Newton methods with momentum
Yiming Yang, Chuan He, Xiao Wang +1
Cubic regularized Newton (CRN) methods have attracted signiffcant research interest because they offer stronger solution guarantees and lower iteration complexity. With the rise of…
A stochastic first-order method with multi-extrapolated momentum for highly smooth unconstrained optimization
Chuan He
In this paper, we consider an unconstrained stochastic optimization problem where the objective function exhibits high-order smoothness. Specifically, we propose a new stochastic f…