7 papers
Muse: Representation Geometry of Muon Beyond Normalized Momentum
Da Chang, Qiankun Shi, Lvgang Zhang +4
Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization. We study this repr…
MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
Da Chang, Ganzhao Yuan
Efficient optimization is essential for training large language models. Although intra-layer selective updates have been explored, a general mechanism that enables fine-grained con…
MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration
Da Chang, Qiankun Shi, Lvgang Zhang +5
Orthogonalized-update optimizers such as Muon improve training of matrix-valued parameters, but existing extensions typically either rescale updates after orthogonalization or use…
On the Convergence of Muon and Beyond
Da Chang, Yongxiang Liu, Ganzhao Yuan +1
The Muon optimizer has demonstrated strong empirical performance in training neural networks with matrix-structured parameters. A central theoretical question is how momentum varia…
AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
Di He, Songjun Tu, Ajay Jaiswal +4
Weight decay is a standard regularization technique for training large language models (LLMs). While it is common to assign a uniform decay rate to every layer, this approach overl…
AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates
Da Chang, Yu Li, Ganzhao Yuan
In the training of large language models (LLMs), updating parameters more efficiently and stably has always been an important challenge. To achieve efficient parameter updates, exi…
Adaptive Lipschitz-Free Proximal Gradient Methods for Stochastic Composite Nonconvex Optimization
Ganzhao Yuan
We propose {\sf ALFPG}, an Adaptive Lipschitz-Free Proximal Gradient framework for stochastic composite nonconvex minimization. The framework adapts its stepsizes using information…