1 citations · 2 across the 4 of their papers we have counts for
5 papers
MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
Chang-Wei Shi, Xu Wang, Wu-Jun Li
The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM p…
Ordered Local Momentum for Asynchronous Distributed Learning under Arbitrary Delays
Chang-Wei Shi, Shi-Shang Wang, Wu-Jun Li
Momentum SGD (MSGD) serves as a foundational optimizer in training deep models due to momentum's key role in accelerating convergence and enhancing generalization. Meanwhile, async…
Ordered Momentum for Asynchronous SGD
Chang-Wei Shi, Yi-Rui Yang, Wu-Jun Li
Distributed learning is essential for training large-scale deep models. Asynchronous SGD (ASGD) and its variants are commonly used distributed learning methods, particularly in sce…
Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
Shen-Yi Zhao, Chang-Wei Shi, Yin-Peng Xie +1
Stochastic gradient descent~(SGD) and its variants have been the dominating optimization methods in machine learning. Compared to SGD with small-batch training, SGD with large-batc…
Global Momentum Compression for Sparse Communication in Distributed Learning
Chang-Wei Shi, Shen-Yi Zhao, Yin-Peng Xie +2
With the rapid growth of data, distributed momentum stochastic gradient descent~(DMSGD) has been widely used in distributed learning, especially for training large-scale deep model…