10 papers
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Sifan Yang, Hongyi Tao, Dingzhi Yu +5
Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization performance. Previous algorithm…
Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2
AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying…
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on -norm Lower Bounds
Hongyi Tao, Dingzhi Yu, Lijun Zhang
Sign-based optimization algorithms, such as SignSGD and Muon, have garnered significant attention for their remarkable performance in training large foundation models. Despite this…
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Dingzhi Yu, Rui Pan, Yuxing Liu +2
Sign-based optimization algorithms, such as SignSGD, have garnered attention for their performance in distributed learning and training large foundation models. Despite their empir…
Improved Approximate Regret for Decentralized Online Continuous Submodular Maximization via Reductions
Yuanyu Wan, Yu Shen, Dingzhi Yu +2
To expand the applicability of decentralized online learning, previous studies have proposed several algorithms for decentralized online continuous submodular maximization (D-OCSM)…
Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2
While adaptive gradient methods are the workhorse of modern machine learning, sign-based optimization algorithms such as Lion and Muon have recently demonstrated superior empirical…
Better Convergence Guarantees for Sign-Based Momentum Methods
Wei Jiang, Dingzhi Yu, Sifan Yang +3
This paper presents an improved analysis for sign-based methods with momentum updates. Traditional sign-based methods obtain a convergence rate of under the…
Group Distributionally Robust Optimization with Flexible Sample Queries
Haomin Bai, Dingzhi Yu, Shuai Li +2
Group distributionally robust optimization (GDRO) aims to develop models that perform well across distributions simultaneously. Existing GDRO algorithms can only process a fixe…
Mirror Descent Under Generalized Smoothness
Dingzhi Yu, Wei Jiang, Hongyi Tao +2
Smoothness is crucial for attaining fast rates in first-order optimization. However, many optimization problems in modern machine learning involve non-smooth objectives. Recent stu…
Efficient Algorithms for Empirical Group Distributionally Robust Optimization and Beyond
Dingzhi Yu, Yunuo Cai, Wei Jiang +1
In this paper, we investigate the empirical counterpart of Group Distributionally Robust Optimization (GDRO), which aims to minimize the maximal empirical risk across distinct…