1 citations · 1 across the 12 of their papers we have counts for
13 papers
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Sifan Yang, Hongyi Tao, Dingzhi Yu +5
Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization performance. Previous algorithm…
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on -norm Lower Bounds
Hongyi Tao, Dingzhi Yu, Lijun Zhang
Sign-based optimization algorithms, such as SignSGD and Muon, have garnered significant attention for their remarkable performance in training large foundation models. Despite this…
Watch the Model Think: On-Policy Extraction of Activation Steering Vectors
Xuanbo Su, Hao Luo, Yingfang Zhang +4
When a model solves a problem on one attempt and fails it on the next, what separates the two is rarely the final answer token; it is the trajectory that reached it. Contrastive ac…
Distributed Online Convex Optimization with Efficient Communication: Improved Algorithm and Lower bounds
Sifan Yang, Wenhao Yang, Wei Jiang +1
We investigate distributed online convex optimization with compressed communication, where learners connected by a network collaboratively minimize a sequence of global loss fu…
Parameter-Free Clustering via Self-Supervised Consensus Maximization (Extended Version)
Lijun Zhang, Suyuan Liu, Siwei Wang +4
Clustering is a fundamental task in unsupervised learning, but most existing methods heavily rely on hyperparameters such as the number of clusters or other sensitive settings, lim…
Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings
Wei Jiang, Lijun Zhang
In this paper, we analyze the convergence properties of the Lion optimizer. First, we establish that the Lion optimizer attains a convergence rate of …
Dual Adaptivity: Universal Algorithms for Minimizing the Adaptive Regret of Convex Functions
Lijun Zhang, Wenhao Yang, Guanghui Wang +2
To deal with changing environments, a new performance measure -- adaptive regret, defined as the maximum static regret over any interval, was proposed in online learning. Under the…
Better Convergence Guarantees for Sign-Based Momentum Methods
Wei Jiang, Dingzhi Yu, Sifan Yang +3
This paper presents an improved analysis for sign-based methods with momentum updates. Traditional sign-based methods obtain a convergence rate of under the…
Group Distributionally Robust Optimization with Flexible Sample Queries
Haomin Bai, Dingzhi Yu, Shuai Li +2
Group distributionally robust optimization (GDRO) aims to develop models that perform well across distributions simultaneously. Existing GDRO algorithms can only process a fixe…
Mirror Descent Under Generalized Smoothness
Dingzhi Yu, Wei Jiang, Hongyi Tao +2
Smoothness is crucial for attaining fast rates in first-order optimization. However, many optimization problems in modern machine learning involve non-smooth objectives. Recent stu…
Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptions
Wei Jiang, Sifan Yang, Yibo Wang +1
This paper explores adaptive variance reduction methods for stochastic optimization based on the STORM technique. Existing adaptive extensions of STORM rely on strong assumptions l…
Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction
Wei Jiang, Sifan Yang, Wenhao Yang +1
Sign stochastic gradient descent (signSGD) is a communication-efficient method that transmits only the sign of stochastic gradients for parameter updating. Existing literature has…
Efficient Algorithms for Empirical Group Distributionally Robust Optimization and Beyond
Dingzhi Yu, Yunuo Cai, Wei Jiang +1
In this paper, we investigate the empirical counterpart of Group Distributionally Robust Optimization (GDRO), which aims to minimize the maximal empirical risk across distinct…