10 papers
Mirror Descent Under Generalized Smoothness
Dingzhi Yu, Wei Jiang, Hongyi Tao +2
Smoothness is crucial for attaining fast rates in first-order optimization. However, many optimization problems in modern machine learning involve non-smooth objectives. Recent stu…
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on -norm Lower Bounds
Hongyi Tao, Dingzhi Yu, Lijun Zhang
Sign-based optimization algorithms, such as SignSGD and Muon, have garnered significant attention for their remarkable performance in training large foundation models. Despite this…
Parameter-Free Clustering via Self-Supervised Consensus Maximization (Extended Version)
Lijun Zhang, Suyuan Liu, Siwei Wang +4
Clustering is a fundamental task in unsupervised learning, but most existing methods heavily rely on hyperparameters such as the number of clusters or other sensitive settings, lim…
ROAST: Rollout-based On-distribution Activation Steering Technique
Xuanbo Su, Hao Luo, Yingfang Zhang +1
Activation steering provides parameter-efficient control over large language models (LLMs) at inference time, but many methods rely on off-distribution supervision and discrete mas…
Distributed Online Convex Optimization with Efficient Communication: Improved Algorithm and Lower bounds
Sifan Yang, Wenhao Yang, Wei Jiang +1
We investigate distributed online convex optimization with compressed communication, where learners connected by a network collaboratively minimize a sequence of global loss fu…
Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings
Wei Jiang, Lijun Zhang
In this paper, we analyze the convergence properties of the Lion optimizer. First, we establish that the Lion optimizer attains a convergence rate of …