1 citations · 1 across the 8 of their papers we have counts for
8 papers
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Dingzhi Yu, Rui Pan, Yuxing Liu +2
Sign-based optimization algorithms, such as SignSGD, have garnered attention for their performance in distributed learning and training large foundation models. Despite their empir…
Unbiased Gradient Low-Rank Projection
Rui Pan, Yang Luo, Yuxing Liu +2
Memory-efficient optimization is critical for training increasingly large language models (LLMs). A popular strategy involves gradient low-rank projection, storing only the project…
Theoretical Analysis on how Learning Rate Warmup Accelerates Convergence
Yuxing Liu, Yuze Ge, Rui Pan +2
Learning rate warmup is a popular and practical technique in training large-scale deep neural networks. Despite the huge success in practice, the theoretical advantages of this str…
GUIDE: Towards Scalable Advising for Research Ideas
Yaowenqi Liu, Bingxu Meng, Rui Pan +4
The field of AI research is advancing at an unprecedented pace, enabling automated hypothesis generation and experimental design across diverse domains such as biology, mathematics…
ASGO: Adaptive Structured Gradient Optimization
Kang An, Yuxing Liu, Rui Pan +4
Training deep neural networks is a structured optimization problem, because the parameters are naturally represented by matrices and tensors rather than by vectors. Under this stru…
Fully First-Order Methods for Decentralized Bilevel Optimization
Xiaoyu Wang, Xuxing Chen, Shiqian Ma +1
This paper focuses on decentralized stochastic bilevel optimization (DSBO) where agents only communicate with their neighbors. We propose Decentralized Stochastic Gradient Descent…
AdaGrad under Anisotropic Smoothness
Yuxing Liu, Rui Pan, Tong Zhang
Adaptive gradient methods have been widely adopted in training large-scale deep neural networks, especially large foundation models. Despite the huge success in practice, their the…
Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise
Rui Pan, Yuxing Liu, Xiaoyu Wang +1
Heavy-ball momentum with decaying learning rates is widely used with SGD for optimizing deep learning models. In contrast to its empirical popularity, the understanding of its theo…