5 papers
Unbiased Gradient Low-Rank Projection
Rui Pan, Yang Luo, Yuxing Liu +2
Memory-efficient optimization is critical for training increasingly large language models (LLMs). A popular strategy involves gradient low-rank projection, storing only the project…
Theoretical Analysis on how Learning Rate Warmup Accelerates Convergence
Yuxing Liu, Yuze Ge, Rui Pan +2
Learning rate warmup is a popular and practical technique in training large-scale deep neural networks. Despite the huge success in practice, the theoretical advantages of this str…
GUIDE: Towards Scalable Advising for Research Ideas
Yaowenqi Liu, Bingxu Meng, Rui Pan +4
The field of AI research is advancing at an unprecedented pace, enabling automated hypothesis generation and experimental design across diverse domains such as biology, mathematics…
ASGO: Adaptive Structured Gradient Optimization
Kang An, Yuxing Liu, Rui Pan +4
Training deep neural networks is a structured optimization problem, because the parameters are naturally represented by matrices and tensors rather than by vectors. Under this stru…
Fully First-Order Methods for Decentralized Bilevel Optimization
Xiaoyu Wang, Xuxing Chen, Shiqian Ma +1
This paper focuses on decentralized stochastic bilevel optimization (DSBO) where agents only communicate with their neighbors. We propose Decentralized Stochastic Gradient Descent…