8 papers
Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less
Yuxing Liu, Jianyu Wang, Tong Zhang
Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetunin…
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Dingzhi Yu, Rui Pan, Yuxing Liu +1
Sign-based optimization algorithms, such as SignSGD, have garnered significant attention for their remarkable performance in distributed learning and training large foundation mode…
Gradient Flow Drifting: Generative Modeling via Wasserstein Gradient Flows of KDE-Approximated Divergences
Jiarui Cao, Zixuan Wei, Yuxin Liu
We reveal a precise mathematical framework about a new family of generative models which we call Gradient Flow Drifting. With this framework, we prove an equivalence between the re…
ASGO: Adaptive Structured Gradient Optimization
Kang An, Yuxing Liu, Rui Pan +4
Training deep neural networks is a structured optimization problem, because the parameters are naturally represented by matrices and tensors rather than by vectors. Under this stru…
Unbiased Gradient Low-Rank Projection
Rui Pan, Yang Luo, Yuxing Liu +2
Memory-efficient optimization is critical for training increasingly large language models (LLMs). A popular strategy involves gradient low-rank projection, storing only the project…
GUIDE: Towards Scalable Advising for Research Ideas
Yaowenqi Liu, Bingxu Meng, Rui Pan +4
The field of AI research is advancing at an unprecedented pace, enabling automated hypothesis generation and experimental design across diverse domains such as biology, mathematics…