3 citations · 3 across the 5 of their papers we have counts for
Showing 2024Show all
3 papers · 1 filter
cs.LG2024★ 3 cited
Cautious Optimizers: Improving Training with One Line of Code
Kaizhao Liang, Lizhang Chen, Bo Liu +1
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcom…
cs.LG2024
Memory-Efficient LLM Training with Online Subspace Descent
Kaizhao Liang, Bo Liu, Lizhang Chen +1
Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimiz…
cs.DC2024
Communication Efficient Distributed Training with Distributed Lion
Bo Liu, Lemeng Wu, Lizhang Chen +5
The Lion optimizer has been a promising competitor with the AdamW for training large AI models, with advantages on memory, computation, and sample efficiency. In this paper, we int…