3 citations · 3 across the 12 of their papers we have counts for
4 papers · 1 filter
Cautious Optimizers: Improving Training with One Line of Code
Kaizhao Liang, Lizhang Chen, Bo Liu +1
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcom…
Memory-Efficient LLM Training with Online Subspace Descent
Kaizhao Liang, Bo Liu, Lizhang Chen +1
Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimiz…
Memory-Efficient Optimization with Factorized Hamiltonian Descent
Son Nguyen, Lizhang Chen, Bo Liu +1
Modern deep learning heavily depends on adaptive optimizers such as Adam and its variants, which are renowned for their capacity to handle model scaling and streamline hyperparamet…
Communication Efficient Distributed Training with Distributed Lion
Bo Liu, Lemeng Wu, Lizhang Chen +5
The Lion optimizer has been a promising competitor with the AdamW for training large AI models, with advantages on memory, computation, and sample efficiency. In this paper, we int…