2 papers
cs.LG2024
Memory-Efficient LLM Training with Online Subspace Descent
Kaizhao Liang, Bo Liu, Lizhang Chen +1
Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimiz…
cs.DC2024
Communication Efficient Distributed Training with Distributed Lion
Bo Liu, Lemeng Wu, Lizhang Chen +5
The Lion optimizer has been a promising competitor with the AdamW for training large AI models, with advantages on memory, computation, and sample efficiency. In this paper, we int…