SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining

Andi Han, Jiaxiang Li, Wei Huang +4

Large language models (LLMs) have shown impressive capabilities across various tasks. However, training LLMs from scratch requires significant computational power and extensive mem…