GWT: Scalable Optimizer State Compression for Large Language Model Training

Ziqing Wen, Ping Luo, Jiahuan Wang +4

Training large language models (LLMs) requires substantial memory, a significant fraction of which is consumed by the moment states maintained by adaptive optimizers such as Adam.…