2 papers
cs.LG2026
Muon: Boosting Muon via Adaptive Second-Moment Preconditioning
Ziyue Liu, Ruijie Zhang, Zhengyang Wang +4
Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonaliza…
cs.LG2024
CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization
Zi Yang, Ziyue Liu, Samridhi Choudhary +4
Training large AI models such as LLMs and DLRMs costs massive GPUs and computing time. The high training cost has become only affordable to big tech companies, meanwhile also causi…