Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon

Sifan Yang, Hongyi Tao, Dingzhi Yu +5

Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization performance. Previous algorithm…