4 papers · 1 filter
Unbiased Gradient Low-Rank Projection
Rui Pan, Yang Luo, Yuxing Liu +2
Memory-efficient optimization is critical for training increasingly large language models (LLMs). A popular strategy involves gradient low-rank projection, storing only the project…
POME: Post Optimization Model Edit via Muon-style Projection
Yong Liu, Di Fu, Yang Luo +4
We introduce Post-Optimization Model Edit (POME), a new algorithm that enhances the performance of fine-tuned large language models using only their pretrained and fine-tuned check…
MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training
Yang Luo, Zangwei Zheng, Ziheng Qin +3
Large-batch training has become a cornerstone in accelerating the training of deep neural networks, yet it poses challenges in optimization and generalization. Existing optimizers…
Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
Yong Liu, Zirui Zhu, Chaoyu Gong +3
While fine-tuning large language models (LLMs) for specific tasks often yields impressive results, it comes at the cost of memory inefficiency due to back-propagation in gradient-b…