3 papers
cs.LG2026
SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates
Ziming Yu, Shuyao Xiao, Xingyu Zhao +6
Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making conv…
cs.LG2025
Mixture of Group Experts for Learning Invariant Representations
Lei Kang, Jia Li, Mi Tian +1
Sparsely activated Mixture-of-Experts (MoE) models effectively increase the number of parameters while maintaining consistent computational costs per token. However, vanilla MoE mo…
cs.LG2024
Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
Ziming Yu, Pan Zhou, Sike Wang +3
Fine-tuning Large Language Models (LLMs) has proven effective for a variety of downstream tasks. However, as LLMs grow in size, the memory demands for backpropagation become increa…