SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates

Ziming Yu, Shuyao Xiao, Xingyu Zhao +6

Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making conv…