Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs

Kairun Zhang, Haoyu Li, Yanjun Zhao +2

Zeroth-order optimizers have recently emerged as an attractive approach for fine-tuning large language models (LLMs), as they avoid backpropagation and can substantially reduce mem…