LLM Zeroth-Order Fine-Tuning is an Inference Workload

Zelin Li, Caiwen Ding

Zeroth-order (ZO) fine-tuning is attractive for large language models because it replaces backpropagation with forward objective evaluations. Existing implementations nevertheless…