Inference-Native Zeroth-Order Optimization for LLMs

Zelin Li, Caiwen Ding

Zeroth-order (ZO) methods train large language models using only forward passes, yet common ZO execution paths perform substantial work beyond what the algorithm itself requires. T…