StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

Xuanlin Yang, Jia Li, Zhouchen Lin

Fine-tuning large language models at scale is increasingly limited by the memory cost of backpropagation. Zeroth-Order (ZO) optimizers reduce this burden by relying only on forward…