Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration

Jiahe Chen, Ziye Ma

Zeroth-order (ZO) optimization has become increasingly popular and important in fine-tuning large language models (LLMs), especially on edge devices due to its ability to adjust th…