Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization

Yicheng Lang, Changsheng Wang, Yihua Zhang +4

Zeroth-order (ZO) optimization provides a gradient-free alternative to first-order (FO) methods by estimating gradients via finite differences of function evaluations, and has rece…