Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

Sijia Liu, Yicheng Lang, Soumyadeep Pal +6

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory…