Scaling Zero-Order Pretraining through Model Sharding

Francois Chaubard, Mykel J. Kochenderfer, Chris Ré

Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient variance grows with perturbed…