Probe-Space Preconditioning for Fast and Stable Zero-Order Training

Francois Chaubard, Mykel J. Kochenderfer, Chris Ré

Backpropagation (BP) dominates deep learning but imposes a massive memory tax. For example, training OPT-30B with Adam requires $\approx$ 600GB of GPU memory (assuming batch size 8…