2 papers
cs.LG2026
Probe-Space Preconditioning for Fast and Stable Zero-Order Training
Francois Chaubard, Mykel J. Kochenderfer, Chris Ré
Backpropagation (BP) dominates deep learning but imposes a massive memory tax. For example, training OPT-30B with Adam requires 600GB of GPU memory (assuming batch size 8…
cs.LG2026
Scaling Zero-Order Pretraining through Model Sharding
Francois Chaubard, Mykel J. Kochenderfer, Chris Ré
Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient variance grows with perturbed…