4 papers · 1 filter
Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients
Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2
Large neural networks are commonly trained across many workers, and communication between them is a major cost. Workers therefore take several local optimization steps before sendi…
Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5
Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignme…
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5
Low-rank gradient compression reduces communication in distributed training by representing updates with rank- factors. Dion is a recent method that approximates Muon, a spectra…
What do near-optimal learning rate schedules look like?
Hiroki Naganuma, Atish Agarwala, Priya Kasimbeg +1
A basic unanswered question in neural network training is: what is the best learning rate schedule shape for a given workload? The choice of learning rate schedule is a key factor…