activity
20102026
most citedOn Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

571 citations · 1.7k across the 33 of their papers we have counts for

collaborators
Showing 2026Show all

1 paper · 1 filter

cs.LG2026

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

Mikail Khona, Aditya Vavre, Boxiang Wang +11

Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale. I…