3 papers
cs.LG2026
A Quadratic Lens on Muon's Polar Update: Benefits, Pitfalls, and the Role of Newton-Schulz
Egor Shulgin, Sam Laing, Antonio Orvieto +1
Muon updates matrix-valued parameters along approximately orthogonalized directions and often trains large language models faster than AdamW, yet it remains unclear when and why or…
cs.LG2026
Amortizing Scaling Law Construction Costs
Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6
Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and paramete…
cs.LG2025
Adam Simplified: Bias Correction Debunked
Sam Laing, Antonio Orvieto
The Adam optimizer is a cornerstone of modern deep learning, yet the empirical necessity of each of its individual components is often taken for granted. This paper presents a focu…