Practical Efficiency of Muon for Pretraining

Essential AI, :, Ishaan Shah +22

We demonstrate that Muon, the simplest instantiation of a second-order optimizer, explicitly expands the Pareto frontier over AdamW on the compute-time tradeoff. We find that Muon…