20 citations · 28 across the 15 of their papers we have counts for
Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Xiaodong Wu, Wenyi Yu, Chao Zhang +1
Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam, yet the mechanism remains incompletely understood. We investigate…
cs.LG2024★ 1 cited
An Improved Empirical Fisher Approximation for Natural Gradient Descent
Xiaodong Wu, Wenyi Yu, Chao Zhang +1
Approximate Natural Gradient Descent (NGD) methods are an important family of optimisers for deep learning models, which use approximate Fisher information matrices to pre-conditio…