Extending $\mu$P: Spectral Conditions for Feature Learning Across Optimizers

Akshita Gupta, Marieme Ngom, Sam Foreman +1

Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance o…