Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers

Nikhil Nayak, Julia White, Urchade Zaratiana +7

Preconditioned optimizers are central to language model training, but their stochastic update rules are usually treated as direct approximations to population preconditioned descen…