When Does Preconditioning Help or Hurt Generalization?

Shun-ichi Amari, Jimmy Ba, Roger Grosse +5

While second order optimizers such as natural gradient descent (NGD) often speed up optimization, their effect on generalization has been called into question. This work presents a…