How Far is Adam from Natural Gradient Descent?

Vihaan Paka-Hegde

Adam is the standard optimizer in deep learning, yet its geometric relationship to natural gradient descent (NGD) contains unresolved questions. We study Adam's full update rule, i…