Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise

Bingbin Liu, Rachit Bansal, Depen Morwani +3

Preconditioned methods are central to deep learning optimization. Predominant approaches include computationally light diagonal preconditioners such as Adam, which rely on gradient…