SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms

Farshed Abdukhakimov, Chulu Xiang, Dmitry Kamzolov +2

Adaptive optimization methods are widely recognized as among the most popular approaches for training Deep Neural Networks (DNNs). Techniques such as Adam, AdaGrad, and AdaHessian…