PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective

Tim Tsz-Kit Lau, Qi Long, Weijie Su

The ever-growing scale of deep learning models and training data underscores the critical importance of efficient optimization methods. While preconditioned gradient methods such a…