Matrix AdaGrad: Row-wise and Column-wise Adaptive Subgradient Methods

Wenpeng Zhang, Runsheng Yu, Peilin Zhao

Adaptive optimization methods such as AdaGrad and Adam are widely used in modern deep neural network training, but their adaptive scaling is primarily designed for vector-valued pa…