Stochastic Gradient Methods with Block Diagonal Matrix Adaptation

Jihun Yun, Aurelie C. Lozano, Eunho Yang

Adaptive gradient approaches that automatically adjust the learning rate on a per-feature basis have been very popular for training deep networks. This rich class of algorithms inc…