A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Acceleration

Li Shen, Congliang Chen, Fangyu Zou +3

Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam…