Dynamic Memory Based Adaptive Optimization

Balázs Szegedy, Domonkos Czifra, Péter Kőrösi-Szabó

Define an optimizer as having memory $k$ if it stores $k$ dynamically changing vectors in the parameter space. Classical SGD has memory $0$, momentum SGD optimizer has $1$ and Adam…