1.6k citations · 1.7k across the 12 of their papers we have counts for
Showing 2019Show all
2 papers · 1 filter
math.OC2019
Why are Adaptive Methods Good for Attention Models?
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit +4
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across…
cs.LG2019★ 1.6k cited
On the Convergence of Adam and Beyond
Sashank J. Reddi, Satyen Kale, Sanjiv Kumar
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradi…