1.6k citations · 3.9k across the 52 of their papers we have counts for
8 papers · 1 filter
Why are Adaptive Methods Good for Attention Models?
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit +4
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across…
AIDE: Fast and Communication Efficient Distributed Optimization
Sashank J. Reddi, Jakub Konečný, Peter Richtárik +2
In this paper, we present two new communication-efficient methods for distributed minimization of an average of functions. The first algorithm is an inexact variant of the DANE alg…
Stochastic Frank-Wolfe Methods for Nonconvex Optimization
Sashank J. Reddi, Suvrit Sra, Barnabas Poczos +1
We study Frank-Wolfe methods for nonconvex stochastic and finite-sum optimization problems. Frank-Wolfe methods (in the convex case) have gained tremendous recent interest in machi…
Fast Stochastic Methods for Nonsmooth Nonconvex Optimization
Sashank J. Reddi, Suvrit Sra, Barnabas Poczos +1
We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonconvex part is smooth and the nonsmooth part is convex. Surprisingly, unlike…
Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds
Hongyi Zhang, Sashank J. Reddi, Suvrit Sra
We study optimization of finite sums of geodesically smooth functions on Riemannian manifolds. Although variance reduction techniques for optimizing finite-sums have witnessed trem…
Stochastic Variance Reduction for Nonconvex Optimization
Sashank J. Reddi, Ahmed Hefny, Suvrit Sra +2
We study nonconvex finite-sum problems and analyze stochastic variance reduced gradient (SVRG) methods for them. SVRG and related methods have recently surged into prominence for c…