106 citations · 499 across the 45 of their papers we have counts for
6 papers · 1 filter
On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
Umut Şimşekli, Mert Gürbüzbalaban, Thanh Huy Nguyen +2
The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the \emph{classical} central…
Randomized Gossiping with Effective Resistance Weights: Performance Guarantees and Applications
Bugra Can, Saeed Soori, Necdet Serhat Aybat +2
The effective resistance between a pair of nodes in a weighted undirected graph is defined as the potential difference induced when a unit current is injected at one node and extra…
ASYNC: A Cloud Engine with Asynchrony and History for Distributed Machine Learning
Saeed Soori, Bugra Can, Mert Gurbuzbalaba +1
ASYNC is a framework that supports the implementation of asynchrony and history for optimization methods on distributed computing platforms. The popularity of asynchronous optimiza…
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
Umut Simsekli, Levent Sagun, Mert Gurbuzbalaban
The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit…
A Universally Optimal Multistage Accelerated Stochastic Gradient Method
Necdet Serhat Aybat, Alireza Fallah, Mert Gurbuzbalaban +1
We study the problem of minimizing a strongly convex, smooth function when we have noisy estimates of its gradient. We propose a novel multistage accelerated algorithm that is univ…
Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distances
Bugra Can, Mert Gurbuzbalaban, Lingjiong Zhu
Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in mac…