71 citations · 558 across the 67 of their papers we have counts for
12 papers · 1 filter
Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clipping
Umut Şimşekli, Mert Gürbüzbalaban, Sinan Yıldırım +1
The injection of heavy-tailed noise into the iterates of stochastic gradient descent (SGD) has garnered growing interest in recent years due to its theoretical and empirical benefi…
Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent
Lingjiong Zhu, Mert Gurbuzbalaban, Anant Raj +1
Algorithmic stability is an important notion that has proven powerful for deriving generalization bounds for practical algorithms. The last decade has witnessed an increasing numbe…
Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize
Mert Gürbüzbalaban, Yuanhan Hu, Umut Şimşekli +1
Cyclic and randomized stepsizes are widely used in the deep learning practice and can often outperform standard stepsize choices such as constant stepsize in SGD. Despite their emp…
Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions
Anant Raj, Lingjiong Zhu, Mert Gürbüzbalaban +1
Heavy-tail phenomena in stochastic gradient descent (SGD) have been reported in several empirical studies. Experimental evidence in previous works suggests a strong interplay betwe…
Penalized Overdamped and Underdamped Langevin Monte Carlo Algorithms for Constrained Sampling
Mert Gürbüzbalaban, Yuanhan Hu, Lingjiong Zhu
We consider the constrained sampling problem where the goal is to sample from a target distribution when is constrained to lie on a convex body $\mathca…
Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares
Anant Raj, Melih Barsbey, Mert Gürbüzbalaban +2
Recent studies have shown that heavy tails can emerge in stochastic optimization and that the heaviness of the tails have links to the generalization error. While these studies hav…
Heavy-Tail Phenomenon in Decentralized SGD
Mert Gurbuzbalaban, Yuanhan Hu, Umut Simsekli +2
Recent theoretical studies have shown that heavy-tails can emerge in stochastic optimization due to `multiplicative noise', even under surprisingly simple settings, such as linear…
Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms
Alexander Camuto, George Deligiannidis, Murat A. Erdogdu +3
Understanding generalization in deep learning has been one of the major challenges in statistical learning theory over the last decade. While recent work has illustrated that the d…
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Alexander Camuto, Xiaoyu Wang, Lingjiong Zhu +3
Gaussian noise injections (GNIs) are a family of simple and widely-used regularisation methods for training neural networks, where one injects additive or multiplicative Gaussian n…
Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo
Mert Gürbüzbalaban, Xuefeng Gao, Yuanhan Hu +1
Stochastic gradient Langevin dynamics (SGLD) and stochastic gradient Hamiltonian Monte Carlo (SGHMC) are two popular Markov Chain Monte Carlo (MCMC) algorithms for Bayesian inferen…
Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
Umut Şimşekli, Lingjiong Zhu, Yee Whye Teh +1
Stochastic gradient descent with momentum (SGDm) is one of the most popular optimization algorithms in deep learning. While there is a rich theory of SGDm for convex problems, the…
Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distances
Bugra Can, Mert Gurbuzbalaban, Lingjiong Zhu
Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in mac…