106 citations · 499 across the 45 of their papers we have counts for
47 papers
Rényi Differential Privacy for Heavy-Tailed SDEs via Fractional Poincaré Inequalities
Benjamin Dupuis, Mert Gürbüzbalaban, Umut Şimşekli +3
Characterizing the differential privacy (DP) of learning algorithms has become a major challenge in recent years. In parallel, many studies suggested investigating the behavior of…
DIGing--SGLD: Decentralized and Scalable Langevin Sampling over Time--Varying Networks
Waheed U. Bajwa, Mert Gurbuzbalaban, Mustafa Ali Kutbay +2
Sampling from a target distribution induced by training data is central to Bayesian learning, with Stochastic Gradient Langevin Dynamics (SGLD) serving as a key tool for scalable p…
Anchored Langevin Algorithms
Mert Gurbuzbalaban, Hoang M. Nguyen, Xicheng Zhang +1
Standard first-order Langevin algorithms such as the unadjusted Langevin algorithm (ULA) are obtained by discretizing the Langevin diffusion and are widely used for sampling in mac…
Accelerated Gradient Methods with Biased Gradient Estimates: Risk Sensitivity, High-Probability Guarantees, and Large Deviation Bounds
Mert Gürbüzbalaban, Yasa Syed, Necdet Serhat Aybat
We study trade-offs between convergence rate and robustness to gradient errors in the context of first-order methods. Our focus is on generalized momentum methods (GMMs)--a broad c…
High-Order Langevin Monte Carlo Algorithms
Thanh Dang, Mert Gurbuzbalaban, Mohammad Rafiqul Islam +2
Langevin algorithms are popular Markov chain Monte Carlo (MCMC) methods for large-scale sampling problems that often arise in data science. We propose Monte Carlo algorithms based…
Algorithmic Stability of Stochastic Gradient Descent with Momentum under Heavy-Tailed Noise
Thanh Dang, Melih Barsbey, A K M Rokonuzzaman Sonet +3
Understanding the generalization properties of optimization algorithms under heavy-tailed noise has gained growing attention. However, the existing theoretical results mainly focus…
Generalized EXTRA stochastic gradient Langevin dynamics
Mert Gurbuzbalaban, Mohammad Rafiqul Islam, Xiaoyu Wang +1
Langevin algorithms are popular Markov Chain Monte Carlo methods for Bayesian learning, particularly when the aim is to sample from the posterior distribution of a parametric model…
Robustly Stable Accelerated Momentum Methods With A Near-Optimal L2 Gain and Performance
Mert Gurbuzbalaban
We consider the problem of minimizing a strongly convex smooth function where the gradients are subject to additive worst-case deterministic errors that are square-summable. We stu…
Accelerated Gradient Methods for Nonconvex Optimization: Escape Trajectories From Strict Saddle Points and Convergence to Local Minima
Rishabh Dixit, Mert Gurbuzbalaban, Waheed U. Bajwa
This paper considers the problem of understanding the behavior of a general class of accelerated gradient methods on smooth nonconvex functions. Motivated by some recent works that…
Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent
Lingjiong Zhu, Mert Gurbuzbalaban, Anant Raj +1
Algorithmic stability is an important notion that has proven powerful for deriving generalization bounds for practical algorithms. The last decade has witnessed an increasing numbe…
High Probability and Risk-Averse Guarantees for a Stochastic Accelerated Primal-Dual Method
Yassine Laguel, Necdet Serhat Aybat, Mert Gürbüzbalaban
We consider stochastic strongly-convex-strongly-concave (SCSC) saddle point (SP) problems which frequently arise in applications ranging from distributionally robust learning to ga…
Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize
Mert Gürbüzbalaban, Yuanhan Hu, Umut Şimşekli +1
Cyclic and randomized stepsizes are widely used in the deep learning practice and can often outperform standard stepsize choices such as constant stepsize in SGD. Despite their emp…
Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions
Anant Raj, Lingjiong Zhu, Mert Gürbüzbalaban +1
Heavy-tail phenomena in stochastic gradient descent (SGD) have been reported in several empirical studies. Experimental evidence in previous works suggests a strong interplay betwe…
Penalized Overdamped and Underdamped Langevin Monte Carlo Algorithms for Constrained Sampling
Mert Gürbüzbalaban, Yuanhan Hu, Lingjiong Zhu
We consider the constrained sampling problem where the goal is to sample from a target distribution when is constrained to lie on a convex body $\mathca…
Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares
Anant Raj, Melih Barsbey, Mert Gürbüzbalaban +2
Recent studies have shown that heavy tails can emerge in stochastic optimization and that the heaviness of the tails have links to the generalization error. While these studies hav…
SAPD+: An Accelerated Stochastic Method for Nonconvex-Concave Minimax Problems
Xuan Zhang, Necdet Serhat Aybat, Mert Gürbüzbalaban
We propose a new stochastic method SAPD+ for solving nonconvex-concave minimax problems of the form , where are closed convex and $…
Entropic Risk-Averse Generalized Momentum Methods
Bugra Can, Mert Gürbüzbalaban
In the context of first-order algorithms subject to random gradient noise, we study the trade-offs between the convergence rate (which quantifies how fast the initial conditions ar…
A Variance-Reduced Stochastic Accelerated Primal Dual Algorithm
Bugra Can, Mert Gurbuzbalaban, Necdet Serhat Aybat
In this work, we consider strongly convex strongly concave (SCSC) saddle point (SP) problems where is -smooth,…
Robust Accelerated Primal-Dual Methods for Computing Saddle Points
Xuan Zhang, Necdet Serhat Aybat, Mert Gürbüzbalaban
We consider strongly-convex-strongly-concave saddle point problems assuming we have access to unbiased stochastic estimates of the gradients. We propose a stochastic accelerated pr…
L-DQN: An Asynchronous Limited-Memory Distributed Quasi-Newton Method
Bugra Can, Saeed Soori, Maryam Mehri Dehnavi +1
This work proposes a distributed algorithm for solving empirical risk minimization problems, called L-DQN, under the master/worker communication model. L-DQN is a distributed limit…
TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion
Saeed Soori, Bugra Can, Baourun Mu +2
This work proposes a time-efficient Natural Gradient Descent method, called TENGraD, with linear convergence guarantees. Computing the inverse of the neural network's Fisher inform…
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Alexander Camuto, Xiaoyu Wang, Lingjiong Zhu +3
Gaussian noise injections (GNIs) are a family of simple and widely-used regularisation methods for training neural networks, where one injects additive or multiplicative Gaussian n…
Boundary Conditions for Linear Exit Time Gradient Trajectories Around Saddle Points: Analysis and Algorithm
Rishabh Dixit, Mert Gurbuzbalaban, Waheed U. Bajwa
Gradient-related first-order methods have become the workhorse of large-scale numerical optimization problems. Many of these problems involve nonconvex objective functions with mul…
Differentially Private Accelerated Optimization Algorithms
Nurdan Kuru, Ş. İlker Birbil, Mert Gurbuzbalaban +1
We present two classes of differentially private optimization algorithms derived from the well-known accelerated first-order methods. The first algorithm is inspired by Polyak's he…