44 citations · 543 across the 47 of their papers we have counts for
5 papers · 1 filter
Heavy-Tailed Diffusion with Denoising Lévy Probabilistic Models
Dario Shariatian, Umut Simsekli, Alain Durmus
Exploring noise distributions beyond Gaussian in diffusion models remains an open challenge. While Gaussian-based models succeed within a unified SDE framework, recent studies sugg…
Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent
Krunoslav Lehman Pavasovic, Alain Durmus, Umut Simsekli
A recent line of empirical studies has demonstrated that SGD might exhibit a heavy-tailed behavior in practical settings, and the heaviness of the tails might correlate with the ov…
Relative Positional Encoding for Transformers with Linear Complexity
Antoine Liutkus, Ondřej Cífka, Shih-Lun Wu +3
Recent advances in Transformer models allow for unprecedented sequence lengths, due to linear space and time complexity. In the meantime, relative positional encoding (RPE) was pro…
Generalized Sliced Wasserstein Distances
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli +2
The Wasserstein distance and its variations, e.g., the sliced-Wasserstein (SW) distance, have recently drawn attention from the machine learning community. The SW distance, specifi…
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
Umut Simsekli, Levent Sagun, Mert Gurbuzbalaban
The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit…