44 citations · 543 across the 47 of their papers we have counts for
9 papers · 1 filter
Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent
Krunoslav Lehman Pavasovic, Alain Durmus, Umut Simsekli
A recent line of empirical studies has demonstrated that SGD might exhibit a heavy-tailed behavior in practical settings, and the heaviness of the tails might correlate with the ov…
Generalization Guarantees via Algorithm-dependent Rademacher Complexity
Sarah Sachs, Tim van Erven, Liam Hodgkinson +2
Algorithm- and data-dependent generalization bounds are required to explain the generalization behavior of modern machine learning algorithms. In this context, there exists informa…
Nonparametric Linear Feature Learning in Regression Through Regularisation
Bertille Follain, Francis Bach
Representation learning plays a crucial role in automated feature selection, particularly in the context of high-dimensional data, where non-parametric methods often struggle. In t…
Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD
Yijun Wan, Melih Barsbey, Abdellatif Zaidi +1
Neural network compression has been an increasingly important subject, not only due to its practical relevance, but also due to its theoretical implications, as there is an explici…
Learning via Wasserstein-Based High Probability Generalisation Bounds
Paul Viallard, Maxime Haddouche, Umut Şimşekli +1
Minimising upper bounds on the population risk or the generalisation gap has been widely used in structural risk minimisation (SRM) -- this is in particular at the core of PAC-Baye…
Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent
Lingjiong Zhu, Mert Gurbuzbalaban, Anant Raj +1
Algorithmic stability is an important notion that has proven powerful for deriving generalization bounds for practical algorithms. The last decade has witnessed an increasing numbe…
Efficient Sampling of Stochastic Differential Equations with Positive Semi-Definite Models
Anant Raj, Umut Şimşekli, Alessandro Rudi
This paper deals with the problem of efficient sampling from a stochastic differential equation, given the drift function and the diffusion matrix. The proposed approach leverages…
Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize
Mert Gürbüzbalaban, Yuanhan Hu, Umut Şimşekli +1
Cyclic and randomized stepsizes are widely used in the deep learning practice and can often outperform standard stepsize choices such as constant stepsize in SGD. Despite their emp…
Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions
Anant Raj, Lingjiong Zhu, Mert Gürbüzbalaban +1
Heavy-tail phenomena in stochastic gradient descent (SGD) have been reported in several empirical studies. Experimental evidence in previous works suggests a strong interplay betwe…