743 citations · 2.1k across the 27 of their papers we have counts for
4 papers · 1 filter
On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
Umut Şimşekli, Mert Gürbüzbalaban, Thanh Huy Nguyen +2
The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the \emph{classical} central…
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
Stéphane d'Ascoli, Levent Sagun, Joan Bruna +1
Despite the phenomenal success of deep neural networks in a broad range of learning tasks, there is a lack of theory to understand the way they work. In particular, Convolutional N…
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
Umut Simsekli, Levent Sagun, Mert Gurbuzbalaban
The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit…
Scaling description of generalization with number of parameters in deep learning
Mario Geiger, Arthur Jacot, Stefano Spigler +6
Supervised deep learning involves the training of neural networks with a large number of parameters. For large enough , in the so-called over-parametrized regime, one can es…