68 citations · 68 across the 5 of their papers we have counts for
7 papers
Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
Soufiane Hayou, Bobby He, Gintare Karolina Dziugaite
We study an approach to learning pruning masks by optimizing the expected loss of stochastic pruning masks, i.e., masks which zero out each weight independently with some weight-sp…
Regularization in ResNet with Stochastic Depth
Soufiane Hayou, Fadhel Ayed
Regularization plays a major role in modern deep learning. From classic techniques such as L1,L2 penalties to other noise-based methods such as Dropout, regularization often yields…
Stable ResNet
Soufiane Hayou, Eugenio Clerico, Bobby He +3
Deep ResNet architectures have achieved state of the art performance on many tasks. While they solve the problem of gradient vanishing, they might suffer from gradient exploding as…
On the Impact of the Activation Function on Deep Neural Networks Training
Soufiane Hayou, Arnaud Doucet, Judith Rousseau
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lea…
On the Selection of Initialization and Activation Function for Deep Neural Networks
Soufiane Hayou, Arnaud Doucet, Judith Rousseau
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lea…
On the overestimation of the largest eigenvalue of a covariance matrix
Soufiane Hayou
In this paper, we use a new approach to prove that the largest eigenvalue of the sample covariance matrix of a normally distributed vector is bigger than the true largest eigenvalu…