68 citations · 160 across the 28 of their papers we have counts for
3 papers · 1 filter
Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
Soufiane Hayou, Bobby He, Gintare Karolina Dziugaite
We study an approach to learning pruning masks by optimizing the expected loss of stochastic pruning masks, i.e., masks which zero out each weight independently with some weight-sp…
Feature Learning and Signal Propagation in Deep Neural Networks
Yizhang Lou, Chris Mingard, Yoonsoo Nam +1
Recent work by Baratin et al. (2021) sheds light on an intriguing pattern that occurs during the training of deep neural networks: some layers align much more with data compared to…
Regularization in ResNet with Stochastic Depth
Soufiane Hayou, Fadhel Ayed
Regularization plays a major role in modern deep learning. From classic techniques such as L1,L2 penalties to other noise-based methods such as Dropout, regularization often yields…