17 citations · 46 across the 22 of their papers we have counts for
3 papers · 1 filter
If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks
Arnu Pretorius, Elan van Biljon, Benjamin van Niekerk +6
Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effe…
Critical initialisation for deep signal propagation in noisy rectifier neural networks
Arnu Pretorius, Elan Van Biljon, Steve Kroon +1
Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influe…
Learning Dynamics of Linear Denoising Autoencoders
Arnu Pretorius, Steve Kroon, Herman Kamper
Denoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influenc…