Quantal synaptic dilution enhances sparse encoding and dropout regularisation in deep networks

Abstract

Dropout is a technique that silences the activity of units stochastically while training deep networks to reduce overfitting. Here we introduce Quantal Synaptic Dilution (QSD), a biologically plausible model of dropout regularisation based on the quantal properties of neuronal synapses, that incorporates heterogeneities in response magnitudes and release probabilities for vesicular quanta. QSD outperforms standard dropout in ReLU multilayer perceptrons, with enhanced sparse encoding at test time when dropout masks are replaced with identity functions, without shifts in trainable weight or bias distributions. For convolutional networks, the method also improves generalisation in computer vision tasks with and without inclusion of additional forms of regularisation. QSD also outperforms standard dropout in recurrent networks for language modelling and sentiment analysis. An advantage of QSD over many variations of dropout is that it can be implemented generally in all conventional deep networks where standard dropout is applicable.

arXiv comment23 pages, 8 figures, including Appendix

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Recurrent Neural Network Regularization
• Wide Residual Networks
• Variational Dropout and the Local Reparameterization Trick
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Effective and Efficient Dropout for Deep Convolutional Neural Networks
• Survey of Dropout Methods for Deep Neural Networks