Understanding Dropout: Training Multi-Layer Perceptrons with Auxiliary Independent Stochastic Neurons

Abstract

In this paper, a simple, general method of adding auxiliary stochastic neurons to a multi-layer perceptron is proposed. It is shown that the proposed method is a generalization of recently successful methods of dropout (Hinton et al., 2012), explicit noise injection (Vincent et al., 2010; Bishop, 1995) and semantic hashing (Salakhutdinov & Hinton, 2009). Under the proposed framework, an extension of dropout which allows using separate dropping probabilities for different hidden neurons, or layers, is found to be available. The use of different dropping probabilities for hidden layers separately is empirically investigated.

ICONIP 2013: Special Session in Deep Learning (v4)

References in corpus (5)

• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Deep Generative Stochastic Networks Trainable by Backprop
• Estimating or Propagating Gradients Through Stochastic Neurons

Cited by in corpus (1)

• On the Origin of Deep Learning