Towards Dropout Training for Convolutional Neural Networks

Abstract

Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in convolutional and pooling layers is still not clear. This paper demonstrates that max-pooling dropout is equivalent to randomly picking activation based on a multinomial distribution at training time. In light of this insight, we advocate employing our proposed probabilistic weighted pooling, instead of commonly used max-pooling, to act as model averaging at test time. Empirical evidence validates the superiority of probabilistic weighted pooling. We also empirically show that the effect of convolutional dropout is not trivial, despite the dramatically reduced possibility of over-fitting due to the convolutional architecture. Elaborately designing dropout training simultaneously in max-pooling and fully-connected layers, we achieve state-of-the-art performance on MNIST, and very competitive results on CIFAR-10 and CIFAR-100, relative to other approaches without data augmentation. Finally, we compare max-pooling dropout and stochastic pooling, both of which introduce stochasticity based on multinomial distributions at pooling stage.

This paper has been published in Neural Networks, http://www.sciencedirect.com/science/article/pii/S0893608015001446

References in corpus (4)

• Improving neural networks by preventing co-adaptation of feature detectors
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Improving Deep Neural Networks with Probabilistic Maxout Units
• An empirical analysis of dropout in piecewise linear networks

Cited by in corpus (17)

• Improved Regularization of Convolutional Neural Networks with Cutout
• R-Drop: Regularized Dropout for Neural Networks
• Deep Learning Autoencoder Approach for Handwritten Arabic Digits Recognition
• Survey of Dropout Methods for Deep Neural Networks
• Light-Weight 1-D Convolutional Neural Network Architecture for Mental Task Identification and Classification Based on Single-Channel EEG
• BiCurNet: Pre-Movement EEG based Neural Decoder for Biceps Curl Trajectory Estimation
• On the Reduction of Variance and Overestimation of Deep Q-Learning
• Provably-Stable Neural Network-Based Control of Nonlinear Systems
• On the Effectiveness of Regularization Against Membership Inference Attacks
• Towards Better Forecasting by Fusing Near and Distant Future Visions
• Improving Neural Network Generalization by Combining Parallel Circuits with Dropout
• RePr: Improved Training of Convolutional Filters
• A Deep Learning Framework for Classification of in vitro Multi-Electrode Array Recordings
• Scale Calibrated Training: Improving Generalization of Deep Networks via Scale-Specific Normalization
• CNN-Based Deep Architecture for Reinforced Concrete Delamination Segmentation Through Thermography
• CHD:Consecutive Horizontal Dropout for Human Gait Feature Extraction
• Classifying States of Cooking Objects Using Convolutional Neural Network