Effective and Efficient Dropout for Deep Convolutional Neural Networks

Abstract

Convolutional Neural networks (CNNs) based applications have become ubiquitous, where proper regularization is greatly needed. To prevent large neural network models from overfitting, dropout has been widely used as an efficient regularization technique in practice. However, many recent works show that the standard dropout is ineffective or even detrimental to the training of CNNs. In this paper, we revisit this issue and examine various dropout variants in an attempt to improve existing dropout-based regularization techniques for CNNs. We attribute the failure of standard dropout to the conflict between the stochasticity of dropout and its following Batch Normalization (BN), and propose to reduce the conflict by placing dropout operations right before the convolutional operation instead of BN, or totally address this issue by replacing BN with Group Normalization (GN). We further introduce a structurally more suited dropout variant Drop-Conv2d, which provides more efficient and effective regularization for deep CNNs. These dropout variants can be readily integrated into the building blocks of CNNs and implemented in existing deep learning platforms. Extensive experiments on benchmark datasets including CIFAR, SVHN and ImageNet are conducted to compare the existing building blocks and the proposed ones with dropout training. Results show that our building blocks improve over state-of-the-art CNNs significantly, which is mainly due to the better regularization and implicit model ensemble effect.

12 pages, 10 figures

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Recurrent Neural Network Regularization
• DARTS: Differentiable Architecture Search
• DropBlock: A regularization method for convolutional networks
• Variational Dropout and the Local Reparameterization Trick
• Dropout Training as Adaptive Regularization
• Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints
• Adjusting for Dropout Variance in Batch Normalization and Weight Initialization

Cited by in corpus (12)

• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• Review: Deep Learning in Electron Microscopy
• Detecting Aedes Aegypti Mosquitoes through Audio Classification with Convolutional Neural Networks
• Parameters Estimation for the Cosmic Microwave Background with Bayesian Neural Networks
• Survey of Dropout Methods for Deep Neural Networks
• Decoding ECoG signal into 3D hand translation using deep learning
• Galaxy classification: a deep learning approach for classifying Sloan Digital Sky Survey images
• Offline Handwritten Chinese Text Recognition with Convolutional Neural Networks
• Estimation of redshift and associated uncertainty of Fermi/LAT extra-galactic sources with Deep Learning
• Quantal synaptic dilution enhances sparse encoding and dropout regularisation in deep networks
• Raw Differentiable Architecture Search for Speech Deepfake and Spoofing Detection
• Dropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation