Survey of Dropout Methods for Deep Neural Networks

Abstract

Dropout methods are a family of stochastic techniques used in neural network training or inference that have generated significant research interest and are widely used in practice. They have been successfully applied in neural network regularization, model compression, and in measuring the uncertainty of neural network outputs. While original formulated for dense neural network layers, recent advances have made dropout methods also applicable to convolutional and recurrent neural network layers. This paper summarizes the history of dropout methods, their various applications, and current areas of research interest. Important proposed methods are described in additional detail.

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Improved Regularization of Convolutional Neural Networks with Cutout
• Recurrent Neural Network Regularization
• Recent Advances in Recurrent Neural Networks
• Regularizing and Optimizing LSTM Language Models
• Variational Dropout and the Local Reparameterization Trick
• Deep and Confident Prediction for Time Series at Uber
• Variational Dropout Sparsifies Deep Neural Networks
• Towards Dropout Training for Convolutional Neural Networks
• Deep Networks with Stochastic Depth
• Dropout Training as Adaptive Regularization
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Swapout: Learning an ensemble of deep architectures
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Dropout as data augmentation
• Effective and Efficient Dropout for Deep Convolutional Neural Networks
• Adversarial Dropout Regularization
• Ising-Dropout: A Regularization Method for Training and Compression of Deep Neural Networks
• Adversarial Dropout for Recurrent Neural Networks

Cited by in corpus (11)

• R-Drop: Regularized Dropout for Neural Networks
• Review: Deep Learning in Electron Microscopy
• Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization
• EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks
• HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization
• EventDrop: data augmentation for event-based learning
• Dual-attention Guided Dropblock Module for Weakly Supervised Object Localization
• Noisy-LSTM: Improving Temporal Awareness for Video Semantic Segmentation
• Locally Learned Synaptic Dropout for Complete Bayesian Inference
• Quantal synaptic dilution enhances sparse encoding and dropout regularisation in deep networks
• RotationOut as a Regularization Method for Neural Network