The Implicit and Explicit Regularization Effects of Dropout

Abstract

Dropout is a widely-used regularization technique, often required to obtain state-of-the-art for a number of architectures. This work demonstrates that dropout introduces two distinct but entangled regularization effects: an explicit effect (also studied in prior work) which occurs since dropout modifies the expected training objective, and, perhaps surprisingly, an additional implicit effect from the stochasticity in the dropout training update. This implicit regularization effect is analogous to the effect of stochasticity in small mini-batch stochastic gradient descent. We disentangle these two effects through controlled experiments. We then derive analytic simplifications which characterize each effect in terms of the derivatives of the model and the loss, for deep neural networks. We demonstrate these simplified, analytic regularizers accurately capture the important aspects of dropout, showing they faithfully replace dropout in practice.

arXiv commentPublished in ICML 2020. Code available at https://github.com/cwein3/dropout-analytical

References in corpus (38)

• Improving neural networks by preventing co-adaptation of feature detectors
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Recurrent Neural Network Regularization
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• Improving Generalization Performance by Switching from Adam to SGD
• Quasi-Recurrent Neural Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Dropout Training as Adaptive Regularization
• Robust Large Margin Deep Neural Networks
• Three Factors Influencing Minima in SGD
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• An Analysis of Neural Language Modeling at Multiple Scales
• Recurrent Dropout without Memory Loss
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Kernel and Rich Regimes in Overparametrized Models
• Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
• A Walk with SGD
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• On the State of the Art of Evaluation in Neural Language Models
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
• Fluctuation-dissipation relations for stochastic gradient descent
• Revisiting Activation Regularization for Language RNNs
• Robust Learning with Jacobian Regularization
• A Bayesian encourages dropout
• On the Implicit Bias of Dropout
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• Regularizing RNNs by Stabilizing Activations
• Altitude Training: Strong Bounds for Single-Layer Dropout
• Dropout as a Low-Rank Regularizer for Matrix Factorization
• Dropout with Expectation-linear Regularization
• Dropout: Explicit Forms and Capacity Control
• On Dropout and Nuclear Norm Regularization

Cited by in corpus (10)

• Explicit Regularisation in Gaussian Noise Injections
• How Does Mixup Help With Robustness and Generalization?
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• On Convergence and Generalization of Dropout Training
• On the Inherent Regularization Effects of Noise Injection During Training
• On Mixup Regularization
• Constraint-Based Regularization of Neural Networks
• UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
• Achieving Small Test Error in Mildly Overparameterized Neural Networks
• Dropout Drops Double Descent