On the Implicit Bias of Dropout
arXiv:1806.09777
Abstract
Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural networks, we show that dropout tends to make the norm of incoming/outgoing weight vectors of all the hidden nodes equal. In addition, we provide a complete characterization of the optimization landscape induced by dropout.
arXiv comment17 pages, 3 figures, In Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML), 2018
Cited by in corpus (22)
- The Modern Mathematics of Deep Learning
- Review: Deep Learning in Electron Microscopy
- ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
- A Survey on Neural Network Interpretability
- The Implicit and Explicit Regularization Effects of Dropout
- Loss Landscapes of Regularized Linear Autoencoders
- How Does Mixup Help With Robustness and Generalization?
- Shape Matters: Understanding the Implicit Bias of the Noise Covariance
- On Convergence and Generalization of Dropout Training
- Dropout: Explicit Forms and Capacity Control
- On Dropout and Nuclear Norm Regularization
- Implicit Regularization of Stochastic Gradient Descent in Natural Language Processing: Observations and Implications
- Label Noise SGD Provably Prefers Flat Global Minimizers
- Almost Sure Convergence of Dropout Algorithms for Neural Networks
- Implicit Regularization and Convergence for Weight Normalization
- Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms
- Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations
- Abstraction Mechanisms Predict Generalization in Deep Neural Networks
- Dropout as a Regularizer of Interaction Effects
- An ETF view of Dropout regularization
- Structured Dropout Variational Inference for Bayesian Neural Networks
- An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network