On Convergence and Generalization of Dropout Training

Abstract

We study dropout in two-layer neural networks with rectified linear unit (ReLU) activations. Under mild overparametrization and assuming that the limiting kernel can separate the data distribution with a positive margin, we show that dropout training with logistic loss achieves $\epsilon$-suboptimality in test error in $O(1/\epsilon )$ iterations.

References in corpus (26)

• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Regularizing and Optimizing LSTM Language Models
• Variational Dropout Sparsifies Deep Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Dropout Training as Adaptive Regularization
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Contextual Bandit Algorithms with Supervised Learning Guarantees
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Learning Sparse Networks Using Targeted Dropout
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• SGD Learns the Conjugate Kernel Class of the Network
• A PAC-Bayesian Tutorial with A Dropout Bound
• Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
• On the Implicit Bias of Dropout
• The Implicit and Explicit Regularization Effects of Dropout
• Altitude Training: Strong Bounds for Single-Layer Dropout
• Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
• Dropout: Explicit Forms and Capacity Control
• On Dropout and Nuclear Norm Regularization
• Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study
• Almost Sure Convergence of Dropout Algorithms for Neural Networks

Cited by in corpus (3)

• Structured Dropout Variational Inference for Bayesian Neural Networks
• The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization
• Why Do Better Loss Functions Lead to Less Transferable Features?