Dropout: Explicit Forms and Capacity Control

Abstract

We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a data-dependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, MNIST, and Fashion-MNIST.

References in corpus (16)

• Improving neural networks by preventing co-adaptation of feature detectors
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Brain Tumor Segmentation with Deep Neural Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Dropout Training as Adaptive Regularization
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Norm-Based Capacity Control in Neural Networks
• A PAC-Bayesian Tutorial with A Dropout Bound
• Optimistic Rates for Learning with a Smooth Loss
• Learning with the Weighted Trace-norm under Arbitrary Sampling Distributions
• On the Implicit Bias of Dropout
• Altitude Training: Strong Bounds for Single-Layer Dropout
• Dropout as a Low-Rank Regularizer for Matrix Factorization
• On Dropout and Nuclear Norm Regularization
• An ETF view of Dropout regularization

Cited by in corpus (11)

• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Explicit Regularisation in Gaussian Noise Injections
• The Implicit and Explicit Regularization Effects of Dropout
• How Does Mixup Help With Robustness and Generalization?
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• On Convergence and Generalization of Dropout Training
• Meta-Learning with Fewer Tasks through Task Interpolation
• Implicit Regularization and Convergence for Weight Normalization
• What training reveals about neural network complexity
• Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines
• The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization