On Dropout and Nuclear Norm Regularization

Abstract

We give a formal and complete characterization of the explicit regularizer induced by dropout in deep linear networks with squared loss. We show that (a) the explicit regularizer is composed of an ${\ell }_2$-path regularizer and other terms that are also re-scaling invariant, (b) the convex envelope of the induced regularizer is the squared nuclear norm of the network map, and (c) for a sufficiently large dropout rate, we characterize the global optima of the dropout objective. We validate our theoretical findings with empirical results.

References in corpus (14)

• Improving neural networks by preventing co-adaptation of feature detectors
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Understanding deep learning requires rethinking generalization
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Comprehensive Analysis of Deep Regression
• Dropout Training as Adaptive Regularization
• Characterizing Implicit Bias in Terms of Optimization Geometry
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Gradient descent aligns the layers of deep linear networks
• Implicit Bias of Gradient Descent on Linear Convolutional Networks
• On the Implicit Bias of Dropout
• Convergence of Gradient Descent on Separable Data
• Altitude Training: Strong Bounds for Single-Layer Dropout
• An ETF view of Dropout regularization

Cited by in corpus (9)

• Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
• The Implicit and Explicit Regularization Effects of Dropout
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• On Convergence and Generalization of Dropout Training
• Dropout: Explicit Forms and Capacity Control
• Almost Sure Convergence of Dropout Algorithms for Neural Networks
• Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Regularization in ResNet with Stochastic Depth