On Dropout and Nuclear Norm Regularization

Abstract

We give a formal and complete characterization of the explicit regularizer induced by dropout in deep linear networks with squared loss. We show that (a) the explicit regularizer is composed of an ${\ell }_2$-path regularizer and other terms that are also re-scaling invariant, (b) the convex envelope of the induced regularizer is the squared nuclear norm of the network map, and (c) for a sufficiently large dropout rate, we characterize the global optima of the dropout objective. We validate our theoretical findings with empirical results.

References in corpus (3)

• Improving neural networks by preventing co-adaptation of feature detectors
• Understanding deep learning requires rethinking generalization
• Altitude Training: Strong Bounds for Single-Layer Dropout