Altitude Training: Strong Bounds for Single-Layer Dropout

Abstract

Dropout training, originally designed for deep neural networks, has been successful on high-dimensional single-layer natural language tasks. This paper proposes a theoretical explanation for this phenomenon: we show that, under a generative Poisson topic model with long documents, dropout training improves the exponent in the generalization bound for empirical risk minimization. Dropout achieves this gain much like a marathon runner who practices at altitude: once a classifier learns to perform reasonably well on training examples that have been artificially corrupted by dropout, it will do very well on the uncorrupted test set. We also show that, under similar conditions, dropout preserves the Bayes decision boundary and should therefore induce minimal bias in high dimensions.

Advances in Neural Information Processing Systems (NIPS), 2014

References in corpus (3)

• Improving neural networks by preventing co-adaptation of feature detectors
• Dropout Training as Adaptive Regularization
• A PAC-Bayesian Tutorial with A Dropout Bound

Cited by in corpus (10)

• Salient Objects in Clutter
• The Implicit and Explicit Regularization Effects of Dropout
• On the Inductive Bias of Dropout
• A Generic Acceleration Framework for Stochastic Composite Optimization
• On Convergence and Generalization of Dropout Training
• Dropout: Explicit Forms and Capacity Control
• On Dropout and Nuclear Norm Regularization
• An ETF view of Dropout regularization
• Surprising properties of dropout in deep networks
• Why Do Better Loss Functions Lead to Less Transferable Features?