To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout

Abstract

Training deep belief networks (DBNs) requires optimizing a non-convex function with an extremely large number of parameters. Naturally, existing gradient descent (GD) based methods are prone to arbitrarily poor local minima. In this paper, we rigorously show that such local minima can be avoided (upto an approximation error) by using the dropout technique, a widely used heuristic in this domain. In particular, we show that by randomly dropping a few nodes of a one-hidden layer neural network, the training objective function, up to a certain approximation error, decreases by a multiplicative factor. On the flip side, we show that for training convex empirical risk minimizers (ERM), dropout in fact acts as a "stabilizer" or regularizer. That is, a simple dropout based GD method for convex ERMs is stable in the face of arbitrary changes to any one of the training points. Using the above assertion, we show that dropout provides fast rates for generalization error in learning (convex) generalized linear models (GLM). Moreover, using the above mentioned stability properties of dropout, we design dropout based differentially private algorithms for solving ERMs. The learned GLM thus, preserves privacy of each of the individual training points while providing accurate predictions for new test points. Finally, we empirically validate our stability assertions for dropout in the context of convex ERMs and show that surprisingly, dropout significantly outperforms (in terms of prediction accuracy) the L2 regularization based methods for several benchmark datasets.

arXiv commentCurrently under review for ICML 2015

References in corpus (6)

• Improving neural networks by preventing co-adaptation of feature detectors
• Intriguing properties of neural networks
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• A PAC-Bayesian Tutorial with A Dropout Bound
• Online Learning, Stability, and Stochastic Gradient Descent
• On the Inductive Bias of Dropout

Cited by in corpus (8)

• Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
• Privacy-Preserving Deep Learning via Weight Transmission
• Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
• Ensemble Robustness and Generalization of Stochastic Deep Learning Algorithms
• Dropout with Expectation-linear Regularization
• Aggregating Votes with Local Differential Privacy: Usefulness, Soundness vs. Indistinguishability
• Differentially Private Dropout
• Stability Enhanced Privacy and Applications in Private Stochastic Gradient Descent