Dropout Drops Double Descent
arXiv:2305.16179 · doi:10.1007/s42081-024-00242-5
Abstract
This study demonstrates that double descent can be mitigated by adding a dropout layer adjacent to the fully connected linear layer. The unexpected double-descent phenomenon garnered substantial attention in recent years, resulting in fluctuating prediction error rates as either sample size or model size increases. Our paper posits that the optimal test error, in terms of the dropout rate, shows a monotonic decrease in linear regression with increasing sample size. Although we do not provide a precise mathematical proof of this statement, we empirically validate through experiments that the test error decreases for each dropout rate. The statement we prove is that the expected test error for each dropout rate within a certain range decreases when the dropout rate is fixed. Our experimental results substantiate our claim, showing that dropout with an optimal dropout rate can yield a monotonic test error curve in nonlinear neural networks. These experiments were conducted using the Fashion-MNIST and CIFAR-10 datasets. These findings imply the potential benefit of incorporating dropout into risk curve scaling to address the peak phenomenon. To our knowledge, this study represents the first investigation into the relationship between dropout and double descent.
References in corpus (11)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- DeepPose: Human Pose Estimation via Deep Neural Networks
- Reconciling modern machine learning practice and the bias-variance trade-off
- Variational Dropout and the Local Reparameterization Trick
- Two models of double descent for weak features
- Surprises in High-Dimensional Ridgeless Least Squares Interpolation
- A new class of generalized Bayes minimax ridge regression estimators
- Multiple Descent: Design Your Own Generalization Curve
- The Implicit and Explicit Regularization Effects of Dropout
- The Implicit Regularization of Ordinary Least Squares Ensembles
- Early Stopping in Deep Networks: Double Descent and How to Eliminate it