On the cross-validation bias due to unsupervised pre-processing
arXiv:1901.08974 · doi:10.1111/rssb.12537
Abstract
Cross-validation is the de facto standard for predictive model evaluation and selection. In proper use, it provides an unbiased estimate of a model's predictive performance. However, data sets often undergo various forms of data-dependent preprocessing, such as mean-centering, rescaling, dimensionality reduction, and outlier removal. It is often believed that such preprocessing stages, if done in an unsupervised manner (that does not incorporate the class labels or response values) are generally safe to do prior to cross-validation. In this paper, we study three commonly-practiced preprocessing procedures prior to a regression analysis: (i) variance-based feature selection; (ii) grouping of rare categorical features; and (iii) feature rescaling. We demonstrate that unsupervised preprocessing can, in fact, introduce a substantial bias into cross-validation estimates and potentially hurt model selection. This bias may be either positive or negative and its exact magnitude depends on all the parameters of the problem in an intricate manner. Further research is needed to understand the real-world impact of this bias across different application domains, particularly when dealing with small sample sizes and high-dimensional data.
arXiv comment31 pages, 6 figures, 1 table. New sections: (4.2.) Experiments on a real dataset; (6.) Potential impact on model selection; (7.1.) Upper bounds based on stability arguments. Updated Fig. 1. with larger sample sizes
References in corpus (7)
- Scikit-learn: Machine Learning in Python
- A survey of cross-validation procedures for model selection
- MLlib: Machine Learning in Apache Spark
- Train faster, generalize better: Stability of stochastic gradient descent
- On the exact Berk-Jones statistics and their p-value calculation
- Generalization Bounds for Uniformly Stable Algorithms
- Sharper bounds for uniformly stable algorithms