A survey of cross-validation procedures for model selection

Abstract

Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recent advances of model selection theory, with a particular emphasis on distinguishing empirical statements from rigorous theoretical results. As a conclusion, guidelines are provided for choosing the best cross-validation procedure according to the particular features of the problem in hand.

References in corpus (6)

• Least angle and ${\ell }_1$ penalized regression: A review
• Consistency of cross validation for comparing regression procedures
• Model selection by resampling penalization
• Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii
• Catching Up Faster by Switching Sooner: A Prequential Solution to the AIC-BIC Dilemma
• Cross-validation in nonparametric regression with outliers

Cited by in corpus (120)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Selective review of offline change point detection methods
• Assessing and tuning brain decoders: cross-validation, caveats, and guidelines
• Studies in Astronomical Time Series Analysis. VI. Bayesian Block Representations
• Evaluating time series forecasting models: An empirical study on performance estimation methods
• Boosting Docking-based Virtual Screening with Deep Learning
• Scenic: A Language for Scenario Specification and Scene Generation
• Vision Transformers For Weeds and Crops Classification Of High Resolution UAV Images
• Compressive Sampling of Polynomial Chaos Expansions: Convergence Analysis and Sampling Strategies
• Ranking in evolving complex networks
• A genomic map of the effects of linked selection in Drosophila
• CRIX an index for cryptocurrencies
• Stacking Models for Nearly Optimal Link Prediction in Complex Networks
• Practical considerations for specifying a super learner
• How to avoid machine learning pitfalls: a guide for academic researchers
• Patterned Probes for High Precision 4D-STEM Bragg Measurements
• Marketing Analytics: Methods, Practice, Implementation, and Links to Other Fields
• Characterization of the K2-18 multi-planetary system with HARPS: A habitable zone super-Earth and discovery of a second, warm super-Earth on a non-coplanar orbit
• A Tutorial on Canonical Correlation Methods
• Review of data analysis in vision inspection of power lines with an in-depth discussion of deep learning technology
• Understanding the effect of hyperparameter optimization on machine learning models for structure design problems
• Photometric redshifts for Quasars in multi band Surveys
• Deep learning in biomedical optics
• A Method of Rapid Quantification of Patient-Specific Organ Dose for CT Using Coupled Deep-Learning based Multi-Organ Segmentation and GPU-accelerated Monte Carlo Dose Computing
• Validating Predictions of Unobserved Quantities
• Lattice Thermal Conductivity Prediction using Symbolic Regression and Machine Learning
• Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI
• Radial velocity follow-up of GJ1132 with HARPS. A precise mass for planet 'b' and the discovery of a second planet
• On Polynomial Chaos Expansion via Gradient-enhanced ${\ell }_1$-minimization
• Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning
• Kernel PCA for type Ia supernovae photometric classification
• Automatic cross-validation in structured models: Is it time to leave out leave-one-out?
• Consistencies and inconsistencies between model selection and link prediction in networks
• Quantifying the Bayesian Evidence for a Planet in Radial Velocity Data
• Maximizing the information learned from finite data selects a simple model
• Optimal cross-validation in density estimation with the $L^2$-loss
• Comparison between different methods of model selection in cosmology
• SUNNY: a Lazy Portfolio Approach for Constraint Solving
• Predicting Chemical Hazard across Taxa through Machine Learning
• Probing the stellar initial mass function with high-$z$ supernovae
• MOST Observations of sigma Ori E: Challenging the Centrifugal Breakout Narrative
• Segmentation of the mean of heteroscedastic data via cross-validation
• Detection, localisation and tracking of pallets using machine learning techniques and 2D range data
• Linguistic data mining with complex networks: a stylometric-oriented approach
• Photometric redshifts with Quasi Newton Algorithm (MLPQNA). Results in the PHAT1 contest
• On the cross-validation bias due to unsupervised pre-processing
• Spectral likelihood expansions for Bayesian inference
• A Hierarchical Bayesian Model for Size Recommendation in Fashion
• Hyperspectral Subspace Identification Using SURE
• Estimating the Trace of the Matrix Inverse by Interpolating from the Diagonal of an Approximate Inverse
• Linear Shrinkage Estimation of Covariance Matrices Using Low-Complexity Cross-Validation
• Cross-validation estimate of the number of clusters in a network
• Sagittarius stream 3-d kinematics from SDSS Stripe 82
• Local primordial non-Gaussianity from the large-scale clustering of photometric DESI luminous red galaxies
• A Novel Regularization Based on the Error Function for Sparse Recovery
• Exhaustive Model Selection in $b\to s\ell \ell$ Decays: Pitting Cross-Validation against AIC${}_c$
• Walking dynamics are symmetric (enough)
• Efficient implementations of echo state network cross-validation
• How to evaluate sentiment classifiers for Twitter time-ordered data?
• Search for CP violation using triple product asymmetries in ${\Lambda }_b^0\to p{K}^{-}{\pi }^{+}{\pi }^{-}$, ${\Lambda }_b^0\to p{K}^{-}{K}^{+}{K}^{-}$ and ${\Xi }_b^0\to p{K}^{-}{K}^{-}{\pi }^{+}$ decays
• Forecasting: theory and practice
• Simultaneous exercise recognition and evaluation in prescribed routines: Approach to virtual coaches
• Structured and Unstructured Outlier Identification for Robust PCA: A Non iterative, Parameter free Algorithm
• Probing non-Markovian quantum dynamics with data-driven analysis: Beyond "black-box" machine learning models
• Heri-Graphs: A Workflow of Creating Datasets for Multi-modal Machine Learning on Graphs of Heritage Values and Attributes with Social Media
• Cross-validation based adaptive sampling for Gaussian process models
• Dimensionality reduction to maximize prediction generalization capability
• A metric space for type Ia supernova spectra
• Robust leave-one-out cross-validation for high-dimensional Bayesian models
• On Model Selection in Cosmology
• Predictive intraday correlations in stable and volatile market environments: Evidence from deep learning
• Experimental reservoir computing with diffractively coupled VCSELs
• Gaussian Markov Random Fields versus Linear Mixed Models for satellite-based PM2.5 assessment: Evidence from the Northeastern USA
• On Designing a Machine Learning Based Wireless Link Quality Classifier
• Regularized Covariance Estimation for Polarization Radar Detection in Compound Gaussian Sea Clutter
• A review of regularised estimation methods and cross-validation in spatiotemporal statistics
• Extensions of stability selection using subsamples of observations and covariates
• Robust Information Criterion for Model Selection in Sparse High-Dimensional Linear Regression Models
• BioWorkbench: A High-Performance Framework for Managing and Analyzing Bioinformatics Experiments
• Variational Bayesian Weighted Complex Network Reconstruction
• Adaptive Detection of Structured Signals in Low-Rank Interference
• Predictions of planet detections with near infrared radial velocities in the up-coming SPIRou Legacy Survey-Planet Search
• Real-Space Inversion and Super-Resolution of Ultrafast Scattering
• Spectral model selection in the electronic measurement of the Boltzmann constant by Johnson noise thermometry
• Stochastic Local Interaction Model: Geostatistics without Kriging
• A Bayesian constitutive model selection framework for biaxial mechanical testing of planar soft tissues: application to porcine aortic valves
• A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection
• Slope heuristics and V-Fold model selection in heteroscedastic regression using strongly localized bases
• Semiparametric zero-inflated modeling in multi-ethnic study of atherosclerosis (MESA)
• Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
• Efficient estimation and correction of selection-induced bias with order statistics
• Root-finding Approaches for Computing Conformal Prediction Set
• Learning to Fairly Classify the Quality of Wireless Links
• Building Robust Machine Learning Models for Small Chemical Science Data: The Case of Shear Viscosity
• A decade of the fast-varying ionospheric and magnetospheric magnetic fields from ground and multi-satellite observations
• Closed-loop Identification of a MSW Grate Incinerator using Bayesian Optimization for Selecting Model Inputs and Structure
• Too Good To Be True: performance overestimation in (re)current practices for Human Activity Recognition
• On Statistical Efficiency in Learning
• Machine learning-based mass density model for hard magnetic 14:2:1 phases using chemical composition-based features
• Reliable Characterization for Improving and Validating Accurate Quantum Operations
• Support Vector Machines and generalisation in HEP
• Transition in the Waiting-Time Distribution of Price-Change Events in a Global Socioeconomic System
• An information criterion for auxiliary variable selection in incomplete data analysis
• Model Monitoring and Dynamic Model Selection in Travel Time-series Forecasting
• The optimal Padé polynomial for reconstruction of luminosity distance based on 10-fold cross-validation
• Informative Bayesian model selection for RR Lyrae star classifiers
• Model Selection in Utility-Maximizing Binary Prediction
• Forbidden Knowledge and Specialized Training: A Versatile Solution for the Two Main Sources of Overfitting in Linear Regression
• Taskmaster Deconstructed: A Quantitative Look at Tension, Volatility, and Viewer Ratings
• The out-of-source error in multi-source cross validation-type procedures
• Audio-based performance evaluation of squash players
• A statistical model for in vivo neuronal dynamics
• Penalized Likelihood Methods for Modeling Count Data
• Approximate Cross-validated Mean Estimates for Bayesian Hierarchical Regression Models
• Model-Twin Randomization (MoTR) for Estimating the Recurring Individual Treatment Effect
• Astrometric Binary Classification Via Artificial Neural Networks
• Impacts of Data Splitting Strategies on Parameterized Link Prediction Algorithms
• Cross-validated covariance estimators for high-dimensional minimum-variance portfolios
• Measurement of Hybrid Rocket Solid Fuel Regression Rate for a Slab Burner using Deep Learning
• adF: A Novel System for Measuring Web Fingerprinting through Ads