Estimating the Prediction Performance of Spatial Models via Spatial k-Fold Cross Validation

Abstract

In machine learning one often assumes the data are independent when evaluating model performance. However, this rarely holds in practise. Geographic information data sets are an example where the data points have stronger dependencies among each other the closer they are geographically. This phenomenon known as spatial autocorrelation (SAC) causes the standard cross validation (CV) methods to produce optimistically biased prediction performance estimates for spatial models, which can result in increased costs and accidents in practical applications. To overcome this problem we propose a modified version of the CV method called spatial k-fold cross validation (SKCV), which provides a useful estimate for model prediction performance without optimistic bias due to SAC. We test SKCV with three real world cases involving open natural data showing that the estimates produced by the ordinary CV are up to 40% more optimistic than those of SKCV. Both regression and classification cases are considered in our experiments. In addition, we will show how the SKCV method can be applied as a criterion for selecting data sampling density for new research area.

18 pages, 12 figures, 1 table

Cited by in corpus (12)

• Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data
• Predicting into unknown space? Estimating the area of applicability of spatial prediction models
• Importance of spatial predictor variable selection in machine learning applications -- Moving from data reproduction to spatial prediction
• Spatial machine-learning model diagnostics: a model-agnostic distance-based approach
• Predicting Drought and Subsidence Risks in France
• Augmenting correlation structures in spatial data using deep generative models
• A review of regularised estimation methods and cross-validation in spatiotemporal statistics
• Foundation for unbiased cross-validation of spatio-temporal models for species distribution modeling
• On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction
• Graphical outputs and Spatial Cross-validation for the R-INLA package using INLAutils
• Prediction & Model Evaluation for Space-Time Data
• mlr3spatiotempcv: Spatiotemporal resampling methods for machine learning in R