Predicting into unknown space? Estimating the area of applicability of spatial prediction models

Abstract

Predictive modelling using machine learning has become very popular for spatial mapping of the environment. Models are often applied to make predictions far beyond sampling locations where new geographic locations might considerably differ from the training data in their environmental properties. However, areas in the predictor space without support of training data are problematic. Since the model has no knowledge about these environments, predictions have to be considered uncertain. Estimating the area to which a prediction model can be reliably applied is required. Here, we suggest a methodology that delineates the "area of applicability" (AOA) that we define as the area, for which the cross-validation error of the model applies. We first propose a "dissimilarity index" (DI) that is based on the minimum distance to the training data in the predictor space, with predictors being weighted by their respective importance in the model. The AOA is then derived by applying a threshold based on the DI of the training data where the DI is calculated with respect to the cross-validation strategy used for model training. We test for the ideal threshold by using simulated data and compare the prediction error within the AOA with the cross-validation error of the model. We illustrate the approach using a simulated case study. Our simulation study suggests a threshold on DI to define the AOA at the .95 quantile of the DI in the training data. Using this threshold, the prediction error within the AOA is comparable to the cross-validation RMSE of the model, while the cross-validation error does not apply outside the AOA. This applies to models being trained with randomly distributed training data, as well as when training data are clustered in space and where spatial cross-validation is applied. We suggest to report the AOA alongside predictions, complementary to validation measures.

16 pages, 10 figures, to be submitted to Methods in Ecology and Evolution

References in corpus (1)

• Estimating the Prediction Performance of Spatial Models via Spatial k-Fold Cross Validation

Cited by in corpus (9)

• Earth System Data Cubes: Avenues for advancing Earth system research
• Comparison of machine learning algorithms for merging gridded satellite and earth-observed precipitation data
• Better, Not Just More: Data-Centric Machine Learning for Earth Observation
• Fine-resolution landscape-scale biomass mapping using a spatiotemporal patchwork of LiDAR coverages
• A review of regularised estimation methods and cross-validation in spatiotemporal statistics
• Mapping historical forest biomass for stock-change assessments at parcel to landscape scales
• From pixels to parcels: flexible, practical small-area uncertainty estimation for spatial averages obtained from aboveground biomass maps
• On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction
• SatHealth: A Multimodal Public Health Dataset with Satellite-based Environmental Factors