Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data

Abstract

Machine-learning algorithms have gained popularity in recent years in the field of ecological modeling due to their promising results in predictive performance of classification problems. While the application of such algorithms has been highly simplified in the last years due to their well-documented integration in commonly used statistical programming languages such as R, there are several practical challenges in the field of ecological modeling related to unbiased performance estimation, optimization of algorithms using hyperparameter tuning and spatial autocorrelation. We address these issues in the comparison of several widely used machine-learning algorithms such as Boosted Regression Trees (BRT), k-Nearest Neighbor (WKNN), Random Forest (RF) and Support Vector Machine (SVM) to traditional parametric algorithms such as logistic regression (GLM) and semi-parametric ones like generalized additive models (GAM). Different nested cross-validation methods including hyperparameter tuning methods are used to evaluate model performances with the aim to receive bias-reduced performance estimates. As a case study the spatial distribution of forest disease Diplodia sapinea in the Basque Country in Spain is investigated using common environmental variables such as temperature, precipitation, soil or lithology as predictors. Results show that GAM and RF (mean AUROC estimates 0.708 and 0.699) outperform all other methods in predictive accuracy. The effect of hyperparameter tuning saturates at around 50 iterations for this data set. The AUROC differences between the bias-reduced (spatial cross-validation) and overoptimistic (non-spatial cross-validation) performance estimates of the GAM and RF are 0.167 (24%) and 0.213 (30%), respectively. It is recommended to also use spatial partitioning for cross-validation hyperparameter tuning of spatial data.

References in corpus (7)

• ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• OpenML: networked science in machine learning
• Tunability: Importance of Hyperparameters of Machine Learning Algorithms
• Estimating the Prediction Performance of Spatial Models via Spatial k-Fold Cross Validation
• mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions
• Empirical AUC for evaluating probabilistic forecasts

Cited by in corpus (12)

• Predicting into unknown space? Estimating the area of applicability of spatial prediction models
• Importance of spatial predictor variable selection in machine learning applications -- Moving from data reproduction to spatial prediction
• Self-attention for raw optical Satellite Time Series Classification
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• A review of regularised estimation methods and cross-validation in spatiotemporal statistics
• Foundation for unbiased cross-validation of spatio-temporal models for species distribution modeling
• A Rigorous Machine Learning Analysis Pipeline for Biomedical Binary Classification: Application in Pancreatic Cancer Nested Case-control Studies with Implications for Bias Assessments
• Experimental Investigation and Evaluation of Model-based Hyperparameter Optimization
• mlr3spatiotempcv: Spatiotemporal resampling methods for machine learning in R
• From Players to Champions: A Generalizable Machine Learning Approach for Match Outcome Prediction with Insights from the FIFA World Cup
• Clutter Classification Using Deep Learning in Multiple Stages
• Machine Learning and Ensemble Approach Onto Predicting Heart Disease