Assessing and tuning brain decoders: cross-validation, caveats, and guidelines

Abstract

Decoding, ie prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within-and across-subject predictions, on multiple datasets --anatomical and functional MRI and MEG-- and simulations. Theory and experiments outline that the popular " leave-one-out " strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be more favorable to use sane defaults, in particular for non-sparse decoders.

NeuroImage, Elsevier, 2016

References in corpus (2)

• Total variation regularization for fMRI-based prediction of behaviour
• Small-sample Brain Mapping: Sparse Recovery on Spatially Correlated Designs with Randomization and Clustering

Cited by in corpus (24)

• Structural neuroimaging as clinical predictor: a review of machine learning applications
• Reproducible evaluation of classification methods in Alzheimer's disease: framework and application to MRI and PET data
• Systematic Misestimation of Machine Learning Performance in Neuroimaging Studies of Depression
• Brain Network Construction and Classification Toolbox (BrainNetClass)
• Toward Generalizable Machine Learning Models in Speech, Language, and Hearing Sciences: Estimating Sample Size and Reducing Overfitting
• On Leakage in Machine Learning Pipelines
• Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research
• Causality in cognitive neuroscience: concepts, challenges, and distributional robustness
• The relationship between linguistic expression and symptoms of depression, anxiety, and suicidal thoughts: A longitudinal study of blog content
• FetMRQC: a robust quality control system for multi-centric fetal brain MRI
• Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
• Reproducible evaluation of diffusion MRI features for automatic classification of patients with Alzheimers disease
• FetMRQC: Automated Quality Control for fetal brain MRI
• Solving large-scale MEG/EEG source localization and functional connectivity problems simultaneously using state-space models
• Modeling Large-Scale Walking and Cycling Networks: A Machine Learning Approach Using Mobile Phone and Crowdsourced Data
• Understanding Graph Isomorphism Network for rs-fMRI Functional Connectivity Analysis
• Vocal markers from sustained phonation in Huntington's Disease
• Voxel selection framework based on meta-heuristic search and mutual information for brain decoding
• Interpreting Encoding and Decoding Models
• Decoding multimodal behavior using time differences of MEG events
• Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance
• Aggregated Hold-Out
• Rewiring Human Brain Networks via Lightweight Dynamic Connectivity Framework: An EEG-Based Stress Validation
• MAGIC: Multi-scale Heterogeneity Analysis and Clustering for Brain Diseases