Statistical quantification of confounding bias in predictive modelling

Abstract

The lack of non-parametric statistical tests for confounding bias significantly hampers the development of robust, valid and generalizable predictive models in many fields of research. Here I propose the partial and full confounder tests, which, for a given confounder variable, probe the null hypotheses of unconfounded and fully confounded models, respectively. The tests provide a strict control for Type I errors and high statistical power, even for non-normally and non-linearly dependent predictions, often seen in machine learning. Applying the proposed tests on models trained on functional brain connectivity data from the Human Connectome Project and the Autism Brain Imaging Data Exchange dataset reveals confounders that were previously unreported or found to be hard to correct for with state-of-the-art confound mitigation approaches. The tests, implemented in the package mlconfound (https://mlconfound.readthedocs.io), can aid the assessment and improvement of the generalizability and neurobiological validity of predictive models and, thereby, foster the development of clinically useful machine learning biomarkers.

arXiv comment20 pages, 7 figures. The manuscript is associated with the the python package `mlconfound`: https://mlconfound.readthedocs.io See manuscript repository, including fully reproducible analysis code, here: https://github.com/pni-lab/mlconfound-manuscript

References in corpus (6)

• Scikit-learn: Machine Learning in Python
• The Hardness of Conditional Independence Testing and the Generalised Covariance Measure
• Detect and Correct Bias in Multi-Site Neuroimaging Datasets
• Exact testing with random permutations
• Statistical quantification of confounding bias in predictive modelling
• Measuring the effects of confounders in medical supervised classification problems: the Confounding Index (CI)

Cited by in corpus (1)

• Statistical quantification of confounding bias in predictive modelling