Algorithm-agnostic significance testing in supervised learning with multimodal data

Abstract

Valid statistical inference is crucial for decision-making but difficult to obtain in supervised learning with multimodal data, e.g., combinations of clinical features, genomic data, and medical images. Multimodal data often warrants the use of black-box algorithms, for instance, random forests or neural networks, which impede the use of traditional variable significance tests. We address this problem by proposing the use of COvariance Measure Tests (COMETs), which are calibrated and powerful tests that can be combined with any sufficiently predictive supervised learning algorithm. We apply COMETs to several high-dimensional, multimodal data sets to illustrate (i) variable significance testing for finding relevant mutations modulating drug-activity, (ii) modality selection for predicting survival in liver cancer patients with multiomics data, and (iii) modality selection with clinical features and medical imaging data. In all applications, COMETs yield results consistent with domain knowledge without requiring data-driven pre-processing which may invalidate type I error control. These novel applications with high-dimensional multimodal data corroborate prior results on the power and robustness of COMETs for significance testing. COMETs are implemented in the comets R package available on CRAN and pycomets Python library available on GitHub. Source code for reproducing all results is available at https://github.com/LucasKook/comets. All data sets used in this work are openly available.

References in corpus (12)

• ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R
• Valid post-selection inference
• MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
• The Hardness of Conditional Independence Testing and the Generalised Covariance Measure
• Risk of Bias in Chest Radiography Deep Learning Foundation Models
• A general framework for inference on algorithm-agnostic variable importance
• Conditional Independence Testing in Hilbert Spaces with Applications to Functional Data Analysis
• The Projected Covariance Measure for assumption-lean variable significance testing
• Rank-transformed subsampling: inference for multiple data splitting and exchangeable p-values
• Model-based causal feature selection for general response types
• Nonparametric Conditional Local Independence Testing
• A general framework for the analysis of kernel-based tests