Can parametric statistical methods be trusted for fMRI based group studies?

Abstract

The most widely used task fMRI analyses use parametric methods that depend on a variety of assumptions. While individual aspects of these fMRI models have been evaluated, they have not been evaluated in a comprehensive manner with empirical data. In this work, a total of 2 million random task fMRI group analyses have been performed using resting state fMRI data, to compute empirical familywise error rates for the software packages SPM, FSL and AFNI, as well as a standard non-parametric permutation method. While there is some variation, for a nominal familywise error rate of 5% the parametric statistical methods are shown to be conservative for voxel-wise inference and invalid for cluster-wise inference; in particular, cluster size inference with a cluster defining threshold of p = 0.01 generates familywise error rates up to 60%. We conduct a number of follow up analyses and investigations that suggest the cause of the invalid cluster inferences is spatial auto correlation functions that do not follow the assumed Gaussian shape. By comparison, the non-parametric permutation test, which is based on a small number of assumptions, is found to produce valid results for voxel as well as cluster wise inference. Using real task data, we compare the results between one parametric method and the permutation test, and find stark differences in the conclusions drawn between the two using cluster inference. These findings speak to the need of validating the statistical methods being used in the neuroimaging field.

Cited by in corpus (66)

• Publication bias and the canonization of false facts
• FMRI Clustering and False Positive Rates
• Accurate autocorrelation modeling substantially improves fMRI reliability
• Persistent homology of time-dependent functional networks constructed from coupled time series
• Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions
• Cluster Failure Revisited: Impact of First Level Design and Data Quality on Cluster False Positive Rates
• Point estimates, Simpson's paradox and nonergodicity in biological sciences
• The canonical semantic network supports residual language function in chronic post-stroke aphasia
• Fast Bayesian whole-brain fMRI analysis with spatial 3D priors
• The Same Analysis Approach: Practical protection against the pitfalls of novel neuroimaging analysis methods
• Is K-fold cross validation the best model selection method for Machine Learning?
• Reproducibility, Replicability, and Repeatability: A survey of reproducible research with a focus on high performance computing
• Neural correlates of episodic memory in the Memento cohort
• Statistical reform and the replication crisis
• Discovering and Deciphering Relationships Across Disparate Data Modalities
• FMRI Clustering in AFNI: False Positive Rates Redux
• Deep Learning in current Neuroimaging: a multivariate approach with power and type I error control but arguable generalization ability
• Reply to Cox et al. and Kessler et al.: Data and code sharing is the way forward for fMRI
• Code Replicability in Computer Graphics
• A connection between the pattern classification problem and the General Linear Model for statistical inference
• New methods for multiple testing in permutation inference for the general linear model
• A Bayesian Heteroscedastic GLM with Application to fMRI Data with Motion Spikes
• Processing, evaluating and understanding FMRI data with afni_proc.py
• A shortcut for Hommel's procedure in linearithmic time
• On the evaluation of research software: the CDUR procedure
• Notip: Non-parametric True Discovery Proportion control for brain imaging
• Expanding the scope of statistical computing: Training statisticians to be software engineers
• Preparing fMRI Data for Statistical Analysis
• A simple model suggesting economically rational sample-size choice drives irreproducibility
• Neurofeminism: feminist critiques of research on sex/gender differences in the neurosciences
• Digital Scientific Notations as a Human-Computer Interface in Computer-Aided Research
• The BioDynaMo Project: Creating a Platform for Large-Scale Reproducible Biological Simulations
• A hypothesis-driven method based on machine learning for neuroimaging data analysis
• Faster Family-wise Error Control for Neuroimaging with a Parametric Bootstrap
• Statistical Agnostic Regression: a machine learning method to validate regression models
• A computational hierarchy in human cortex
• Estimation of Expected Euler Characteristic Curves of Nonstationary Smooth Gaussian Random Fields
• Derandomizing Knockoffs
• Formal Methods: From Academia to Industrial Practice. A Travel Guide
• The SPDE approach for Gaussian and non-Gaussian fields: 10 years and still running
• A Review on MR Based Human Brain Parcellation Methods
• Interpretable High-Dimensional Inference Via Score Projection with an Application in Neuroimaging
• Reply to Chen et al.: Parametric methods for cluster inference perform worse for two-sided t-tests
• Assessing Dynamic Effects on a Bayesian Matrix-Variate Dynamic Linear Model: an Application to fMRI Data Analysis
• Which Findings from the Functional Neuromaging Literature Can We Trust?
• Analysis of BOLD fMRI Signal Preprocessing Pipeline on Different Datasets while Reducing False Positive Rates
• Computational and informatics advances for reproducible data analysis in neuroimaging
• A practical model-based segmentation approach for improved activation detection in single-subject functional Magnetic Resonance Imaging studies
• Is Bonferroni correction more sensitive than Random Field Theory for most fMRI studies?
• A sensitivity analysis to quantify the impact of neuroimaging preprocessing strategies on subsequent statistical analyses
• Residual permutation test for regression coefficient testing
• Statistical Agnostic Mapping: a Framework in Neuroimaging based on Concentration Inequalities
• A Serverless Tool for Platform Agnostic Computational Experiment Management
• Identification of brain states, transitions, and communities using functional MRI
• Adaptive Huber Regression on Markov-dependent Data
• FACT: Fast closed testing for exchangeable local tests
• Learning Interpretable Models for Coupled Networks Under Domain Constraints
• A Test for Shared Patterns in Cross-modal Brain Activation Analysis
• Human Echolocation in Static Situations: Auditory Models of Detection Thresholds for Distance, Pitch, Loudness and Timbre
• No crisis should go to waste
• Rethinking the Experiment
• A defense of using resting state fMRI as null data for estimating false positive rates
• Multi-Objective Cognitive Model: a supervised approach for multi-subject fMRI analysis
• Guidelines for data analysis scripts
• Small Animal Multivariate Brain Analysis (SAMBA): A High Throughput Pipeline with a Validation Framework
• On the term "randomization test"