False confidence, non-additive beliefs, and valid statistical inference

Abstract

Statistics has made tremendous advances since the times of Fisher, Neyman, Jeffreys, and others, but the fundamental and practically relevant questions about probability and inference that puzzled our founding fathers remain unanswered. To bridge this gap, I propose to look beyond the two dominating schools of thought and ask the following three questions: what do scientists need out of statistics, do the existing frameworks meet these needs, and, if not, how to fill the void? To the first question, I contend that scientists seek to convert their data, posited statistical model, etc., into calibrated degrees of belief about quantities of interest. To the second question, I argue that any framework that returns additive beliefs, i.e., probabilities, necessarily suffers from {\em false confidence}---certain false hypotheses tend to be assigned high probability---and, therefore, risks systematic bias. This reveals the fundamental importance of {\em non-additive beliefs} in the context of statistical inference. But non-additivity alone is not enough so, to the third question, I offer a sufficient condition, called {\em validity}, for avoiding false confidence, and present a framework, based on random sets and belief functions, that provably meets this condition. Finally, I discuss characterizations of p-values and confidence intervals in terms of valid non-additive beliefs, which imply that users of these classical procedures are already following the proposed framework without knowing it.

60 pages, 12 figures. Comments welcome at https://www.researchers.one/article/2019-02-1

References in corpus (16)

• Abandon Statistical Significance
• Decision-Making with Belief Functions: a Review
• Inferential models: A framework for prior-free posterior probabilistic inference
• Frequentist statistics as a theory of inductive inference
• Calibrating general posterior credible regions
• The Sources of Kolmogorov's Grundbegriffe
• Satellite conjunction analysis and the false confidence theorem
• On an inferential model construction using generalized associations
• Gibbs posterior inference on the minimum clinically important difference
• Frameworks for prior-free posterior probabilistic inference
• Exact prior-free probabilistic inference on the heritability coefficient in a linear mixed model
• A statistical inference course based on p-values
• A mathematical characterization of confidence as valid belief
• Optimal inferential models for a Poisson mean
• Valid uncertainty quantification about the model in a linear regression setting
• Fiducial on a string

Cited by in corpus (17)

• Satellite conjunction analysis and the false confidence theorem
• Belief functions induced by random fuzzy sets: A general framework for representing uncertain and fuzzy evidence
• A comparison of learning rate selection methods in generalized Bayesian inference
• Gibbs posterior concentration rates under sub-exponential type losses
• Model-free posterior inference on the area under the receiver operating characteristic curve
• Validity, consonant plausibility measures, and conformal prediction
• Valid inferential models for prediction in supervised learning problems
• Gibbs posterior inference on multivariate quantiles
• Direct and approximately valid probabilistic inference on a class of statistical functionals
• Generalized inferential models for censored data
• A possibility-theoretic solution to Basu's Bayesian--frequentist via media
• Uncertainty Quantification of Surrogate Models using Conformal Prediction
• Inferential models and possibility measures
• Asymptotic efficiency of inferential models and a possibilistic Bernstein--von Mises theorem
• Large-sample theory for inferential models: a possibilistic Bernstein--von Mises theorem
• Generalized inferential models for meta-analyses based on few studies
• Computationally efficient variational-like approximations of possibilistic inferential models