On "Confirmatory" Methodological Research in Statistics and Related Fields

Abstract

Empirical substantive research, such as in the life or social sciences, is commonly categorized into the two modes exploratory and confirmatory, both of which are essential to scientific progress. The former is also referred to as hypothesis-generating or data-contingent research, while the latter is also called hypothesis-testing research. In the context of empirical methodological research in statistics, however, the exploratory-confirmatory distinction has received very little attention so far. Our paper aims to fill this gap. First, we revisit the concept of empirical methodological research through the lens of the exploratory-confirmatory distinction. Second, we examine current practice with respect to this distinction through a literature survey including 115 articles from the field of biostatistics. Third, we provide practical recommendations toward a more appropriate design, interpretation, and reporting of empirical methodological research in light of this distinction. In particular, we argue that both modes of research are crucial to methodological progress, but that most published studies -- even if sometimes disguised as confirmatory -- are essentially exploratory in nature. We emphasize that it may be adequate to consider empirical methodological research as a continuum between "pure" exploration and "strict" confirmation, recommend transparently reporting the mode of conducted research within the spectrum between exploratory and confirmatory, and stress the importance of study protocols written before conducting the study, especially in confirmatory methodological research.

36 pages; version published in Statistics in Medicine

References in corpus (9)

• Using simulation studies to evaluate statistical methods
• A Plea for Neutral Comparison Studies in Computational Sciences
• Large-scale benchmark study of survival prediction methods using multi-omics data
• Phases of methodological research in biostatistics - building the evidence base for new methods
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• Pitfalls and potentials in simulation studies: Questionable research practices in comparative simulation studies allow for spurious claims of superiority of any method
• Benchmarking in cluster analysis: A white paper
• Explaining the optimistic performance evaluation of newly proposed methods: a cross-design validation experiment
• Fair coins tend to land on the same side they started: Evidence from 350,757 flips