Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values

Abstract

The influential claim that most published results are false raised concerns about the trustworthiness and integrity of science. Since then, there have been numerous attempts to examine the rate of false-positive results that have failed to settle this question empirically. Here we propose a new way to estimate the false positive risk and apply the method to the results of (randomized) clinical trials in top medical journals. Contrary to claims that most published results are false, we find that the traditional significance criterion of $\alpha =.05$ produces a false positive risk of 13%. Adjusting $\alpha$ to .01 lowers the false positive risk to less than 5%. However, our method does provide clear evidence of publication bias that leads to inflated effect size estimates. These results provide a solid empirical foundation for evaluations of the trustworthiness of medical research.

References in corpus (4)

• Bayesian Model-Averaged Meta-Analysis in Medicine
• Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics
• The assessment of replication success based on relative effect size
• The sceptical Bayes factor for the assessment of replication success