Using simulation studies to evaluate statistical methods
arXiv:1712.03198 · doi:10.1002/sim.8086
Abstract
Simulation studies are computer experiments that involve creating data by pseudorandom sampling. The key strength of simulation studies is the ability to understand the behaviour of statistical methods because some 'truth' (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analysed and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting and presentation. In particular, this tutorial provides: a structured approach for planning and reporting simulation studies, which involves defining aims, data-generating mechanisms, estimands, methods and performance measures ('ADEMP'); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine that included at least one simulation study and identify areas for improvement.
31 pages, 9 figures (2 in appendix), 8 tables (1 in appendix)
References in corpus (3)
Cited by in corpus (59)
- Understanding overfitting in random forest for probability estimation: a visualization and simulation study
- A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint
- Confidence intervals of prediction accuracy measures for multivariable prediction models based on the bootstrap-based optimism correction methods
- Methods for Population Adjustment with Limited Access to Individual Patient Data: A Review and Simulation Study
- Linguistically inspired roadmap for building biologically reliable protein language models
- A Bayesian dose-response meta-analysis model: simulation study and application
- Parametric G-computation for Compatible Indirect Treatment Comparisons with Limited Individual Patient Data
- Pitfalls and potentials in simulation studies: Questionable research practices in comparative simulation studies allow for spurious claims of superiority of any method
- Clinical Prediction Models to Predict the Risk of Multiple Binary Outcomes: a comparison of approaches
- Model-based ROC (mROC) curve: examining the effect of case-mix and model calibration on the ROC plot
- Mixed effects models for healthcare longitudinal data with an informative visiting process: a Monte Carlo simulation study
- How to apply multiple imputation in propensity score matching with partially observed confounders: a simulation study and practical recommendations
- Opening the random forest black box by the analysis of the mutual impact of features
- Bayesian Variable Selection For Survival Data Using Inverse Moment Priors
- G-computation and doubly robust standardisation for continuous-time data: a comparison with inverse probability weighting
- Implementation of an alternative method for assessing competing risks: restricted mean time lost
- Explaining the optimistic performance evaluation of newly proposed methods: a cross-design validation experiment
- Analysis of in-vivo skin anisotropy using elastic wave measurements and Bayesian modelling
- Power priors for replication studies
- Obtaining interpretable parameters from reparameterizing longitudinal models: transformation matrices between growth factors in two parameter spaces
- Transportability of model-based estimands in evidence synthesis
- Evaluation of network-guided random forest for disease gene discovery
- Evaluating hybrid controls methodology in early-phase oncology trials: a simulation study based on the MORPHEUS-UC trial
- Simulation study to evaluate when Plasmode simulation is superior to parametric simulation in estimating the mean squared error of the least squares estimator in linear regression
- Non-parametric inference on calibration of predicted risks
- Analysis of dynamic restricted mean survival time based on pseudo-observations
- Model-based standardization using multiple imputation
- A Flexible Multi-Metric Bayesian Framework for Decision-Making in Phase II Multi-Arm Multi-Stage Studies
- DagSim: Combining DAG-based model structure with unconstrained data types and relations for flexible, transparent, and modularized data simulation
- Two-stage matching-adjusted indirect comparison
- Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their results
- Degree irregularity and rank probability bias in network meta-analysis
- The Sensitivity of Bayesian Kernel Machine Regression (BKMR) to Data Distribution: A Comprehensive Simulation Analysis
- On "Confirmatory" Methodological Research in Statistics and Related Fields
- How should parallel cluster randomized trials with a baseline period be analyzed? A survey of estimands and common estimators
- Analysis of cohort stepped wedge cluster-randomized trials with non-ignorable dropout via joint modeling
- Informed Bayesian survival analysis
- Inference procedures in sequential trial emulation with survival outcomes: comparing confidence intervals based on the sandwich variance estimator, bootstrap and jackknife
- Network Meta-Analysis: A Statistical Physics Perspective
- Empirical sandwich variance estimator for iterated conditional expectation g-computation
- A Bayesian hierarchical mixture cure modelling framework to utilize multiple survival datasets for long-term survivorship estimates: A case study from previously untreated metastatic melanoma
- Multiple imputation of missing covariates when using the Fine-Gray model
- Revisiting Optimal Allocations for Binary Responses: Insights from Considering Type-I Error Rate Control
- Rethinking the handling of method failure in comparison studies
- Healthy Live Births Should be Considered as Competing Events when Estimating the Total Effect of Prenatal Medication Use on Pregnancy Outcomes
- Estimands and Their Implications for Evidence Synthesis for Oncology: A Simulation Study of Treatment Switching in Meta-Analysis
- A flexible Bayesian g-formula for causal survival analyses with time-dependent confounding
- Potential outcome simulation for efficient head-to-head comparison of adaptive dose-finding designs
- Randomization-based Inference for MCP-Mod
- Methods of multi-indication meta-analysis for health technology assessment: a simulation study
- Impact of Near-Positivity Violations on IPTW-Estimated Marginal Structural Survival Models With Time-Dependent Confounding
- Outcomes truncated by death in RCTs: a simulation study on the survivor average causal effect
- Power calculation for cross-sectional stepped wedge cluster randomized trials with a time-to-event endpoint
- Predictive Analytics of Varieties of Potatoes
- Causal Effect Estimation with TMLE: Handling Missing Data and Near-Violations of Positivity
- Handling missing data when estimating causal effects with Targeted Maximum Likelihood Estimation
- Discrimination performance in illness-death models with interval-censored disease data
- An analysis of machine learning approaches for enhancing decision-making in complex discrete choice tasks
- Multilevel Metamodels: Enhancing Inference, Interpretability, and Generalizability in Monte Carlo Simulation Studies