An Evaluation Toolkit to Guide Model Selection and Cohort Definition in Causal Inference

Abstract

Real world observational data, together with causal inference, allow the estimation of causal effects when randomized controlled trials are not available. To be accepted into practice, such predictive models must be validated for the dataset at hand, and thus require a comprehensive evaluation toolkit, as introduced here. Since effect estimation cannot be evaluated directly, we turn to evaluating the various observable properties of causal inference, namely the observed outcome and treatment assignment. We developed a toolkit that expands established machine learning evaluation methods and adds several causal-specific ones. Evaluations can be applied in cross-validation, in a train-test scheme, or on the training data. Multiple causal inference methods are implemented within the toolkit in a way that allows modular use of the underlying machine learning models. Thus, the toolkit is agnostic to the machine learning model that is used. We showcase our approach using a rheumatoid arthritis cohort (consisting of about 120K patients) extracted from the IBM MarketScan(R) Research Database. We introduce an iterative pipeline of data definition, model definition, and model evaluation. Using this pipeline, we demonstrate how each of the evaluation components helps drive model selection and refinement of data extraction criteria in a way that provides more reproducible results and ensures that the causal question is answerable with available data. Furthermore, we show how the evaluation toolkit can be used to ensure that performance is maintained when applied to subsets of the data, thus allowing exploration of questions that move towards personalized medicine.

arXiv comment23 pages, 5 figures

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• Matching Methods for Causal Inference: A Review and a Look Forward
• Distinguishing cause from effect using observational data: methods and benchmarks
• Estimating individual treatment effect: generalization bounds and algorithms
• Assessing lack of common support in causal inference using Bayesian nonparametrics: Implications for evaluating the effect of breastfeeding on children's cognitive outcomes
• Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition
• Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis

Cited by in corpus (8)

• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• Causalvis: Visualizations for Causal Inference
• RealCause: Realistic Causal Inference Benchmarking
• Propensity score models are better when post-calibrated
• High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation
• Positivity Validation Detection and Explainability via Zero Fraction Multi-Hypothesis Testing and Asymmetrically Pruned Decision Trees
• Density-based interpretable hypercube region partitioning for mixed numeric and categorical data
• Finding Valid Adjustments under Non-ignorability with Minimal DAG Knowledge