paper

Reevaluating Causal Estimation Methods with Data from a Product Release

arXiv:2601.11845

Abstract

Machine learning methods estimate flexible relationships between confounders, treatments and outcomes, making unconfoundedness assumptions in causal analysis more palatable. How successful are these approaches in recovering ground truth baselines? We analyze new data covering an experimental rollout of a software product feature for one group of users and a voluntary opt-in to the same feature for another similar group. Recovering causal effects from nonrandomized data is feasible--but only with careful modeling choices. We offer best practices for more credible treatment effect estimation in modern, high-dimensional datasets and evaluate which tests provide meaningful signals about the reliability of estimated effects.

arXiv comment53 pages