Experimental Site Selection under Directional Distribution Shifts

Abstract

Multi-site studies are widely used across the social sciences, education, and medicine to learn the effects of interventions across heterogeneous populations. But deployment populations, on which a given intervention is implemented, may differ systematically from the experimental population on which the intervention is evaluated. I propose a method for selecting sites with minimax guarantees against directional distribution shifts, which allows the researcher to guard against specified threats to the external validity of a multi-site study. I use Wasserstein Distributionally Robust Optimization to select sites with guarantees against directional shifts, provide a novel cutting-plane algorithm to implement the approach, and prove its optimality with respect to minimizing worst-case shifts. I highlight the empirical performance of the method with with two semi-synthetic simulations: I re-analyse the Tennessee STAR experiment, under the assumption that the study population underrepresented low-SES schools, and \citet{Crepon_2015}, which assesses the impact of increasing access to microcredit in rural Morocco, under the assumption that the study population was skewed towards villages with larger market potential. In these examples, I show that the WDRO method minimizes mean bias, average mean squared error, and $W_1$ distance under specified directional distribution shifts.