Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique

Abstract

Freedman [Adv. in Appl. Math. 40 (2008) 180-193; Ann. Appl. Stat. 2 (2008) 176-196] critiqued ordinary least squares regression adjustment of estimated treatment effects in randomized experiments, using Neyman's model for randomization inference. Contrary to conventional wisdom, he argued that adjustment can lead to worsened asymptotic precision, invalid measures of precision, and small-sample bias. This paper shows that in sufficiently large samples, those problems are either minor or easily fixed. OLS adjustment cannot hurt asymptotic precision when a full set of treatment-covariate interactions is included. Asymptotically valid confidence intervals can be constructed with the Huber-White sandwich standard error estimator. Checks on the asymptotic approximations are illustrated with data from Angrist, Lang, and Oreopoulos's [Am. Econ. J.: Appl. Econ. 1:1 (2009) 136--163] evaluation of strategies to improve college students' achievement. The strongest reasons to support Freedman's preference for unadjusted estimates are transparency and the dangers of specification search.

Published in at http://dx.doi.org/10.1214/12-AOAS583 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (2)

• Exact and asymptotically robust permutation tests
• On regression adjustments in experiments with several treatments

Cited by in corpus (84)

• Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment
• Lasso adjustments of treatment effect estimates in randomized experiments
• High-dimensional regression adjustments in randomized experiments
• Sharp bounds on the variance in randomized experiments
• Regression adjustment in completely randomized experiments with a diverging number of covariates
• Regression analysis for covariate-adaptive randomization: A robust and efficient inference perspective
• Randomization Tests for Weak Null Hypotheses in Randomized Experiments
• Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods
• Generalized Optimal Matching Methods for Causal Inference
• Regression Discontinuity Designs Using Covariates
• Shrinkage Estimators in Online Experiments
• Design-Based Ratio Estimators and Central Limit Theorems for Clustered, Blocked RCTs
• Gaussian Prepivoting for Finite Population Causal Inference
• Central limit theorems via Stein's method for randomized experiments under interference
• Insights on Variance Estimation for Blocked and Matched Pairs Designs
• Inference on Average Treatment Effect under Minimization and Other Covariate-Adaptive Randomization Methods
• Covariate Adjustment in Randomized Clinical Trials with Missing Covariate and Outcome Data
• Limitless Regression Discontinuity
• Model-assisted analysis of covariance estimators for stepped wedge cluster randomized experiments
• Covariate-adjusted Fisher randomization tests for the average treatment effect
• A paradox from randomization-based causal inference
• Asymptotic Theory of Rerandomization in Treatment-Control Experiments
• Improved Precision in Estimating Average Treatment Effects
• Machine Learning for Variance Reduction in Online Experiments
• Randomization Tests that Condition on Non-Categorical Covariate Balance
• Nonparametric identification is not enough, but randomized controlled trials are
• Large Linguistic Models: Investigating LLMs' metalinguistic abilities
• The Generalized Oaxaca-Blinder Estimator
• A nonparametric Bayesian analysis of heterogeneous treatment effects in digital experimentation
• Causal Inference: A Missing Data Perspective
• Optimal Rerandomization via a Criterion that Provides Insurance Against Failed Experiments
• Is it who you are or where you are? Accounting for compositional differences in cross-site treatment variation
• An Empirical Comparison of Parametric and Permutation Tests for Regression Analysis of Randomized Experiments
• Covariate adjustment in randomization-based causal inference for 2K factorial designs
• Regression-adjusted average treatment effect estimates in stratified randomized experiments
• Propensity Score Weighting for Covariate Adjustment in Randomized Clinical Trials
• Regression Discontinuity Design with Potentially Many Covariates
• Decomposing Treatment Effect Variation
• Anytime-Valid Linear Models and Regression Adjusted Causal Inference in Randomized Experiments
• Model-assisted analyses of cluster-randomized experiments
• The Local Randomization Framework for Regression Discontinuity Designs: A Review and Some Extensions
• Stable discovery of interpretable subgroups via calibration in causal studies
• Bridging preference-based instrumental variable studies and cluster-randomized encouragement experiments: study design, noncompliance, and average cluster effect ratio
• Design-based theory for Lasso adjustment in randomized block experiments and rerandomized experiments
• Towards Optimal Variance Reduction in Online Controlled Experiments
• A Lasso-OLS Hybrid Approach to Covariate Selection and Average Treatment Effect Estimation for Clustered RCTs Using Design-Based Methods
• Randomization-Based Causal Inference from Unbalanced 2^2 Split-Plot Designs
• Improving the Power of the Randomization Test
• Regression adjustments for estimating the global treatment effect in experiments with interference
• Model selection for estimation of causal parameters
• Heterogeneous Treatment Effect Estimation through Deep Learning
• Randomization does not imply unconfoundedness
• Model-assisted complier average treatment effect estimates in randomized experiments with non-compliance and a binary outcome
• Randomization Inference for Treatment Effect Variation
• Sharpening randomization-based causal inference for $2^2$ factorial designs with binary outcomes
• The LOOP Estimator: Adjusting for Covariates in Randomized Experiments
• Rejective Sampling, Rerandomization and Regression Adjustment in Survey Experiments
• Unifying Design-based Inference: On Bounding and Estimating the Variance of any Linear Estimator in any Experimental Design
• Enhanced inference for distributions and quantiles of individual treatment effects in various experiments
• Covariate adjustment in subgroup analyses of randomized clinical trials: A propensity score approach
• Sharp bounds for variance of treatment effect estimators in the finite population in the presence of covariates
• Robust and Efficient Estimation of Potential Outcome Means under Random Assignment
• Rerandomization and Regression Adjustment
• Two seemingly paradoxical results in linear models: the variance inflation factor and the analysis of covariance
• Analyzing two-stage experiments in the presence of interference
• Dealing With Ratio Metrics in A/B Testing at the Presence of Intra-User Correlation and Segments
• Efficient Balanced Treatment Assignments for Experimentation
• Randomized Controlled Trials without Data Retention
• Randomization-based joint central limit theorem and efficient covariate adjustment in stratified $2^K$ factorial experiments
• Randomization-based Test for Censored Outcomes: A New Look at the Logrank Test
• Bridging Finite and Super Population Causal Inference
• Rerandomization with Diminishing Covariate Imbalance and Diverging Number of Covariates
• Reasoning about Interference Between Units
• Variance Reduction for Causal Inference
• Unifying Design-based Inference: A New Variance Estimation Principle
• Leveraging Population Outcomes to Improve the Generalization of Experimental Results
• Network experimentation at scale
• The Frisch--Waugh--Lovell Theorem for Standard Errors
• Improving Precision through Adjustment for Prognostic Variables in Group Sequential Trial Designs: Impact of Baseline Variables, Short-Term Outcomes, and Treatment Effect Heterogeneity
• Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence
• A Comparison of Methods of Inference in Randomized Experiments from a Restricted Set of Allocations
• Cluster-Robust Variance Estimation for Dyadic Data
• The P-LOOP Estimator: Covariate Adjustment for Paired Experiments
• Randomization Inference for Peer Effects