Is Inference Conditional on Not Rejecting a Pre-test Less Reliable than Unconditional Inference?

Abstract

Researchers often report an estimate and confidence interval only if a specification test does not reject - for example, only if a pre-trend test supports a difference-in-differences design. Does this make the reported confidence interval less reliable? We show that when the identifying assumptions hold, the answer is no: conditional on passing the pre-test, the confidence interval may over-cover but cannot under-cover. When the assumptions fail, pre-testing can reduce coverage, but under an additional condition it still improves coverage relative to not pre-testing. A calibration to 12 differences-in-differences examples suggests modest effects of pre-testing on coverage.

arXiv comment44 pages. Minor changes compared to v3 (in particular, a few fixes in the proof of Theorem 2)