Information, Privacy and Stability in Adaptive Data Analysis
arXiv:1706.00820
Abstract
Traditional statistical theory assumes that the analysis to be performed on a given data set is selected independently of the data themselves. This assumption breaks downs when data are re-used across analyses and the analysis to be performed at a given stage depends on the results of earlier stages. Such dependency can arise when the same data are used by several scientific studies, or when a single analysis consists of multiple stages. How can we draw statistically valid conclusions when data are re-used? This is the focus of a recent and active line of work. At a high level, these results show that limiting the information revealed by earlier stages of analysis controls the bias introduced in later stages by adaptivity. Here we review some known results in this area and highlight the role of information-theoretic concepts, notably several one-shot notions of mutual information.
15 pages, first drafted February 2017. A version of this survey appears in the Information Theory Society Newsletter
References in corpus (11)
- Valid post-selection inference
- Duality Between Smooth Min- and Max-Entropies
- Optimal Inference After Model Selection
- Chain Rules for Smooth Min- and Max-Entropies
- Smooth Max-Information as One-Shot Generalization for Mutual Information
- Selective sampling after solving a convex problem
- On-Average KL-Privacy and its equivalence to Generalization for Max-Entropy Mechanisms
- MAGIC: a general, powerful and tractable method for selective inference
- Bayesian Adaptive Data Analysis Guarantees from Subgaussianity
- A Minimax Theory for Adaptive Data Analysis
- Challenges in Bayesian Adaptive Data Analysis