Confidence Sets and Hypothesis Testing in a Likelihood-Free Inference Setting
arXiv:2002.10399
Abstract
Parameter estimation, statistical tests and confidence sets are the cornerstones of classical statistics that allow scientists to make inferences about the underlying process that generated the observed data. A key question is whether one can still construct hypothesis tests and confidence sets with proper coverage and high power in a so-called likelihood-free inference (LFI) setting; that is, a setting where the likelihood is not explicitly known but one can forward-simulate observable data according to a stochastic model. In this paper, we present (Approximate Computation via Odds Ratio Estimation), a frequentist approach to LFI that first formulates the classical likelihood ratio test (LRT) as a parametrized classification problem, and then uses the equivalence of tests and confidence sets to build confidence regions for parameters of interest. We also present a goodness-of-fit procedure for checking whether the constructed tests and confidence regions are valid. is based on the key observation that the LRT statistic, the rejection probability of the test, and the coverage of the confidence set are conditional distribution functions which often vary smoothly as a function of the parameters of interest. Hence, instead of relying solely on samples simulated at fixed parameter settings (as is the convention in standard Monte Carlo solutions), one can leverage machine learning tools and data simulated in the neighborhood of a parameter to improve estimates of quantities of interest. We demonstrate the efficacy of with both theoretical and empirical results. Our implementation is available on Github.
arXiv comment20 pages, 8 figures, 6 tables, 4 algorithm boxes
References in corpus (20)
- Scikit-learn: Machine Learning in Python
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC
- Practical Bayesian Optimization of Machine Learning Algorithms
- A Unified Approach to the Classical Statistical Analysis of Small Signals
- The frontier of simulation-based inference
- Deep Learning and its Application to LHC Physics
- Validating Bayesian Inference Algorithms with Simulation-Based Calibration
- Fast likelihood-free cosmology with neural density estimators and active learning
- Learning in Implicit Generative Models
- Mining gold from implicit models to improve likelihood-free inference
- Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
- Bayesian optimisation for likelihood-free cosmological inference
- Likelihood-free inference with emulator networks
- Introduction to Monte Carlo methods
- Variational Bayes with Synthetic Likelihood
- ABC-CDE: Towards Approximate Bayesian Computation with Complex High-Dimensional Data and Limited Simulations
- Dynamic Likelihood-free Inference via Ratio Estimation (DIRE)
- Modeling Smooth Backgrounds and Generic Localized Signals with Gaussian Processes
- Lectures on Statistics in Theory: Prelude to Statistics in Practice