A survey of Monte Carlo methods for noisy and costly densities with application to reinforcement learning and ABC

Abstract

This survey gives an overview of Monte Carlo methodologies using surrogate models, for dealing with densities which are intractable, costly, and/or noisy. This type of problem can be found in numerous real-world scenarios, including stochastic optimization and reinforcement learning, where each evaluation of a density function may incur some computationally-expensive or even physical (real-world activity) cost, likely to give different results each time. The surrogate model does not incur this cost, but there are important trade-offs and considerations involved in the choice and design of such methodologies. We classify the different methodologies into three main classes and describe specific instances of algorithms under a unified notation. A modular scheme which encompasses the considered methods is also presented. A range of application scenarios is discussed, with special attention to the likelihood-free setting and reinforcement learning. Several numerical comparisons are also provided.

References in corpus (20)

• The pseudo-marginal approach for efficient Monte Carlo computations
• A Survey of Monte Carlo Methods for Parameter Estimation
• On the efficiency of pseudo-marginal random walk Metropolis algorithms
• Speeding Up MCMC by Efficient Data Subsampling
• Calibrate, Emulate, Sample
• On Markov chain Monte Carlo methods for tall data
• Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models
• Accelerating Asymptotically Exact MCMC for Computationally Intensive Models via Local Approximations
• Marginal likelihood computation for model selection and hypothesis testing: an extensive review
• Convergence properties of pseudo-marginal Markov chain Monte Carlo algorithms
• Active emulation of computer codes with Gaussian processes -- Application to remote sensing
• Delayed acceptance particle MCMC for exact inference in stochastic kinetic models
• Importance Gaussian Quadrature
• Speeding Up MCMC by Delayed Acceptance and Data Subsampling
• Efficient acquisition rules for model-based approximate Bayesian computation
• The Recycling Gibbs Sampler for Efficient Learning
• Variational Bayesian Monte Carlo with Noisy Likelihoods
• Query Efficient Posterior Estimation in Scientific Experiments via Bayesian Active Learning
• Adaptive Independent Sticky MCMC algorithms
• Adaptive quadrature schemes for Bayesian inference via active learning