Optimal Bayesian design for model discrimination via classification

Abstract

Performing optimal Bayesian design for discriminating between competing models is computationally intensive as it involves estimating posterior model probabilities for thousands of simulated datasets. This issue is compounded further when the likelihood functions for the rival models are computationally expensive. A new approach using supervised classification methods is developed to perform Bayesian optimal model discrimination design. This approach requires considerably fewer simulations from the candidate models than previous approaches using approximate Bayesian computation. Further, it is easy to assess the performance of the optimal design through the misclassification error rate. The approach is particularly useful in the presence of models with intractable likelihoods but can also provide computational advantages when the likelihoods are manageable.

Major revision of previous version: use trees with cross-validation and random forests with out-of-bag predictions to estimate expected loss; training set-based loss estimates are not used anymore; a post-processing step utilising Gaussian process regression is added to the optimisation routine; examples were re-run; extensive reorganisation of contents

References in corpus (7)

• Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem
• Statistical Modeling of Spatial Extremes
• Stationary max-stable fields associated to negative definite functions
• Optimal discrimination designs
• ABC model selection for spatial extremes models applied to South Australian maximum temperature data
• A synthetic likelihood-based Laplace approximation for efficient design of biological processes
• A design criterion for symmetric model discrimination based on nominal confidence sets

Cited by in corpus (4)

• Variational Bayesian experimental design for geophysical applications: seismic source location, amplitude versus offset inversion, and estimating CO2 saturations in a subsurface reservoir
• Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Bounds
• A design criterion for symmetric model discrimination based on nominal confidence sets
• Intelligent data collection for network discrimination in material flow analysis using Bayesian optimal experimental design