Variational Sequential Optimal Experimental Design using Reinforcement Learning

Abstract

We present variational sequential optimal experimental design (vsOED), a novel method for optimally designing a finite sequence of experiments within a Bayesian framework with information-theoretic criteria. vsOED employs a one-point reward formulation with variational posterior approximations, providing a provable lower bound to the expected information gain. Numerical methods are developed following an actor-critic reinforcement learning approach, including derivation and estimation of variational and policy gradients to optimize the design policy, and posterior approximation using Gaussian mixture models and normalizing flows. vsOED accommodates nuisance parameters, implicit likelihoods, and multiple candidate models, while supporting flexible design criteria that can target designs for model discrimination, parameter inference, goal-oriented prediction, and their weighted combinations. We demonstrate vsOED across various engineering and science applications, illustrating its superior sample efficiency compared to existing sequential experimental design algorithms.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Representation Learning with Contrastive Predictive Coding
• Normalizing Flows: An Introduction and Review of Current Methods
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Solving inverse problems using conditional invertible neural networks
• Optimal experimental design: Formulations and computations
• A Consistent Bayesian Formulation for Stochastic Inverse Problems Based on Push-forward Measures
• Goal-Oriented Optimal Design of Experiments for Large-Scale Bayesian Linear Inverse Problems
• Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning
• Sequential Bayesian optimal experimental design via approximate dynamic programming
• Convergence of Probability Densities using Approximate Models for Forward and Inverse Problems in Uncertainty Quantification
• Variational Bayesian experimental design for geophysical applications: seismic source location, amplitude versus offset inversion, and estimating CO2 saturations in a subsurface reservoir
• Variational Bayesian Optimal Experimental Design with Normalizing Flows
• Prediction-Oriented Bayesian Active Learning
• Probabilistic Bayesian optimal experimental design using conditional normalizing flows
• On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows
• Goal-Oriented Bayesian Optimal Experimental Design for Nonlinear Models using Markov Chain Monte Carlo
• Amortized Active Causal Induction with Deep Reinforcement Learning