Replication or exploration? Sequential design for stochastic simulation experiments

Abstract

We investigate the merits of replication, and provide methods for optimal design (including replicates), with the goal of obtaining globally accurate emulation of noisy computer simulation experiments. We first show that replication can be beneficial from both design and computational perspectives, in the context of Gaussian process surrogate modeling. We then develop a lookahead based sequential design scheme that can determine if a new run should be at an existing input location (i.e., replicate) or at a new one (explore). When paired with a newly developed heteroskedastic Gaussian process model, our dynamic design scheme facilitates learning of signal and noise relationships which can vary throughout the input space. We show that it does so efficiently, on both computational and statistical grounds. In addition to illustrative synthetic examples, we demonstrate performance on two challenging real-data simulation experiments, from inventory management and epidemiology.

34 pages, 9 figures

Cited by in corpus (24)

• libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculations
• Gaussian Process Assisted Active Learning of Physical Laws
• Design choice and machine learning model performances
• Sequential Bayesian experimental design for calibration of expensive simulation models
• Assessing Bayes factor surfaces using interactive visualization and computer surrogate modeling
• Evaluating Gaussian Process Metamodels and Sequential Designs for Noisy Level Set Estimation
• Achieving Robustness to Aleatoric Uncertainty with Heteroscedastic Bayesian Optimisation
• Kernels over Sets of Finite Sets using RKHS Embeddings, with Application to Bayesian (Combinatorial) Optimization
• Sensitivity Analysis in the Presence of Intrinsic Stochasticity for Discrete Fracture Network Simulations
• Predicting the Output From a Stochastic Computer Model When a Deterministic Approximation is Available
• Scalable Thompson Sampling using Sparse Gaussian Process Models
• Augmenting a simulation campaign for hybrid computer model and field data experiments
• Simulation Experiment Design for Calibration via Active Learning
• Distance-distributed design for Gaussian process surrogates
• A portfolio approach to massively parallel Bayesian optimization
• Adaptive Batching for Gaussian Process Surrogates with Application in Noisy Level Set Estimation
• Goal-oriented adaptive sampling under random field modelling of response probability distributions
• Active Learning in CNNs via Expected Improvement Maximization
• Diagnostics for Stochastic Gaussian Process Emulators
• The Kalai-Smorodinski solution for many-objective Bayesian optimization
• Active Learning for Manifold Gaussian Process Regression
• Bayes Linear Emulation of Simulator Networks
• On-site surrogates for large-scale calibration
• Active Learning with Adaptive Non-Stationary Kernel for Continuous-Fidelity Surrogate Models