Validating Predictions of Unobserved Quantities

Abstract

The ultimate purpose of most computational models is to make predictions, commonly in support of some decision-making process (e.g., for design or operation of some system). The quantities that need to be predicted (the quantities of interest or QoIs) are generally not experimentally observable before the prediction, since otherwise no prediction would be needed. Assessing the validity of such extrapolative predictions, which is critical to informed decision-making, is challenging. In classical approaches to validation, model outputs for observed quantities are compared to observations to determine if they are consistent. By itself, this consistency only ensures that the model can predict the observed quantities under the conditions of the observations. This limitation dramatically reduces the utility of the validation effort for decision making because it implies nothing about predictions of unobserved QoIs or for scenarios outside of the range of observations. However, there is no agreement in the scientific community today regarding best practices for validation of extrapolative predictions made using computational models. The purpose of this paper is to propose and explore a validation and predictive assessment process that supports extrapolative predictions for models with known sources of error. The process includes stochastic modeling, calibration, validation, and predictive assessment phases where representations of known sources of uncertainty and error are built, informed, and tested. The proposed methodology is applied to an illustrative extrapolation problem involving a misspecified nonlinear oscillator.

References in corpus (1)

• A survey of cross-validation procedures for model selection

Cited by in corpus (14)

• A critical review of statistical calibration/prediction models handling data inconsistency and model inadequacy
• The parameters uncertainty inflation fallacy
• Global Sensitivity Analysis and Estimation of Model Error, Toward Uncertainty Quantification in Scramjet Computations
• Understanding the rheology of kaolinite clay suspensions using Bayesian inference
• Enforcing Boundary Conditions on Physical Fields in Bayesian Inversion
• Solving Irregular and Data-enriched Differential Equations using Deep Neural Networks
• Optimal Design of Validation Experiments for the Prediction of Quantities of Interest
• Embedded Model Error Representation for Bayesian Model Calibration
• Quantification of Model Uncertainty in RANS Simulations: A Review
• Embedded discrepancy operators in reduced models of interacting species
• A Stochastic Operator Approach to Model Inadequacy with Applications to Contaminant Transport
• Stochastic Subspace via Probabilistic Principal Component Analysis for Characterizing Model Error
• EnLLVM: Ensemble Based Nonlinear Bayesian Filtering Using Linear Latent Variable Models
• Embedded Model Form Uncertainty Quantification with Measurement Noise for Bayesian Model Calibration