Empowering Bayesian Neural Networks with Functional Priors through Anchored Ensembling for Mechanics Surrogate Modeling Applications

Abstract

In recent years, neural networks (NNs) have become increasingly popular for surrogate modeling tasks in mechanics and materials modeling applications. While traditional NNs are deterministic functions that rely solely on data to learn the input--output mapping, casting NN training within a Bayesian framework allows to quantify uncertainties, in particular epistemic uncertainties that arise from lack of training data, and to integrate a priori knowledge via the Bayesian prior. However, the high dimensionality and non-physicality of the NN parameter space, and the complex relationship between parameters (NN weights) and predicted outputs, renders both prior design and posterior inference challenging. In this work we present a novel BNN training scheme based on anchored ensembling that can integrate a priori information available in the function space, from e.g. low-fidelity models. The anchoring scheme makes use of low-rank correlations between NN parameters, learnt from pre-training to realizations of the functional prior. We also perform a study to demonstrate how correlations between NN weights, which are often neglected in existing BNN implementations, is critical to appropriately transfer knowledge between the function-space and parameter-space priors. Performance of our novel BNN algorithm is first studied on a small 1D example to illustrate the algorithm's behavior in both interpolation and extrapolation settings. Then, a thorough assessment is performed on a multi--input--output materials surrogate modeling example, where we demonstrate the algorithm's capabilities both in terms of accuracy and quality of the uncertainty estimation, for both in-distribution and out-of-distribution data.

arXiv comment24 pages, 14 figures

References in corpus (13)

• Scikit-learn: Machine Learning in Python
• Overcoming catastrophic forgetting in neural networks
• MCMC using Hamiltonian dynamics
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Bayesian Physics-Informed Neural Networks for real-world nonlinear dynamical systems
• Efficient Uncertainty Quantification for Dynamic Subsurface Flow with Surrogate by Theory-guided Neural Network
• Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling
• Learning Functional Priors and Posteriors from Data and Physics
• Solution of Physics-based Bayesian Inverse Problems with Deep Generative Priors
• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors
• Neural Network Training Using ${\ell }_1$-Regularization and Bi-fidelity Data
• Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning
• Bayesian Neural Networks with Domain Knowledge Priors