Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial

Abstract

On top of machine learning models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Toward this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. Two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage and the remaining useful life prediction of turbofan engines.

References in corpus (63)

• Intriguing properties of neural networks
• Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
• Variational Inference: A Review for Statisticians
• Spectral Normalization for Generative Adversarial Networks
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• MCMC using Hamiltonian dynamics
• Popular Ensemble Methods: An Empirical Study
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Fourier Neural Operator for Parametric Partial Differential Equations
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Gaussian process regression for forecasting battery state of health
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• Transfusion: Understanding Transfer Learning for Medical Imaging
• A Physics Informed Machine Learning Approach for Reconstructing Reynolds Stress Modeling Discrepancies Based on DNS Data
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Evidential Deep Learning to Quantify Classification Uncertainty
• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons
• Deep Generative Design: Integration of Topology Optimization and Generative Models
• Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (2001)
• Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification
• Digital twin, physics-based model, and machine learning applied to damage detection in structures
• Artificial Intelligence and Statistics
• Predictive Uncertainty Estimation via Prior Networks
• Deep Ensembles: A Loss Landscape Perspective
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• SINDy-PI: A Robust Algorithm for Parallel Implicit Sparse Identification of Nonlinear Dynamics
• Sparse Polynomial Chaos Expansions: Literature Survey and Benchmark
• Microstructural Materials Design via Deep Adversarial Learning Methodology
• A Survey of Uncertainty in Deep Neural Networks
• Optimization of photonic crystal nanocavities based on deep learning
• Compressive Sampling of Polynomial Chaos Expansions: Convergence Analysis and Sampling Strategies
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Sparse Pseudospectral Approximation Method
• The reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort study
• Intelligent Bearing Fault Diagnosis Method Combining Mixed Input and Hybrid CNN-MLP model
• Understanding Measures of Uncertainty for Adversarial Example Detection
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Data-driven polynomial chaos expansion for machine learning regression
• Adaptive multi-fidelity polynomial chaos approach to Bayesian inference in inverse problems
• Practical Deep Learning with Bayesian Principles
• A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture
• Wireframe-Based UI Design Search Through Image Autoencoder
• Physics-Informed and Hybrid Machine Learning in Additive Manufacturing: Application to Fused Filament Fabrication
• Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms
• On some neural network architectures that can represent viscosity solutions of certain high dimensional Hamilton--Jacobi partial differential equations
• Compressive sensing adaptation for polynomial chaos expansions
• The Case for Bayesian Deep Learning
• On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Non-Stationary Spectral Kernels
• Probabilistic learning on manifolds constrained by nonlinear partial differential equations for small datasets
• Notes on the Behavior of MC Dropout
• Simulation free reliability analysis: A physics-informed deep learning based approach
• Dynaformer: A Deep Learning Model for Ageing-aware Battery Discharge Prediction
• Multifidelity Data Fusion via Gradient-Enhanced Gaussian Process Regression
• A stochastic Stein Variational Newton method
• Towards Goal, Feasibility, and Diversity-Oriented Deep Generative Models in Design

Cited by in corpus (6)

• Physics-Informed Machine Learning for Battery Degradation Diagnostics: A Comparison of State-of-the-Art Methods
• Graph Evidential Learning for Anomaly Detection
• A Priori Uncertainty Quantification of Reacting Turbulence Closure Models using Bayesian Neural Networks
• A review of simulation, measurement techniques, and development in chip thermal design
• Diffusion Model-based Parameter Estimation in Dynamic Power Systems
• Navigating Uncertainties in Machine Learning for Structural Dynamics: A Comprehensive Survey of Probabilistic and Non-Probabilistic Approaches in Forward and Inverse Problems