Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data

Abstract

Surrogate modeling and uncertainty quantification tasks for PDE systems are most often considered as supervised learning problems where input and output data pairs are used for training. The construction of such emulators is by definition a small data problem which poses challenges to deep learning approaches that have been developed to operate in the big data regime. Even in cases where such models have been shown to have good predictive capability in high dimensions, they fail to address constraints in the data implied by the PDE model. This paper provides a methodology that incorporates the governing equations of the physical model in the loss/likelihood functions. The resulting physics-constrained, deep learning models are trained without any labeled data (e.g. employing only input data) and provide comparable predictive responses with data-driven models while obeying the constraints of the problem at hand. This work employs a convolutional encoder-decoder neural network approach as well as a conditional flow-based generative model for the solution of PDEs, surrogate model construction, and uncertainty quantification tasks. The methodology is posed as a minimization problem of the reverse Kullback-Leibler (KL) divergence between the model predictive density and the reference conditional density, where the later is defined as the Boltzmann-Gibbs distribution at a given inverse temperature with the underlying potential relating to the PDE system of interest. The generalization capability of these models to out-of-distribution input is considered. Quantification and interpretation of the predictive uncertainty is provided for a number of problems.

51 pages, 18 figures, submitted to Journal of Computational Physics

References in corpus (47)

• Conditional Generative Adversarial Nets
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• DGM: A deep learning algorithm for solving partial differential equations
• Conditional Image Generation with PixelCNN Decoders
• Solving high-dimensional partial differential equations using deep learning
• On Calibration of Modern Neural Networks
• Deep learning for universal linear embeddings of nonlinear dynamics
• Variational Inference with Normalizing Flows
• Toward Multimodal Image-to-Image Translation
• Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations
• Deep Image Prior
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Density estimation using Real NVP
• A unified deep artificial neural network approach to partial differential equations in complex geometries
• Solving ill-posed inverse problems using iterative deep neural networks
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification
• Variational Dropout and the Local Reparameterization Trick
• Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• Training-image based geostatistical inversion using a spatial generative adversarial neural network
• Deep convolutional encoder-decoder networks for uncertainty quantification of dynamic multiphase flow in heterogeneous media
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Accelerating Eulerian Fluid Simulation With Convolutional Networks
• Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
• PDE-Net: Learning PDEs from Data
• A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• Neural Network Renormalization Group
• Probabilistic Numerics and Uncertainty in Computations
• Quantifying model form uncertainty in Reynolds-averaged turbulence models with Bayesian deep neural networks
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Diversity-Sensitive Conditional Generative Adversarial Networks
• A Deep Neural Network Surrogate for High-Dimensional Random Partial Differential Equations
• Spherical CNNs on Unstructured Grids
• Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations
• Convolutional Gaussian Processes
• Do Deep Generative Models Know What They Don't Know?
• Structured Bayesian Gaussian process latent variable model: applications to data-driven dimensionality reduction and high-dimensional inversion
• Computing stationary solutions of the two-dimensional Gross-Pitaevskii equation with Deflated continuation
• Bayesian model and dimension reduction for uncertainty propagation: applications in random media
• A multiscale neural network based on hierarchical matrices
• Deep Multiscale Model Learning
• Probabilistic Numerical Methods for Partial Differential Equations and Bayesian Inverse Problems

Cited by in corpus (190)

• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• DeepXDE: A deep learning library for solving differential equations
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain
• Physics-Informed Multi-LSTM Networks for Metamodeling of Nonlinear Structures
• Deep learning in nano-photonics: inverse design and beyond
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks
• Efficient training of physics-informed neural networks via importance sampling
• Physics-informed deep learning for incompressible laminar flows
• Physics-Informed Neural Network for Modelling the Thermochemical Curing Process of Composite-Tool Systems During Manufacture
• Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems
• PhyCRNet: Physics-informed Convolutional-Recurrent Network for Solving Spatiotemporal PDEs
• Transfer learning based multi-fidelity physics informed deep neural network
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks
• A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Super-resolution and denoising of fluid flow using physics-informed convolutional neural networks without high-resolution labels
• Deep-learning-based surrogate flow modeling and geological parameterization for data assimilation in 3D subsurface flow
• Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM
• Simulator-free Solution of High-Dimensional Stochastic Elliptic Partial Differential Equations using Deep Neural Networks
• Deep Learning in Deterministic Computational Mechanics
• Physics-informed PointNet: A deep learning solver for steady-state incompressible flows and thermal fields on multiple sets of irregular geometries
• Physics-Informed Neural Operator for Learning Partial Differential Equations
• Integration of adversarial autoencoders with residual dense convolutional networks for estimation of non-Gaussian hydraulic conductivities
• A Review of Machine Learning Methods Applied to Structural Dynamics and Vibroacoustic
• Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability
• On closures for reduced order models $-$ A spectrum of first-principle to machine-learned avenues
• Physics-Informed Neural Nets for Control of Dynamical Systems
• Physics guided machine learning using simplified theories
• D3M: A deep domain decomposition method for partial differential equations
• Multiphase flow prediction with deep neural networks
• When and why PINNs fail to train: A neural tangent kernel perspective
• Characterizing possible failure modes in physics-informed neural networks
• Deep learning of free boundary and Stefan problems
• A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture
• A mesh-free method for interface problems using the deep learning approach
• CCSNet: a deep learning modeling suite for CO${}_2$ storage
• Temperature Field Inversion of Heat-Source Systems via Physics-Informed Neural Networks
• Multilevel domain decomposition-based architectures for physics-informed neural networks
• Physics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-fidelity Data Fusion
• An overview on deep learning-based approximation methods for partial differential equations
• Efficient Uncertainty Quantification for Dynamic Subsurface Flow with Surrogate by Theory-guided Neural Network
• Solving inverse problems using conditional invertible neural networks
• A novel meta-learning initialization method for physics-informed neural networks
• A deep learning framework for solution and discovery in solid mechanics
• Shape-constrained Symbolic Regression -- Improving Extrapolation with Prior Knowledge
• Theory-guided Auto-Encoder for Surrogate Construction and Inverse Modeling
• Physics-informed neural networks (PINNs) for fluid mechanics: A review
• Weak Form Theory-guided Neural Network (TgNN-wf) for Deep Learning of Subsurface Single and Two-phase Flow
• Enforcing Dirichlet boundary conditions in physics-informed neural networks and variational physics-informed neural networks
• Surrogate and inverse modeling for two-phase flow in porous media via theory-guided convolutional neural network
• A Derivative-Free Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks
• Efficient Deep Learning Techniques for Multiphase Flow Simulation in Heterogeneous Porous Media
• Learning physics-constrained subgrid-scale closures in the small-data regime for stable and accurate LES
• Three-dimensional deep learning-based reduced order model for unsteady flow dynamics with variable Reynolds number
• Teaching the Incompressible Navier-Stokes Equations to Fast Neural Surrogate Models in 3D
• Neural reparameterization improves structural optimization
• Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks
• Bayesian Physics-Informed Extreme Learning Machine for Forward and Inverse PDE Problems with Noisy Data
• Adaptive deep density approximation for Fokker-Planck equations
• Multi-Output Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Uncertainties
• On the use of graph neural networks and shape-function-based gradient computation in the deep energy method
• Finite element interpolated neural networks for solving forward and inverse problems
• A Helmholtz equation solver using unsupervised learning: Application to transcranial ultrasound
• Deep neural networks for the prediction of the optical properties and the free-form inverse design of metamaterials
• A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
• Variational training of neural network approximations of solution maps for physical models
• GINNs: Graph-Informed Neural Networks for Multiscale Physics
• Coercing Machine Learning to Output Physically Accurate Results
• Using neural networks to accelerate the solution of the Boltzmann equation
• On the Use of Neural Networks for Full Waveform Inversion
• Neural modal ordinary differential equations: Integrating physics-based modeling with neural ordinary differential equations for modeling high-dimensional monitored structures
• Physics-driven Learning of the Steady Navier-Stokes Equations using Deep Convolutional Neural Networks
• RBF-MGN:Solving spatiotemporal PDEs with Physics-informed Graph Neural Network
• Efficient nonlinear manifold reduced order model
• Designing Accurate Emulators for Scientific Processes using Calibration-Driven Deep Models
• Fully probabilistic deep models for forward and inverse problems in parametric PDEs
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• VarNet: Variational Neural Networks for the Solution of Partial Differential Equations
• A non-cooperative meta-modeling game for automated third-party calibrating, validating, and falsifying constitutive laws with parallelized adversarial attacks
• Physics-constrained Deep Learning for Robust Inverse ECG Modeling
• Adaptive physics-informed neural operator for coarse-grained non-equilibrium flows
• Physics-integrated hybrid framework for model form error identification in nonlinear dynamical systems
• A probabilistic generative model for semi-supervised training of coarse-grained surrogates and enforcing physical constraints through virtual observables
• Bayesian learning of orthogonal embeddings for multi-fidelity Gaussian Processes
• Solving time dependent Fokker-Planck equations via temporal normalizing flow
• Simulation free reliability analysis: A physics-informed deep learning based approach
• A Deep Learning Approach for the solution of Probability Density Evolution of Stochastic Systems
• Sparse Polynomial Chaos expansions using Variational Relevance Vector Machines
• A hybrid data driven-physics constrained Gaussian process regression framework with deep kernel for uncertainty quantification
• Dynamically orthogonal tensor methods for high-dimensional nonlinear PDEs
• MillimeterDL: Deep Learning Simulations of the Microwave Sky
• Physics Informed Deep Learning for Transport in Porous Media. Buckley Leverett Problem
• Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEs
• A physics and data co-driven surrogate modeling method for high-dimensional rare event simulation
• Dynamic tensor approximation of high-dimensional nonlinear PDEs
• PCNN: A physics-constrained neural network for multiphase flows
• Joint Deep Reversible Regression Model and Physics-Informed Unsupervised Learning for Temperature Field Reconstruction
• A Combined Data-driven and Physics-driven Method for Steady Heat Conduction Prediction using Deep Convolutional Neural Networks
• Encoding Invariances in Deep Generative Models
• Bi-fidelity Variational Auto-encoder for Uncertainty Quantification
• Making Invisible Visible: Data-Driven Seismic Inversion with Spatio-temporally Constrained Data Augmentation
• Physics-informed Information Field Theory for Modeling Physical Systems with Uncertainty Quantification
• Unveiling the frontiers of deep learning: innovations shaping diverse domains
• Interpretable AI forecasting for numerical relativity waveforms of quasi-circular, spinning, non-precessing binary black hole mergers
• Bayesian neural networks for weak solution of PDEs with uncertainty quantification
• Mutual Information for Explainable Deep Learning of Multiscale Systems
• opPINN: Physics-Informed Neural Network with operator learning to approximate solutions to the Fokker-Planck-Landau equation
• Solving PDE-constrained Control Problems Using Operator Learning
• Deep recurrent-convolutional neural network learning and physics Kalman filtering comparison in dynamic load identification
• Physics-Constrained Bayesian Neural Network for Fluid Flow Reconstruction with Sparse and Noisy Data
• Improved Surrogate Modeling of Fluid Dynamics with Physics-Informed Neural Networks
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• Hard Encoding of Physics for Learning Spatiotemporal Dynamics
• Curriculum learning for data-driven modeling of dynamical systems
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically
• Modelling parametric uncertainty in PDEs models via Physics-Informed Neural Networks
• Deep-learning-based reduced-order modeling for subsurface flow simulation
• Stability analysis of hierarchical tensor methods for time-dependent PDEs
• Deep Learning of Dynamic Subsurface Flow via Theory-guided Generative Adversarial Network
• Solution of FPK Equation for Stochastic Dynamics Subjected to Additive Gaussian Noise via Deep Learning Approach
• Can physical information aid the generalization ability of Neural Networks for hydraulic modeling?
• Physics-Aware Neural Implicit Solvers for multiscale, parametric PDEs with applications in heterogeneous media
• A Tailored Convolutional Neural Network for Nonlinear Manifold Learning of Computational Physics Data using Unstructured Spatial Discretizations
• Learning the random variables in Monte Carlo simulations with stochastic gradient descent: Machine learning for parametric PDEs and financial derivative pricing
• Physics-informed Convolutional Neural Networks for Temperature Field Prediction of Heat Source Layout without Labeled Data
• Learning on dynamic statistical manifolds
• Meshfree Variational Physics Informed Neural Networks (MF-VPINN): an adaptive training strategy
• Diffusion-Based Surrogate Modeling and Multi-Fidelity Calibration
• Deep adaptive sampling for surrogate modeling without labeled data
• Machine learning-based multiscale constitutive modelling: Development and application to dual-porosity mass transfer
• Bayesian Sparse learning with preconditioned stochastic gradient MCMC and its applications
• Splitting physics-informed neural networks for inferring the dynamics of integer- and fractional-order neuron models
• Physics-informed Deep Learning to Solve Three-dimensional Terzaghi Consolidation Equation: Forward and Inverse Problems
• Identification of Forced Oscillation Sources in Wind Farms using E-SINDy
• Learning Differential Operators for Interpretable Time Series Modeling
• Power Module Heat Sink Design Optimization with Ensembles of Data-Driven Polynomial Chaos Surrogate Models
• A deep surrogate approach to efficient Bayesian inversion in PDE and integral equation models
• Physics Encoded Blocks in Residual Neural Network Architectures for Digital Twin Models
• Physics-informed semantic inpainting: Application to geostatistical modeling
• Learning Incompressible Fluid Dynamics from Scratch -- Towards Fast, Differentiable Fluid Models that Generalize
• Adaptive Sparse Polynomial Chaos Expansions via Leja Interpolation
• PCE-PINNs: Physics-Informed Neural Networks for Uncertainty Propagation in Ocean Modeling
• Physics Informed Deep Learning for Flow and Transport in Porous Media
• CDNNs: The coupled deep neural networks for coupling of the Stokes and Darcy-Forchheimer problems
• Solving Elliptic Equations with Brownian Motion: Bias Reduction and Temporal Difference Learning
• A Bayesian Multiscale Deep Learning Framework for Flows in Random Media
• Inexact Sequential Quadratic Optimization for Minimizing a Stochastic Objective Function Subject to Deterministic Nonlinear Equality Constraints
• Efficient PINNs via Multi-Head Unimodular Regularization of the Solutions Space
• Point-DeepONet: Predicting Nonlinear Fields on Non-Parametric Geometries under Variable Load Conditions
• Physics-Guided Discovery of Highly Nonlinear Parametric Partial Differential Equations
• Learning Composable Energy Surrogates for PDE Order Reduction
• Physics-Informed Neural Networks for Solving Multiscale Mode-Resolved Phonon Boltzmann Transport Equation
• Label-free timing analysis of SiPM-based modularized detectors with physics-constrained deep learning
• Variational Autoencoding of PDE Inverse Problems
• Adaptive Physics-Informed Neural Networks for Markov-Chain Monte Carlo
• Physics-enhanced Neural Networks in the Small Data Regime
• Physics-aware, probabilistic model order reduction with guaranteed stability
• A discrete physics-informed training for projection-based reduced order models with neural networks
• A Deep Learning Density Shaping Model Predictive Gust Load Alleviation Control of a Compliant Wing Subjected to Atmospheric Turbulence
• Inverse Aerodynamic Design of Gas Turbine Blades using Probabilistic Machine Learning
• Exploring Generative Physics Models with Scientific Priors in Inertial Confinement Fusion
• Spatiotemporal wall pressure forecast of a rectangular cylinder with physics-aware DeepU-Fourier neural network
• A universal deep learning strategy for designing high-quality-factor photonic resonances
• Deep learning of parameterized equations with applications to uncertainty quantification
• Application of mixed-variable physics-informed neural networks to solve normalised momentum and energy transport equations for 2D internal convective flow
• Improved Surrogates in Inertial Confinement Fusion with Manifold and Cycle Consistencies
• Reconstructing High-resolution Turbulent Flows Using Physics-Guided Neural Networks
• Physics-Informed Machine Learning Models for Predicting the Progress of Reactive-Mixing
• Probabilistic partition of unity networks for high-dimensional regression problems
• A Statistician Teaches Deep Learning
• Solving the linear transport equation by a deep neural network approach
• Clustered active-subspace based local Gaussian Process emulator for high-dimensional and complex computer models
• Physics-informed neural networks for non-Newtonian fluid thermo-mechanical problems: an application to rubber calendering process
• High Throughput Training of Deep Surrogates from Large Ensemble Runs
• A survey of unsupervised learning methods for high-dimensional uncertainty quantification in black-box-type problems
• Uncertainty quantification for ptychography using normalizing flows
• Variational autoencoder for generating realistic $N$-body simulations for dark matter halos
• Solving time-dependent parametric PDEs by multiclass classification-based reduced order model
• Recurrent convolutional neural network for the surrogate modeling of subsurface flow simulation
• MFPC-Net: Multi-fidelity Physics-Constrained Neural Process
• Data-informed Emulators for Multi-Physics Simulations
• AutoSurrogate: An LLM-Driven Multi-Agent Framework for Autonomous Construction of Deep Learning Surrogate Models in Subsurface Flow
• An evolve-then-correct reduced order model for hidden fluid dynamics