Multi-variance replica exchange stochastic gradient MCMC for inverse and forward Bayesian physics-informed neural network

Abstract

Physics-informed neural network (PINN) has been successfully applied in solving a variety of nonlinear non-convex forward and inverse problems. However, the training is challenging because of the non-convex loss functions and the multiple optima in the Bayesian inverse problem. In this work, we propose a multi-variance replica exchange stochastic gradient Langevin diffusion method to tackle the challenge of the multiple local optima in the optimization and the challenge of the multiple modal posterior distribution in the inverse problem. Replica exchange methods are capable of escaping from the local traps and accelerating the convergence. However, it may not be efficient to solve mathematical inversion problems by using the vanilla replica method directly since the method doubles the computational cost in evaluating the forward solvers (likelihood functions) in the two chains. To address this issue, we propose to make different assumptions on the energy function estimation and this facilities one to use solvers of different fidelities in the likelihood function evaluation. We give an unbiased estimate of the swapping rate and give an estimation of the discretization error of the scheme. To verify our idea, we design and solve four inverse problems which have multiple modes. The proposed method is also employed to train the Bayesian PINN to solve the forward and inverse problems; faster and more accurate convergence has been observed when compared to the stochastic gradient Langevin diffusion (SGLD) method and vanila replica exchange methods.

References in corpus (15)

• B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Computational multiscale methods for quasi-gas dynamic equations
• Non-convex Learning via Replica Exchange Stochastic Gradient MCMC
• Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization
• Accelerating Nonconvex Learning via Replica Exchange Langevin Diffusion
• Non-Asymptotic Analysis of Fractional Langevin Monte Carlo for Non-Convex Optimization
• Multi-agent Reinforcement Learning Accelerated MCMC on Multiscale Inversion Problem
• Computational multiscale methods for parabolic wave approximations in heterogeneous media
• Bayesian Sparse learning with preconditioned stochastic gradient MCMC and its applications
• An adaptive Hessian approximated stochastic gradient MCMC method
• A deep neural network approach on solving the linear transport model under diffusive scaling

Cited by in corpus (3)

• Accelerated replica exchange stochastic gradient Langevin diffusion enhanced Bayesian DeepONet for solving noisy parametric PDEs
• Hybrid explicit-implicit learning for multiscale problems with time dependent source
• Theoretical and numerical studies of inverse source problem for the linear parabolic equation with sparse boundary measurements