Multi-agent Reinforcement Learning Accelerated MCMC on Multiscale Inversion Problem

Abstract

In this work, we propose a multi-agent actor-critic reinforcement learning (RL) algorithm to accelerate the multi-level Monte Carlo Markov Chain (MCMC) sampling algorithms. The policies (actors) of the agents are used to generate the proposal in the MCMC steps; and the critic, which is centralized, is in charge of estimating the long term reward. We verify our proposed algorithm by solving an inverse problem with multiple scales. There are several difficulties in the implementation of this problem by using traditional MCMC sampling. Firstly, the computation of the posterior distribution involves evaluating the forward solver, which is very time consuming for a problem with heterogeneous. We hence propose to use the multi-level algorithm. More precisely, we use the generalized multiscale finite element method (GMsFEM) as the forward solver in evaluating a posterior distribution in the multi-level rejection procedure. Secondly, it is hard to find a function which can generate samplings which are meaningful. To solve this issue, we learn an RL policy as the proposal generator. Our experiments show that the proposed method significantly improves the sampling process

References in corpus (10)

• Continuous control with deep reinforcement learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Soft Actor-Critic Algorithms and Applications
• Dueling Network Architectures for Deep Reinforcement Learning
• Adaptive multiscale model reduction with Generalized Multiscale Finite Element Methods
• Constraint Energy Minimizing Generalized Multiscale Finite Element Method
• Non-local Multi-continua Upscaling for Flows in Heterogeneous Fractured Media
• Improving Simulation Efficiency of MCMC for Inverse Modeling of Hydrologic Systems with a Kalman-Inspired Proposal Distribution
• Computational multiscale methods for quasi-gas dynamic equations

Cited by in corpus (7)

• HEI: hybrid explicit-implicit learning for multiscale problems
• Accelerated replica exchange stochastic gradient Langevin diffusion enhanced Bayesian DeepONet for solving noisy parametric PDEs
• Hybrid explicit-implicit learning for multiscale problems with time dependent source
• Theoretical and numerical studies of inverse source problem for the linear parabolic equation with sparse boundary measurements
• Probabilistic Compute-in-Memory Design For Efficient Markov Chain Monte Carlo Sampling
• Multi-variance replica exchange stochastic gradient MCMC for inverse and forward Bayesian physics-informed neural network
• A deep neural network approach on solving the linear transport model under diffusive scaling