Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control

Abstract

Model-based reinforcement learning (MBRL) is believed to have much higher sample efficiency compared to model-free algorithms by learning a predictive model of the environment. However, the performance of MBRL highly relies on the quality of the learned model, which is usually built in a black-box manner and may have poor predictive accuracy outside of the data distribution. The deficiencies of the learned model may prevent the policy from being fully optimized. Although some uncertainty analysis-based remedies have been proposed to alleviate this issue, model bias still poses a great challenge for MBRL. In this work, we propose to leverage the prior knowledge of underlying physics of the environment, where the governing laws are (partially) known. In particular, we developed a physics-informed MBRL framework, where governing equations and physical constraints are utilized to inform the model learning and policy search. By incorporating the prior information of the environment, the quality of the learned model can be notably improved, while the required interactions with the environment are significantly reduced, leading to better sample efficiency and learning performance. The effectiveness and merit have been demonstrated over a handful of classic control problems, where the environments are governed by canonical ordinary/partial differential equations.

arXiv comment23 figures; 2 tables

References in corpus (25)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Addressing Function Approximation Error in Actor-Critic Methods
• Dueling Network Architectures for Deep Reinforcement Learning
• Turbulence Modeling in the Age of Data
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain
• A Physics Informed Machine Learning Approach for Reconstructing Reynolds Stress Modeling Discrepancies Based on DNS Data
• Physics-informed learning of governing equations from scarce data
• Artificial Neural Networks trained through Deep Reinforcement Learning discover control strategies for active flow control
• Control of a Quadrotor with Reinforcement Learning
• Learning Continuous Control Policies by Stochastic Value Gradients
• Learning Latent Dynamics for Planning from Pixels
• Uncovering near-wall blood flow from sparse data with physics-informed neural networks
• Model-Ensemble Trust-Region Policy Optimization
• Super-resolution and denoising of fluid flow using physics-informed convolutional neural networks without high-resolution labels
• Applying deep reinforcement learning to active flow control in turbulent conditions
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• On the state space geometry of the Kuramoto-Sivashinsky flow in a periodic domain
• Controlling Rayleigh-Bénard convection via Reinforcement Learning
• Control of chaotic systems by Deep Reinforcement Learning
• Finding Efficient Swimming Strategies in a Three Dimensional Chaotic Flow by Reinforcement Learning
• Learning to Control Active Matter
• Propagation of Input Uncertainty in Presence of Model-Form Uncertainty: A Multi-fidelity Approach for CFD Applications