Data-assimilated model-informed reinforcement learning

Abstract

The control of spatio-temporally chaos is challenging because of high dimensionality and unpredictability. Model-free reinforcement learning (RL) discovers optimal control policies by interacting with the system, typically requiring observations of the full physical state. In practice, sensors often provide only partial and noisy measurements (observations) of the system. The objective of this paper is to develop a framework that enables the control of chaotic systems with partial and noisy observability. The proposed method, data-assimilated model-informed reinforcement learning (DA-MIRL), integrates (i) low-order models to approximate high-dimensional dynamics; (ii) sequential data assimilation to correct the model prediction when observations become available; and (iii) an off-policy actor-critic RL algorithm to adaptively learn an optimal control strategy based on the corrected state estimates. We test DA-MIRL on the spatiotemporally chaotic solutions of the Kuramoto-Sivashinsky equation. We estimate the full state of the environment with (i) a physics-based model, here, a coarse-grained model; and (ii) a data-driven model, here, the control-aware echo state network, which is proposed in this paper. We show that DA-MIRL successfully estimates and suppresses the chaotic dynamics of the environment in real time from partial observations and approximate models. This work opens opportunities for the control of partially observable chaotic systems.

References in corpus (10)

• Artificial Neural Networks trained through Deep Reinforcement Learning discover control strategies for active flow control
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• On the state space geometry of the Kuramoto-Sivashinsky flow in a periodic domain
• Control of chaotic systems by Deep Reinforcement Learning
• Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensing
• A Priori Estimation Of Memory Effects In Coarse-Grained Nonlinear Systems Using The Mori-Zwanzig Formalism
• Inferring unknown unknowns: Regularized bias-aware ensemble Kalman filter
• Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning
• Reconstruction, forecasting, and stability of chaotic dynamics from partial data
• Online model learning with data-assimilated reservoir computers