Accelerating Deep Reinforcement Learning strategies of Flow Control through a multi-environment approach

Abstract

Deep Reinforcement Learning (DRL) has recently been proposed as a methodology to discover complex Active Flow Control (AFC) strategies [Rabault, J., Kuchta, M., Jensen, A., Reglade, U., & Cerardi, N. (2019): "Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control", Journal of Fluid Mechanics, 865, 281-302]. However, while promising results were obtained on a simple 2D benchmark flow at a moderate Reynolds number, considerable speedups will be required to investigate more challenging flow configurations. In the case of DRL trained with Computational Fluid Dynamics (CFD) data, it was found that the CFD part, rather than training the Artificial Neural Network, was the limiting factor for speed of execution. Therefore, speedups should be obtained through a combination of two approaches. The first one, which is well documented in the literature, is to parallelize the numerical simulation itself. The second one is to adapt the DRL algorithm for parallelization. Here, a simple strategy is to use several independent simulations running in parallel to collect experiences faster. In the present work, we discuss this solution for parallelization. We illustrate that perfect speedups can be obtained up to the batch size of the DRL agent, and slightly suboptimal scaling still takes place for an even larger number of simulations. This is, therefore, an important step towards enabling the study of more sophisticated Fluid Mechanics problems through DRL.

References in corpus (9)

• Artificial Neural Networks trained through Deep Reinforcement Learning discover control strategies for active flow control
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Closed-loop separation control using machine learning
• Control of chaotic systems by Deep Reinforcement Learning
• Jet mixing optimization using machine learning control
• Optimal control of circular cylinder wakes using long control horizons
• Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
• Deep Reinforcement Learning achieves flow control of the 2D Karman Vortex Street

Cited by in corpus (31)

• Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning
• Applying deep reinforcement learning to active flow control in turbulent conditions
• Deep Learning in Deterministic Computational Mechanics
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• A review on Deep Reinforcement Learning for Fluid Mechanics
• A review on deep reinforcement learning for fluid mechanics: an update
• Reinforcement-learning-based control of confined cylinder wakes with stability analyses
• Experimental velocity data estimation for imperfect particle images using machine learning
• Controlling Rayleigh-Bénard convection via Reinforcement Learning
• Deep Reinforcement Learning for Turbulence Modeling in Large Eddy Simulations
• Reinforcement Learning of Control Strategies for Reducing Skin Friction Drag in a Fully Developed Channel Flow
• DRLinFluids -- An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM
• Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensing
• Comparative analysis of machine learning methods for active flow control
• Bluff body uses deep-reinforcement-learning trained active flow control to achieve hydrodynamic stealth
• Effective control of two-dimensional Rayleigh--Bénard convection: invariant multi-agent reinforcement learning is all you need
• Deep Reinforcement Learning for Computational Fluid Dynamics on HPC Systems
• Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence
• Machine learning flow control with few sensor feedback and measurement noise
• Aero-Nef: Neural Fields for Rapid Aircraft Aerodynamics Simulations
• Reinforcement-learning-based actuator selection method for active flow control
• Multi-fidelity reinforcement learning framework for shape optimization
• An adaptive and energy-maximizing control of wave energy converters using extremum-seeking approach
• Reinforcement-learning-based control of convectively-unstable flows
• Discovering explicit Reynolds-averaged turbulence closures for turbulent separated flows through deep learning-based symbolic regression with non-linear corrections
• Toward Discretization-Consistent Closure Schemes for Large Eddy Simulation Using Reinforcement Learning
• Prediction of wall-bounded turbulence from wall quantities using convolutional neural networks
• Bio-inspired variable-stiffness flaps for hybrid flow control, tuned via reinforcement learning
• Deep reinforcement learning for the management of the wall regeneration cycle in wall-bounded turbulent flows
• Investigation of reinforcement learning for shape optimization of profile extrusion dies
• An Embedded Feature Selection Framework for Control