Artificial Neural Networks trained through Deep Reinforcement Learning discover control strategies for active flow control

Abstract

We present the first application of an Artificial Neural Network trained through a Deep Reinforcement Learning agent to perform active flow control. It is shown that, in a 2D simulation of the Karman vortex street at moderate Reynolds number (Re = 100), our Artificial Neural Network is able to learn an active control strategy from experimenting with the mass flow rates of two jets on the sides of a cylinder. By interacting with the unsteady wake, the Artificial Neural Network successfully stabilizes the vortex alley and reduces drag by about 8%. This is performed while using small mass flow rates for the actuation, on the order of 0.5% of the mass flow rate intersecting the cylinder cross section once a new pseudo-periodic shedding regime is found. This opens the way to a new class of methods for performing active flow control.

References in corpus (8)

• Deep Learning in Neural Networks: An Overview
• Continuous control with deep reinforcement learning
• TensorFlow: A system for large-scale machine learning
• Playing Atari with Deep Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Closed-loop separation control using machine learning
• Drag reduction of a car model by linear genetic programming control
• A statistical learning strategy for closed-loop control of fluid flows

Cited by in corpus (83)

• Machine Learning for Fluid Mechanics
• Machine Learning in Aerodynamic Shape Optimization
• Unsupervised deep learning for super-resolution reconstruction of turbulence
• Assessment of supervised machine learning methods for fluid flows
• Accelerating Deep Reinforcement Learning strategies of Flow Control through a multi-environment approach
• Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning
• Deep Neural Networks for Nonlinear Model Order Reduction of Unsteady Flows
• Improving aircraft performance using machine learning: a review
• Super-Resolution Analysis via Machine Learning: A Survey for Fluid Flows
• Applying deep reinforcement learning to active flow control in turbulent conditions
• Deep Learning in Deterministic Computational Mechanics
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• A review on Deep Reinforcement Learning for Fluid Mechanics
• A review on deep reinforcement learning for fluid mechanics: an update
• Reinforcement-learning-based control of confined cylinder wakes with stability analyses
• High-fidelity reconstruction of turbulent flow from spatially limited data using enhanced super-resolution generative adversarial network
• Data-driven recovery of hidden physics in reduced order modeling of fluid flows
• Experimental velocity data estimation for imperfect particle images using machine learning
• Generalization techniques of neural networks for fluid flow estimation
• Controlling Rayleigh-Bénard convection via Reinforcement Learning
• A transformer-based synthetic-inflow generator for spatially-developing turbulent boundary layers
• Deep Reinforcement Learning for Turbulence Modeling in Large Eddy Simulations
• Deep Model Predictive Control with Online Learning for Complex Physical Systems
• Control of chaotic systems by Deep Reinforcement Learning
• Cluster-based feedback control of turbulent post-stall separated flows
• Super-resolution reconstruction of turbulent flow at various Reynolds numbers based on generative adversarial networks
• Reinforcement Learning of Control Strategies for Reducing Skin Friction Drag in a Fully Developed Channel Flow
• DRLinFluids -- An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM
• A Neural Network based Shock Detection and Localization Approach for Discontinuous Galerkin Methods
• Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensing
• Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows
• Comparative analysis of machine learning methods for active flow control
• Bluff body uses deep-reinforcement-learning trained active flow control to achieve hydrodynamic stealth
• Effective control of two-dimensional Rayleigh--Bénard convection: invariant multi-agent reinforcement learning is all you need
• Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence
• Deep Reinforcement Learning for Computational Fluid Dynamics on HPC Systems
• Machine learning flow control with few sensor feedback and measurement noise
• Multiscale Proper Orthogonal Decomposition (mPOD) of TR-PIV data-- a Case Study on Stationary and Transient Cylinder Wake Flows
• Stabilization of the fluidic pinball with gradient-enriched machine learning control
• Reinforcement-learning-based actuator selection method for active flow control
• Multi-fidelity reinforcement learning framework for shape optimization
• ModalPINN: an extension of Physics-Informed Neural Networks with enforced truncated Fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors
• An adaptive and energy-maximizing control of wave energy converters using extremum-seeking approach
• Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning
• Reinforcement-learning-based control of convectively-unstable flows
• Stabilization of a multi-frequency open cavity flow with gradient-enriched machine learning control
• Stochastic optimal well control in subsurface reservoirs using reinforcement learning
• Discovering explicit Reynolds-averaged turbulence closures for turbulent separated flows through deep learning-based symbolic regression with non-linear corrections
• Toward Discretization-Consistent Closure Schemes for Large Eddy Simulation Using Reinforcement Learning
• Robust training approach of neural networks for fluid flow state estimations
• Bio-inspired variable-stiffness flaps for hybrid flow control, tuned via reinforcement learning
• Machine learning and control engineering: The model-free case
• Solving PDE-constrained Control Problems Using Operator Learning
• Model-Based Reinforcement Learning for Control of Strongly-Disturbed Unsteady Aerodynamic Flows
• Reinforcement-learning-based control of turbulent channel flows at high Reynolds numbers
• Genetically-inspired convective heat transfer enhancement in a turbulent boundary layer
• Predicting the wall-shear stress and wall pressure through convolutional neural networks
• Active flow control of a turbulent separation bubble through deep reinforcement learning
• WaterLily.jl: A differentiable and backend-agnostic Julia solver for incompressible viscous flow around dynamic bodies
• Reinforcement Learning for Active Flow Control in Experiments
• SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning
• Flow control by a hybrid use of machine learning and control theory
• Deep reinforcement learning for the management of the wall regeneration cycle in wall-bounded turbulent flows
• Self-tuning model predictive control for wake flows
• How to Control Hydrodynamic Force on Fluidic Pinball via Deep Reinforcement Learning
• An Embedded Feature Selection Framework for Control
• Stabilising viscous extensional flows using Reinforcement Learning
• Deep Reinforcement Learning in Fluid Mechanics: a promising method for both Active Flow Control and Shape Optimization
• How neural networks learn to classify chaotic time series
• Deep reinforcement learning for the control of conjugate heat transfer with application to workpiece cooling
• Optimization of Second-Order Transport Models for Transition-Continuum Flows
• Variational Optimization Based Reinforcement Learning for Infinite Dimensional Stochastic Systems
• Data-assimilated model-informed reinforcement learning
• Towards human-interpretable, automated learning of feedback control for the mixing layer
• xMLC -- A Toolkit for Machine Learning Control
• Data augmentation for the POD formulation of the parametric laminar incompressible Navier-Stokes equations
• Deep Reinforcement Learning for Active Flow Control around a Circular Cylinder Using Unsteady-mode Plasma Actuators
• Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control
• Reinforcement-learning-assisted control of four-roll mills: geometric symmetry and inertial effect
• Learning swimming escape patterns for larval fish under energy constraints
• Modelling of Flow Past Long Cylindrical Structures
• Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics
• Adaptive optimization of wave energy conversion in oscillatory wave surge converters via SPH simulation and deep reinforcement learning