Zermelo's problem: Optimal point-to-point navigation in 2D turbulent flows using Reinforcement Learning

Abstract

To find the path that minimizes the time to navigate between two given points in a fluid flow is known as Zermelo's problem. Here, we investigate it by using a Reinforcement Learning (RL) approach for the case of a vessel which has a slip velocity with fixed intensity, Vs , but variable direction and navigating in a 2D turbulent sea. We show that an Actor-Critic RL algorithm is able to find quasi-optimal solutions for both time-independent and chaotically evolving flow configurations. For the frozen case, we also compared the results with strategies obtained analytically from continuous Optimal Navigation (ON) protocols. We show that for our application, ON solutions are unstable for the typical duration of the navigation process, and are therefore not useful in practice. On the other hand, RL solutions are much more robust with respect to small changes in the initial conditions and to external noise, even when V s is much smaller than the maximum flow velocity. Furthermore, we show how the RL approach is able to take advantage of the flow properties in order to reach the target, especially when the steering speed is small.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• A Brief Survey of Deep Reinforcement Learning
• Active Particles in Complex and Crowded Environments
• Machine Learning for Fluid Mechanics
• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
• Cascades and transitions in turbulent flows
• Attractor Reconstruction by Machine Learning
• Measurement of Lagrangian velocity in fully developed turbulence
• Heavy particle concentration in turbulence at dissipative and inertial scales
• Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks
• Flow Navigation by Smart Microswimmers via Reinforcement Learning
• Learning to school in the presence of hydrodynamic interactions
• Probing the spatiotemporal dynamics of catalytic Janus particles with single-particle tracking and differential dynamic microscopy
• Measurement of Anomalous Diffusion Using Recurrent Neural Networks
• Optimal steering of a smart active particle
• Pulling and Pushing a Cargo With a Catalytically Active Carrier
• Finding Efficient Swimming Strategies in a Three Dimensional Chaotic Flow by Reinforcement Learning
• Synchronization to big-data: nudging the Navier-Stokes equations for data assimilation of turbulent flows
• Inferring flow parameters and turbulent configuration with physics-informed data-assimilation and spectral nudging
• Compressed Convolutional LSTM: An Efficient Deep Learning framework to Model High Fidelity 3D Turbulence
• Smart Inertial Particles
• Where do small, weakly inertial particles go in a turbulent flow?
• From Deep to Physics-Informed Learning of Turbulence: Diagnostics
• Reinforcement Learning of Artificial Microswimmers
• Transport in finite size systems: an exit time approach
• Optimal Control Strategies for Active Particle Navigation

Cited by in corpus (39)

• Scientific multi-agent reinforcement learning for wall-models of turbulent flows
• Learning Efficient Navigation in Vortical Flow Fields
• Machine learning strategies for path-planning microswimmers in turbulent flows
• Controlling Rayleigh-Bénard convection via Reinforcement Learning
• Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows
• Hydrodynamics can Determine the Optimal Route for Microswimmer Navigation
• Reinforcement learning of optimal active particle navigation
• Navigation of micro-swimmers in steady flow: the importance of symmetries
• Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number
• Optimal active particle navigation meets machine learning
• Optimal navigation of microswimmers in complex and noisy environments
• Finite-Horizon, Energy-Optimal Trajectories in Unsteady Flows
• Optimal navigation strategies for microswimmers on curved manifolds
• Optimal tracking strategies in a turbulent flow
• Flow reconstruction by multiresolution optimization of a discrete loss with automatic differentiation
• Long-distance migration with minimal energy consumption in a thermal turbulent environment
• Energetic cost of microswimmer navigation: the role of body shape
• Learning how to find targets in the micro-world: The case of intermittent active Brownian particles
• Taming Lagrangian Chaos with Multi-Objective Reinforcement Learning
• Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models
• Real-time optimal control of high-dimensional parametrized systems by deep learning-based reduced order models
• SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning
• Optimal Navigation in Microfluidics via the Optimization of a Discrete Loss
• Learning to swim efficiently in a nonuniform flow field
• Short term vs. long term: optimization of microswimmer navigation on different time horizons
• Deep reinforcement learning for the management of the wall regeneration cycle in wall-bounded turbulent flows
• Migration of self-propelling agent in a turbulent environment with minimal energy consumption
• Efficient survival strategy for zooplankton in turbulence
• Adaptive active Brownian particles searching for targets of unknown positions
• Proportional integral derivative controller assisted reinforcement learning for path following by autonomous underwater vehicles
• Super-resolution reconstruction of turbulent flows from a single Lagrangian trajectory
• A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows
• An account on links between Finsler and Lorentz Geometries for Riemannian Geometers
• Reinforcement Learning reveals fundamental limits on the mixing of active particles
• Active gyrotactic stability of microswimmers using hydromechanical signals
• Run-and-Tumble Particles Learning Chemotaxis
• A study of first-passage time minimization via Q-learning in heated gridworlds
• Optimal navigation in two-dimensional flows: Control theory and reinforcement learning
• Learning to traverse convective flows at moderate to high Rayleigh numbers