Optimal navigation in two-dimensional flows: Control theory and reinforcement learning

Abstract

Zermelo's navigation problem seeks the trajectory of minimal travel time between two points in a fluid flow. We address this problem for an agent -- such as a floating drone or active particle -- that is advected by a two-dimensional flow, self-propels at a fixed speed smaller than or comparable to the characteristic flow velocity, and can steer its direction. The flows considered span increasing levels of complexity, from steady solid-body rotation and time-dependent sink-vortex to the Taylor-Green flow and turbulence in the inverse energy cascade regime. Although optimal-control theory provides time-minimizing trajectories, these solutions become unstable in chaotic regimes characterized by positive finite-time Lyapunov exponents. To design robust navigation strategies, we apply reinforcement learning and compare Q-learning with a one-step actor-critic algorithm. Both methods achieve successful navigation, yielding mean travel times within 3-10% of optimal-control solutions in regular flows, while the discrepancy increases to 35-75% in time-dependent turbulent flows. Finally, we show that agents trained on coarse-grained turbulent flows generalize to the full velocity field. This robustness to incomplete flow information is essential for practical navigation in real-world oceanic and atmospheric environments.

References in corpus (17)

• Flow Navigation by Smart Microswimmers via Reinforcement Learning
• Super-Resolution Analysis via Machine Learning: A Survey for Fluid Flows
• Zermelo's problem: Optimal point-to-point navigation in 2D turbulent flows using Reinforcement Learning
• Machine learning strategies for path-planning microswimmers in turbulent flows
• Inverse energy cascade and emergence of large coherent vortices in turbulence driven by Faraday waves
• Optimal steering of a smart active particle
• Finding Efficient Swimming Strategies in a Three Dimensional Chaotic Flow by Reinforcement Learning
• Three-dimensional fluid motion in Faraday waves: creation of vorticity and generation of two-dimensional turbulence
• Reinforcement learning of optimal active particle navigation
• Surfing on turbulence: a strategy for planktonic navigation
• Optimal active particle navigation meets machine learning
• Optimal navigation of microswimmers in complex and noisy environments
• Optimal navigation strategies for microswimmers on curved manifolds
• Formation and decay of eddy currents generated by crossed surface waves
• Optimal tracking strategies in a turbulent flow
• Inferring parameters and reconstruction of two-dimensional turbulent flows with physics-informed neural networks
• A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows