A Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation Assurance

Abstract

A novel deep multi-agent reinforcement learning framework is proposed to identify and resolve conflicts among a variable number of aircraft in a high-density, stochastic, and dynamic sector. Currently the sector capacity is constrained by human air traffic controller's cognitive limitation. We investigate the feasibility of a new concept (autonomous separation assurance) and a new approach to push the sector capacity above human cognitive limitation. We propose the concept of using distributed vehicle autonomy to ensure separation, instead of a centralized sector air traffic controller. Our proposed framework utilizes Proximal Policy Optimization (PPO) that we modify to incorporate an attention network. This allows the agents to have access to variable aircraft information in the sector in a scalable, efficient approach to achieve high traffic throughput under uncertainty. Agents are trained using a centralized learning, decentralized execution scheme where one neural network is learned and shared by all agents. The proposed framework is validated on three challenging case studies in the BlueSky air traffic control environment. Numerical results show the proposed framework significantly reduces offline training time, increases performance, and results in a more efficient policy.

26 pages

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• StarCraft II: A New Challenge for Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Emergent Tool Use From Multi-Agent Autocurricula
• An Autonomous Free Airspace En-route Controller using Deep Reinforcement Learning Techniques

Cited by in corpus (4)

• A Survey on Reinforcement Learning in Aviation Applications
• Separation Assurance in Urban Air Mobility Systems using Shared Scheduling Protocols
• A Reinforcement Learning Approach to Quiet and Safe UAM Traffic Management
• Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance