Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning

Abstract

A central question in robotics is how to design a control system for an agile mobile robot. This paper studies this question systematically, focusing on a challenging setting: autonomous drone racing. We show that a neural network controller trained with reinforcement learning (RL) outperformed optimal control (OC) methods in this setting. We then investigated which fundamental factors have contributed to the success of RL or have limited OC. Our study indicates that the fundamental advantage of RL over OC is not that it optimizes its objective better but that it optimizes a better objective. OC decomposes the problem into planning and control with an explicit intermediate representation, such as a trajectory, that serves as an interface. This decomposition limits the range of behaviors that can be expressed by the controller, leading to inferior control performance when facing unmodeled effects. In contrast, RL can directly optimize a task-level objective and can leverage domain randomization to cope with model uncertainty, allowing the discovery of more robust control responses. Our findings allowed us to push an agile drone to its maximum performance, achieving a peak acceleration greater than 12 times the gravitational acceleration and a peak velocity of 108 kilometers per hour. Our policy achieved superhuman control within minutes of training on a standard workstation. This work presents a milestone in agile robotics and sheds light on the role of RL and OC in robot control.

References in corpus (10)

• Learning Quadrupedal Locomotion over Challenging Terrain
• Learning robust perceptive locomotion for quadrupedal robots in the wild
• Control of a Quadrotor with Reinforcement Learning
• Whole-Body Nonlinear Model Predictive Control Through Contacts for Quadrupeds
• Deep Drone Racing: From Simulation to Reality with Domain Randomization
• Air taxi service for urban mobility: A critical review of recent developments, future challenges, and opportunities
• Time-Optimal Planning for Quadrotor Waypoint Flight
• Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight
• NeuroBEM: Hybrid Aerodynamic Quadrotor Model
• The Artificial Intelligence behind the winning entry to the 2019 AI Robotic Racing Competition

Cited by in corpus (22)

• Autonomous Drone Racing: A Survey
• Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics
• Optimality Principles in Spacecraft Neural Guidance and Control
• End-to-end Neural Network Based Quadcopter control
• Survey of Simulators for Aerial Robots: An Overview and In-Depth Systematic Comparisons
• High-Speed Interception Multicopter Control by Image-based Visual Servoing
• Target Search and Navigation in Heterogeneous Robot Systems with Deep Reinforcement Learning
• End-to-end Reinforcement Learning for Time-Optimal Quadcopter Flight
• Monte Carlo Tree Search with Spectral Expansion for Planning with Dynamical Systems
• Race Against the Machine: a Fully-annotated, Open-design Dataset of Autonomous and Piloted High-speed Flight
• Learning Quadrotor Control From Visual Features Using Differentiable Simulation
• Learning Agile Swimming: An End-to-End Approach without CPGs
• Optimal Control in Soft and Active Matter
• Learning to Walk and Fly with Adversarial Motion Priors
• AVOCADO: Adaptive Optimal Collision Avoidance driven by Opinion
• Swooper: Learning High-Speed Aerial Grasping With a Simple Gripper
• RESC: A Reinforcement Learning Based Search-to-Control Framework for Quadrotor Local Planning in Dense Environments
• Low-Latency Event-Based Velocimetry for Quadrotor Control in a Narrow Pipe
• Autonomous UAV Flight Navigation in Confined Spaces: A Reinforcement Learning Approach
• On Your Own: Pro-level Autonomous Drone Racing in Uninstrumented Arenas
• Mobile Robot Exploration Without Maps via Out-of-Distribution Deep Reinforcement Learning
• SPIRAL: Self-Play Incremental Racing Algorithm for Learning in Multi-Drone Competitions