Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents

Abstract

Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural network performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Comparing success rates for completing a challenging race track by autonomous flight, our results show that the attention-prediction based controller (88% success rate) outperforms the RGB-image (61% success rate) and feature-tracks (55% success rate) controller baselines. Furthermore, visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.

arXiv comment9 pages, 6 figures

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator
• End to End Learning for Self-Driving Cars
• Deep Visual Attention Prediction
• Learning High-Speed Flight in the Wild
• Deep Drone Racing: From Simulation to Reality with Domain Randomization
• Time-Optimal Planning for Quadrotor Waypoint Flight
• Fast, Autonomous Flight in GPS-Denied and Cluttered Environments
• FlightGoggles: A Modular Framework for Photorealistic Camera, Exteroceptive Sensor, and Dynamics Simulation
• Autonomous drone race: A computationally efficient vision-based navigation and control strategy
• Human-Piloted Drone Racing: Visual Processing and Control
• Flightmare: A Flexible Quadrotor Simulator
• RAPTOR: Robust and Perception-aware Trajectory Replanning for Quadrotor Fast Flight
• DeepRacing: Parameterized Trajectories for Autonomous Racing
• Expertise Affects Drone Racing Performance
• GPA-Teleoperation: Gaze Enhanced Perception-aware Safe Assistive Aerial Teleoperation

Cited by in corpus (4)

• Autonomous Drone Racing: A Survey
• Reinforcement Learning for Shared Autonomy Drone Landings
• The Power of Input: Benchmarking Zero-Shot Sim-To-Real Transfer of Reinforcement Learning Control Policies for Quadrotor Control
• TagTeam: Towards Wearable-Assisted, Implicit Guidance for Human--Drone Teams