Deep Drone Racing: From Simulation to Reality with Domain Randomization

Abstract

Dynamically changing environments, unreliable state estimation, and operation under severe resource constraints are fundamental challenges that limit the deployment of small autonomous drones. We address these challenges in the context of autonomous, vision-based drone racing in dynamic environments. A racing drone must traverse a track with possibly moving gates at high speed. We enable this functionality by combining the performance of a state-of-the-art planning and control system with the perceptual awareness of a convolutional neural network (CNN). The resulting modular system is both platform- and domain-independent: it is trained in simulation and deployed on a physical quadrotor without any fine-tuning. The abundance of simulated data, generated via domain randomization, makes our system robust to changes of illumination and gate appearance. To the best of our knowledge, our approach is the first to demonstrate zero-shot sim-to-real transfer on the task of agile drone flight. We extensively test the precision and robustness of our system, both in simulation and on a physical platform, and show significant improvements over the state of the art.

Accepted as a Regular Paper to the IEEE Transactions on Robotics Journal. arXiv admin note: substantial text overlap with arXiv:1806.08548

References in corpus (11)

• ORB-SLAM: a Versatile and Accurate Monocular SLAM System
• VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator
• Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age
• Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios
• Differential Flatness of Quadrotor Dynamics Subject to Rotor Drag for Accurate Tracking of High-Speed Trajectories
• Aggressive Quadrotor Flight through Narrow Gaps with Onboard Sensing and Computing using Active Vision
• Driving Policy Transfer via Modularity and Abstraction
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Sim-to-Real Robot Learning from Pixels with Progressive Nets
• Autonomous drone race: A computationally efficient vision-based navigation and control strategy

Cited by in corpus (44)

• Learning High-Speed Flight in the Wild
• Time-Optimal Planning for Quadrotor Waypoint Flight
• Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning
• Autonomous Drone Racing: A Survey
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• Finite-Horizon LQR Control of Quadrotors on $S{E}_2(3)$
• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
• Learning Minimum-Time Flight in Cluttered Environments
• Flightmare: A Flexible Quadrotor Simulator
• Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics
• Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• Learning High-Level Policies for Model Predictive Control
• TartanAir: A Dataset to Push the Limits of Visual SLAM
• Multi-visual modality micro drone-based structural damage detection
• Learning to Fly via Deep Model-Based Reinforcement Learning
• High-Speed Interception Multicopter Control by Image-based Visual Servoing
• The Artificial Intelligence behind the winning entry to the 2019 AI Robotic Racing Competition
• Reinforcement Learning for Shared Autonomy Drone Landings
• AirSim Drone Racing Lab
• Domain Adaptive Detection of MAVs: A Benchmark and Noise Suppression Network
• Using Simulation Optimization to Improve Zero-shot Policy Transfer of Quadrotors
• Vision-based Navigation of Unmanned Aerial Vehicles in Orchards: An Imitation Learning Approach
• CPC: Complementary Progress Constraints for Time-Optimal Quadrotor Trajectories
• Learning Agile Swimming: An End-to-End Approach without CPGs
• Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis
• MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• Deep Transformer Network for Monocular Pose Estimation of Shipborne Unmanned Aerial Vehicle
• Robust Navigation for Racing Drones based on Imitation Learning and Modularization
• Time-Optimal Online Replanning for Agile Quadrotor Flight
• Model-based controller assisted domain randomization for transient vibration suppression of nonlinear powertrain system with parametric uncertainty
• AutoTune: Controller Tuning for High-Speed Flight
• Autonomous UAV Flight Navigation in Confined Spaces: A Reinforcement Learning Approach
• Deep Reinforcement Learning with Linear Quadratic Regulator Regions
• Improving the Robustness of Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control
• Training Lightweight CNNs for Human-Nanodrone Proximity Interaction from Small Datasets using Background Randomization
• Dynamics-Invariant Quadrotor Control using Scale-Aware Deep Reinforcement Learning
• SPIRAL: Self-Play Incremental Racing Algorithm for Learning in Multi-Drone Competitions
• Robustifying Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control
• Online Algorithms and Policies Using Adaptive and Machine Learning Approaches
• Learn by Observation: Imitation Learning for Drone Patrolling from Videos of A Human Navigator
• Vision-Based Autonomous Drone Control using Supervised Learning in Simulation