FlightGoggles: A Modular Framework for Photorealistic Camera, Exteroceptive Sensor, and Dynamics Simulation

Abstract

FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it provides the ability to combine (i) synthetic exteroceptive measurements generated in silico in real time and (ii) vehicle dynamics and proprioceptive measurements generated in motio by vehicle(s) in a motion-capture facility. FlightGoggles is capable of simulating a virtual-reality environment around autonomous vehicle(s). While a vehicle is in flight in the FlightGoggles virtual reality environment, exteroceptive sensors are rendered synthetically in real time while all complex extrinsic dynamics are generated organically through the natural interactions of the vehicle. The FlightGoggles framework allows for researchers to accelerate development by circumventing the need to estimate complex and hard-to-model interactions such as aerodynamics, motor mechanics, battery electrochemistry, and behavior of other agents. The ability to perform vehicle-in-the-loop experiments with photorealistic exteroceptive sensor simulation facilitates novel research directions involving, e.g., fast and agile autonomous flight in obstacle-rich environments, safe human interaction, and flexible sensor selection. FlightGoggles has been utilized as the main test for selecting nine teams that will advance in the AlphaPilot autonomous drone racing challenge. We survey approaches and results from the top AlphaPilot teams, which may be of independent interest.

Initial version appeared at IROS 2019. Supplementary material can be found at https://flightgoggles.mit.edu. Revision includes description of new FlightGoggles features, such as a photogrammetric model of the MIT Stata Center, new rendering settings, and a Python API

Cited by in corpus (16)

• A Survey of Wireless Networks for Future Aerial COMmunications (FACOM)
• Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight
• Autonomous Drone Racing: A Survey
• Learning Minimum-Time Flight in Cluttered Environments
• Flightmare: A Flexible Quadrotor Simulator
• Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents
• Survey of Simulators for Aerial Robots: An Overview and In-Depth Systematic Comparisons
• L1-Adaptive MPPI Architecture for Robust and Agile Control of Multirotors
• Autonomous Drone Racing with Deep Reinforcement Learning
• AirSim Drone Racing Lab
• Learning Model Predictive Control for Quadrotors
• Visual-based Safe Landing for UAVs in Populated Areas: Real-time Validation in Virtual Environments
• UAV-Borne Mapping Algorithms for Low-Altitude and High-Speed Drone Applications
• Continuous-time State & Dynamics Estimation using a Pseudo-Spectral Parameterization
• Fast-Racing: An Open-source Strong Baseline for SE(3) Planning in Autonomous Drone Racing
• SPIRAL: Self-Play Incremental Racing Algorithm for Learning in Multi-Drone Competitions