Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios

Abstract

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high speed motions or in scenes characterized by high dynamic range. However, event cameras output only little information when the amount of motion is limited, such as in the case of almost still motion. Conversely, standard cameras provide instant and rich information about the environment most of the time (in low-speed and good lighting scenarios), but they fail severely in case of fast motions, or difficult lighting such as high dynamic range or low light scenes. In this paper, we present the first state estimation pipeline that leverages the complementary advantages of these two sensors by fusing in a tightly-coupled manner events, standard frames, and inertial measurements. We show on the publicly available Event Camera Dataset that our hybrid pipeline leads to an accuracy improvement of 130% over event-only pipelines, and 85% over standard-frames-only visual-inertial systems, while still being computationally tractable. Furthermore, we use our pipeline to demonstrate - to the best of our knowledge - the first autonomous quadrotor flight using an event camera for state estimation, unlocking flight scenarios that were not reachable with traditional visual-inertial odometry, such as low-light environments and high-dynamic range scenes.

8 pages, 9 figures, 2 tables

References in corpus (1)

• The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM

Cited by in corpus (63)

• Event-based Vision: A Survey
• Event-based Moving Object Detection and Tracking
• A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation
• Deep Drone Racing: From Simulation to Reality with Domain Randomization
• Event-based Stereo Visual Odometry
• Continuous-Time Visual-Inertial Odometry for Event Cameras
• Asynchronous, Photometric Feature Tracking using Events and Frames
• Event-Based Motion Segmentation by Motion Compensation
• Focus Is All You Need: Loss Functions For Event-based Vision
• Semi-Dense 3D Reconstruction with a Stereo Event Camera
• Autonomous Drone Racing: A Survey
• Event-aided Direct Sparse Odometry
• Visual SLAM: What are the Current Trends and What to Expect?
• VECtor: A Versatile Event-Centric Benchmark for Multi-Sensor SLAM
• ESVIO: Event-based Stereo Visual Inertial Odometry
• Autonomous Quadrotor Flight despite Rotor Failure with Onboard Vision Sensors: Frames vs. Events
• Event-based Motion Segmentation with Spatio-Temporal Graph Cuts
• From SLAM to Situational Awareness: Challenges and Survey
• Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision
• Asynchronous Spatial Image Convolutions for Event Cameras
• Neuromorphic Event-Based Slip Detection and suppression in Robotic Grasping and Manipulation
• SODFormer: Streaming Object Detection with Transformer Using Events and Frames
• Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers
• CMax-SLAM: Event-based Rotational-Motion Bundle Adjustment and SLAM System using Contrast Maximization
• Sensors, SLAM and Long-term Autonomy: A Review
• Long-Duration Fully Autonomous Operation of Rotorcraft Unmanned Aerial Systems for Remote-Sensing Data Acquisition
• Redesigning SLAM for Arbitrary Multi-Camera Systems
• IDOL: A Framework for IMU-DVS Odometry using Lines
• The Spatio-Temporal Poisson Point Process: A Simple Model for the Alignment of Event Camera Data
• The GRIFFIN Perception Dataset: Bridging the Gap Between Flapping-Wing Flight and Robotic Perception
• Event Collapse in Contrast Maximization Frameworks
• ESVO2: Direct Visual-Inertial Odometry with Stereo Event Cameras
• Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments
• A Fast Geometric Regularizer to Mitigate Event Collapse in the Contrast Maximization Framework
• Event-based Stereo Visual Odometry with Native Temporal Resolution via Continuous-time Gaussian Process Regression
• Applications of Spiking Neural Networks in Visual Place Recognition
• SGE: Structured Light System Based on Gray Code with an Event Camera
• Event Transformer+. A multi-purpose solution for efficient event data processing
• A 5-Point Minimal Solver for Event Camera Relative Motion Estimation
• FAITH: Fast iterative half-plane focus of expansion estimation using event-based optic flow
• EF-Calib: Spatiotemporal Calibration of Event- and Frame-Based Cameras Using Continuous-Time Trajectories
• DH-PTAM: A Deep Hybrid Stereo Events-Frames Parallel Tracking And Mapping System
• Visual Odometry with Neuromorphic Resonator Networks
• ES-PTAM: Event-based Stereo Parallel Tracking and Mapping
• An N-Point Linear Solver for Line and Motion Estimation with Event Cameras
• Enhancing exploration algorithms for navigation with visual SLAM
• Learning to Segment Dynamic Objects using SLAM Outliers
• Event-based Camera Simulation using Monte Carlo Path Tracing with Adaptive Denoising
• A Fast and Accurate Optical Flow Camera for Resource-Constrained Edge Applications
• On the Benefits of Visual Stabilization for Frame- and Event-based Perception
• High-definition event frame generation using SoC FPGA devices
• Event-RGB Fusion for Spacecraft Pose Estimation Under Harsh Lighting
• Asynchronous Event-Inertial Odometry using a Unified Gaussian Process Regression Framework
• Visual Inertial Odometry using Focal Plane Binary Features (BIT-VIO)
• Neuromorphic Vision Data Coding: Classifying and Reviewing the Literature
• Event-based Photometric Bundle Adjustment
• VibES: Induced Vibration for Persistent Event-Based Sensing
• Event-based Mosaicing Bundle Adjustment
• AsynEIO: Asynchronous Monocular Event-Inertial Odometry Using Gaussian Process Regression
• ESL: Event-based Structured Light
• Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow
• Event-based Civil Infrastructure Visual Defect Detection: ev-CIVIL Dataset and Benchmark
• LuvHarris: A Practical Corner Detector for Event-cameras