Asynchronous, Photometric Feature Tracking using Events and Frames

Abstract

We present a method that leverages the complementarity of event cameras and standard cameras to track visual features with low-latency. Event cameras are novel sensors that output pixel-level brightness changes, called "events". They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, because the same scene pattern can produce different events depending on the motion direction, establishing event correspondences across time is challenging. By contrast, standard cameras provide intensity measurements (frames) that do not depend on motion direction. Our method extracts features on frames and subsequently tracks them asynchronously using events, thereby exploiting the best of both types of data: the frames provide a photometric representation that does not depend on motion direction and the events provide low-latency updates. In contrast to previous works, which are based on heuristics, this is the first principled method that uses raw intensity measurements directly, based on a generative event model within a maximum-likelihood framework. As a result, our method produces feature tracks that are both more accurate (subpixel accuracy) and longer than the state of the art, across a wide variety of scenes.

arXiv comment22 pages, 15 figures, Video: https://youtu.be/A7UfeUnG6c4

References in corpus (8)

• ORB-SLAM: a Versatile and Accurate Monocular SLAM System
• The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
• Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars
• Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios
• A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation
• Event-based, 6-DOF Camera Tracking from Photometric Depth Maps
• Event-based Camera Pose Tracking using a Generative Event Model
• Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation

Cited by in corpus (23)

• Semi-Dense 3D Reconstruction with a Stereo Event Camera
• Learning Dense and Continuous Optical Flow from an Event Camera
• Event-based visual place recognition with ensembles of temporal windows
• Asynchronous Tracking-by-Detection on Adaptive Time Surfaces for Event-based Object Tracking
• ESVO2: Direct Visual-Inertial Odometry with Stereo Event Cameras
• VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows
• Event Camera Calibration of Per-pixel Biased Contrast Threshold
• Bringing a Blurry Frame Alive at High Frame-Rate with an Event Camera
• EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning
• Events-to-Video: Bringing Modern Computer Vision to Event Cameras
• Event-based Action Recognition Using Timestamp Image Encoding Network
• FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras
• Bilevel Inverse Problems in Neuromorphic Imaging
• EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera
• SE-Harris and eSUSAN: Asynchronous Event-Based Corner Detection Using Megapixel Resolution CeleX-V Camera
• Event-based Timestamp Image Encoding Network for Human Action Recognition and Anticipation
• EventHPE: Event-based 3D Human Pose and Shape Estimation
• Single Image Optical Flow Estimation with an Event Camera
• FPGA-Based Hardware Architecture for Contrast Maximization in Event-Based Vision
• Motion Equivariance OF Event-based Camera Data with the Temporal Normalization Transform
• Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction
• Asynchronous "Events" are Better For Motion Estimation
• Distance Surface for Event-Based Optical Flow