Event-based Vision: A Survey

Abstract

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world.

References in corpus (24)

• Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age
• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• A scalable multi-core architecture with heterogeneous memory structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs)
• The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
• The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
• Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars
• Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios
• EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
• Event-based Moving Object Detection and Tracking
• A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation
• HFirst: A Temporal Approach to Object Recognition
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• Neuromorphic Deep Learning Machines
• Event-based, 6-DOF Camera Tracking from Photometric Depth Maps
• Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception
• Continuous-Time Visual-Inertial Odometry for Event Cameras
• Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks
• Event-Based Motion Segmentation by Motion Compensation
• Focus Is All You Need: Loss Functions For Event-based Vision
• Semi-Dense 3D Reconstruction with a Stereo Event Camera
• Asynchronous Spatial Image Convolutions for Event Cameras
• Vertical Landing for Micro Air Vehicles using Event-Based Optical Flow
• Independent Motion Detection with Event-driven Cameras

Cited by in corpus (176)

• Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racing
• Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review
• Event-based Stereo Visual Odometry
• Event-Based Fusion for Motion Deblurring with Cross-modal Attention
• Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipeline
• Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight
• Autonomous Drone Racing: A Survey
• Visual SLAM: What are the Current Trends and What to Expect?
• The Hilti SLAM Challenge Dataset
• VECtor: A Versatile Event-Centric Benchmark for Multi-Sensor SLAM
• Secrets of Event-Based Optical Flow
• ESVIO: Event-based Stereo Visual Inertial Odometry
• Autonomous Aerial Delivery Vehicles, a Survey of Techniques on how Aerial Package Delivery is Achieved
• Autonomous Quadrotor Flight despite Rotor Failure with Onboard Vision Sensors: Frames vs. Events
• Learning Dense and Continuous Optical Flow from an Event Camera
• Visuo-Haptic Object Perception for Robots: An Overview
• Exploring the landscapes of "computing": digital, neuromorphic, unconventional -- and beyond
• Event-based Imaging Velocimetry -- An Assessment of Event-based Cameras for the Measurement of Fluid Flows
• From SLAM to Situational Awareness: Challenges and Survey
• Event-based visual place recognition with ensembles of temporal windows
• Free as a Bird: Event-based Dynamic Sense-and-Avoid for Ornithopter Robot Flight
• Dense Continuous-Time Optical Flow from Events and Frames
• Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision
• Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation
• Motion Robust High-Speed Light-Weighted Object Detection With Event Camera
• SODFormer: Streaming Object Detection with Transformer Using Events and Frames
• STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks
• Optical flow estimation from event-based cameras and spiking neural networks
• Moving Object Detection for Event-based vision using Graph Spectral Clustering
• Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers
• Neuromorphic control for optic-flow-based landings of MAVs using the Loihi processor
• StereoSpike: Depth Learning with a Spiking Neural Network
• E-MLB: Multilevel Benchmark for Event-Based Camera Denoising
• CMax-SLAM: Event-based Rotational-Motion Bundle Adjustment and SLAM System using Contrast Maximization
• Event-based imaging velocimetry using pulsed illumination
• 3ET: Efficient Event-based Eye Tracking using a Change-Based ConvLSTM Network
• HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks
• Neuromorphic Eye-in-Hand Visual Servoing
• Vision-based Learning for Drones: A Survey
• Multi-Event-Camera Depth Estimation and Outlier Rejection by Refocused Events Fusion
• Event-based Background-Oriented Schlieren
• The GRIFFIN Perception Dataset: Bridging the Gap Between Flapping-Wing Flight and Robotic Perception
• Event Collapse in Contrast Maximization Frameworks
• ESVO2: Direct Visual-Inertial Odometry with Stereo Event Cameras
• Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow
• Neuromorphic Perception and Navigation for Mobile Robots: A Review
• Neuromorphic Control
• Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments
• E-Calib: A Fast, Robust and Accurate Calibration Toolbox for Event Cameras
• Lightweight Event-based Optical Flow Estimation via Iterative Deblurring
• Neuromorphic Imaging with Joint Image Deblurring and Event Denoising
• Stereo Event-based, 6-DOF Pose Tracking for Uncooperative Spacecraft
• A topological solution to object segmentation and tracking
• Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras
• In the Blink of an Eye: Event-based Emotion Recognition
• Neuromorphic High-Frequency 3D Dancing Pose Estimation in Dynamic Environment
• Time Encoding via Unlimited Sampling: Theory, Algorithms and Hardware Validation
• LECalib: Line-Based Event Camera Calibration
• Neuromorphic metamaterials for mechanosensing and perceptual associative learning
• Fast Event-based Optical Flow Estimation by Triplet Matching
• Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams
• EVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction
• Asynchronous Blob Tracker for Event Cameras
• Embodied Neuromorphic Artificial Intelligence for Robotics: Perspectives, Challenges, and Research Development Stack
• Event-based Stereo Depth Estimation: A Survey
• Computation-Communication Trade-offs and Sensor Selection in Real-time Estimation for Processing Networks
• Neuromorphic Visual Scene Understanding with Resonator Networks
• A Multi-Layer Software Architecture for Aerial Cognitive Multi-Robot Systems in Power Line Inspection Tasks
• A Fast Geometric Regularizer to Mitigate Event Collapse in the Contrast Maximization Framework
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Evolved Neuromorphic Control for High Speed Divergence-based Landings of MAVs
• EvGNN: An Event-driven Graph Neural Network Accelerator for Edge Vision
• Event-based Stereo Visual Odometry with Native Temporal Resolution via Continuous-time Gaussian Process Regression
• Event-based vision on FPGAs -- a survey
• Applications of Spiking Neural Networks in Visual Place Recognition
• An assessment of event-based imaging velocimetry for efficient estimation of low-dimensional coordinates in turbulent flows
• BIT-VO: Visual Odometry at 300 FPS using Binary Features from the Focal Plane
• EvRepSL: Event-Stream Representation via Self-Supervised Learning for Event-Based Vision
• RN-Net: Reservoir Nodes-Enabled Neuromorphic Vision Sensing Network
• A Composable Dynamic Sparse Dataflow Architecture for Efficient Event-based Vision Processing on FPGA
• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation
• hARMS: A Hardware Acceleration Architecture for Real-Time Event-Based Optical Flow
• A Heterogeneous RISC-V based SoC for Secure Nano-UAV Navigation
• SGE: Structured Light System Based on Gray Code with an Event Camera
• High-temporal-resolution event-based vehicle detection and tracking
• Classification of Primitive Manufacturing Tasks from Filtered Event Data
• A compact neuromorphic system for ultra-energy-efficient, on-device robot localization
• Event-Based Imaging of Levitated Microparticles
• N-Omniglot, a large-scale neuromorphic dataset for spatio-temporal sparse few-shot learning
• Asynchrony Increases Efficiency: Time Encoding of Videos and Low-Rank Signals
• FAITH: Fast iterative half-plane focus of expansion estimation using event-based optic flow
• Low-power, Continuous Remote Behavioral Localization with Event Cameras
• TTPOINT: A Tensorized Point Cloud Network for Lightweight Action Recognition with Event Cameras
• Motion-Aware Optical Camera Communication with Event Cameras
• EF-Calib: Spatiotemporal Calibration of Event- and Frame-Based Cameras Using Continuous-Time Trajectories
• Memory-Efficient Graph Convolutional Networks for Object Classification and Detection with Event Cameras
• Traffic Sign Detection With Event Cameras and DCNN
• Dense Voxel 3D Reconstruction Using a Monocular Event Camera
• Passive Non-line-of-sight Imaging for Moving Targets with an Event Camera
• Data-Driven Feature Tracking for Event Cameras With and Without Frames
• Visual Odometry with Neuromorphic Resonator Networks
• Moving Object Detection for Event-based Vision using k-means Clustering
• Precise spiking motifs in neurobiological and neuromorphic data
• Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment
• Event-based tracking of human hands
• ES-PTAM: Event-based Stereo Parallel Tracking and Mapping
• An N-Point Linear Solver for Line and Motion Estimation with Event Cameras
• Encoding Event-Based Data With a Hybrid SNN Guided Variational Auto-encoder in Neuromorphic Hardware
• Tracking 6-DoF Object Motion from Events and Frames
• Sampling Theory of Jointly Bandlimited Time-vertex Graph Signals
• Event-based dataset for the detection and classification of manufacturing assembly tasks
• X-maps: Direct Depth Lookup for Event-based Structured Light Systems
• Fourier-based Action Recognition for Wildlife Behavior Quantification with Event Cameras
• Event Encryption: Rethinking Privacy Exposure for Neuromorphic Imaging
• Cross-correlation image analysis for real-time particle tracking
• ASAP: Adaptive Transmission Scheme for Online Processing of Event-based Algorithms
• MULi-Ev: Maintaining Unperturbed LiDAR-Event Calibration
• Learning Parallax for Stereo Event-based Motion Deblurring
• Event-based Camera Simulation using Monte Carlo Path Tracing with Adaptive Denoising
• Interpolation-Based Event Visual Data Filtering Algorithms
• Photonic Neuromorphic Accelerators for Event-Based Imaging Flow Cytometry
• Event-enhanced Passive Non-line-of-sight imaging for moving objects with Physical embedding
• Long Distance GNSS-Denied Visual Inertial Navigation for Autonomous Fixed Wing Unmanned Air Vehicles: SO(3) Manifold Filter based on Virtual Vision Sensor
• An Asynchronous Intensity Representation for Framed and Event Video Sources
• Neuromorphic computing for attitude estimation onboard quadrotors
• Sensors for Mobile Robots
• Covariant spatio-temporal receptive fields for spiking neural networks
• Neuromorphic detection and cooling of microparticles in arrays
• Event Fusion Photometric Stereo Network
• MouseSIS: A Frames-and-Events Dataset for Space-Time Instance Segmentation of Mice
• Optimising Event-Driven Spiking Neural Network with Regularisation and Cutoff
• Current trends and future directions in event-based control
• Asynchronous Multi-Object Tracking with an Event Camera
• Ensemble-Based Event Camera Place Recognition Under Varying Illumination
• EvTexture++: Event-Driven Texture Enhancement for Video Super-Resolution
• Optical Flow-Guided 6DoF Object Pose Tracking with an Event Camera
• On the Benefits of Visual Stabilization for Frame- and Event-based Perception
• High-definition event frame generation using SoC FPGA devices
• Neuromorphic Vision Data Coding: Classifying and Reviewing the Literature
• Combined Physics and Event Camera Simulator for Slip Detection
• Hardware-Accelerated Event-Graph Neural Networks for Low-Latency Time-Series Classification on SoC FPGA
• Asynchronous Event-Inertial Odometry using a Unified Gaussian Process Regression Framework
• Bilevel Inverse Problems in Neuromorphic Imaging
• Hardware-Algorithm Re-engineering of Retinal Circuit for Intelligent Object Motion Segmentation
• Event-RGB Fusion for Spacecraft Pose Estimation Under Harsh Lighting
• Event-Based Method for High-Speed 3D Deformation Measurement under Extreme Illumination Conditions
• Hardware-Free Event Cameras Temporal Synchronization Based on Event Density Alignment
• Spiking Two-Stream Methods with Unsupervised STDP-based Learning for Action Recognition
• YCB-Ev 1.1: Event-vision dataset for 6DoF object pose estimation
• Drone Detection Using a Low-Power Neuromorphic Virtual Tripwire
• Event-based reconstruction of time-resolved centreline deformation of flapping flags
• Towards End-to-End Neuromorphic Event-based 3D Object Reconstruction Without Physical Priors
• Event-based Photometric Bundle Adjustment
• Event-based Detectors for Laser Guide Star Tip-Tilt Sensing
• Event-based Visual Tracking in Dynamic Environments
• Scalable Event-Based Video Streaming for Machines with MoQ
• VibES: Induced Vibration for Persistent Event-Based Sensing
• IMA-Catcher: An IMpact-Aware Nonprehensile Catching Framework based on Combined Optimization and Learning
• Inter-event Interval Microscopy for Event Cameras
• Event-based Mosaicing Bundle Adjustment
• Low-Latency Event-Based Velocimetry for Quadrotor Control in a Narrow Pipe
• Non-Uniform Exposure Imaging via Neuromorphic Shutter Control
• Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow
• Zero-Shot Temporal Resolution Domain Adaptation for Spiking Neural Networks
• E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations
• A neuromorphic camera for tracking passive and active matter with lower data throughput
• Asynchronous Event-Based Spectroscopy for Microsecond-Resolved Spectral Reconstruction
• FPGA-Based Hardware Architecture for Contrast Maximization in Event-Based Vision
• Event2Audio: Event-Based Optical Vibration Sensing
• ESL: Event-based Structured Light
• Hardware-aware Graph Neural Networks prunning for embedded event-based vision
• Sequence-SOD: Bio-inspired Sequence-aware Spiking ObjectDetection for Event Cameras
• Real-Time Frame- and Event-based Object Detection with Spiking Neural Networks on Edge Neuromorphic Hardware: Design, Deployment and Benchmark
• AsynEIO: Asynchronous Monocular Event-Inertial Odometry Using Gaussian Process Regression
• Event-based Civil Infrastructure Visual Defect Detection: ev-CIVIL Dataset and Benchmark
• 2022 Roadmap on Neuromorphic Computing and Engineering