EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

Abstract

Event-based cameras have shown great promise in a variety of situations where frame based cameras suffer, such as high speed motions and high dynamic range scenes. However, developing algorithms for event measurements requires a new class of hand crafted algorithms. Deep learning has shown great success in providing model free solutions to many problems in the vision community, but existing networks have been developed with frame based images in mind, and there does not exist the wealth of labeled data for events as there does for images for supervised training. To these points, we present EV-FlowNet, a novel self-supervised deep learning pipeline for optical flow estimation for event based cameras. In particular, we introduce an image based representation of a given event stream, which is fed into a self-supervised neural network as the sole input. The corresponding grayscale images captured from the same camera at the same time as the events are then used as a supervisory signal to provide a loss function at training time, given the estimated flow from the network. We show that the resulting network is able to accurately predict optical flow from events only in a variety of different scenes, with performance competitive to image based networks. This method not only allows for accurate estimation of dense optical flow, but also provides a framework for the transfer of other self-supervised methods to the event-based domain.

9 pages, 5 figures, 1 table. Accompanying video: https://youtu.be/eMHZBSoq0sE. Dataset: https://daniilidis-group.github.io/mvsec/, Robotics: Science and Systems 2018

References in corpus (7)

• The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
• The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
• Stacked Hourglass Networks for Human Pose Estimation
• DDD17: End-To-End DAVIS Driving Dataset
• Guided Optical Flow Learning
• Bioinspired Visual Motion Estimation
• ABMOF: A Novel Optical Flow Algorithm for Dynamic Vision Sensors

Cited by in corpus (65)

• Event-based Vision: A Survey
• Event-based Moving Object Detection and Tracking
• Event-based Stereo Visual Odometry
• Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception
• Event-Based Motion Segmentation by Motion Compensation
• Graph-based Spatial-temporal Feature Learning for Neuromorphic Vision Sensing
• Focus Is All You Need: Loss Functions For Event-based Vision
• Learning to Detect Objects with a 1 Megapixel Event Camera
• Secrets of Event-Based Optical Flow
• Event-based Motion Segmentation with Spatio-Temporal Graph Cuts
• Learning Dense and Continuous Optical Flow from an Event Camera
• Dense Continuous-Time Optical Flow from Events and Frames
• Motion Robust High-Speed Light-Weighted Object Detection With Event Camera
• Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation
• Asynchronous Tracking-by-Detection on Adaptive Time Surfaces for Event-based Object Tracking
• Optical flow estimation from event-based cameras and spiking neural networks
• Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers
• StereoSpike: Depth Learning with a Spiking Neural Network
• E-MLB: Multilevel Benchmark for Event-Based Camera Denoising
• Globally-Optimal Contrast Maximisation for Event Cameras
• LDNet: End-to-End Lane Marking Detection Approach Using a Dynamic Vision Sensor
• Event-based Background-Oriented Schlieren
• The Spatio-Temporal Poisson Point Process: A Simple Model for the Alignment of Event Camera Data
• Event Collapse in Contrast Maximization Frameworks
• Unsupervised Learning of Dense Optical Flow, Depth and Egomotion from Sparse Event Data
• Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow
• Neuromorphic Perception and Navigation for Mobile Robots: A Review
• Globally-Optimal Event Camera Motion Estimation
• EV-Catcher: High-Speed Object Catching Using Low-latency Event-based Neural Networks
• Lightweight Event-based Optical Flow Estimation via Iterative Deblurring
• Fast Event-based Optical Flow Estimation by Triplet Matching
• EVReflex: Dense Time-to-Impact Prediction for Event-based Obstacle Avoidance
• Hardware/Software co-design with ADC-Less In-memory Computing Hardware for Spiking Neural Networks
• Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy
• EvRepSL: Event-Stream Representation via Self-Supervised Learning for Event-Based Vision
• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation
• hARMS: A Hardware Acceleration Architecture for Real-Time Event-Based Optical Flow
• CycAs: Self-supervised Cycle Association for Learning Re-identifiable Descriptions
• Automotive Object Detection via Learning Sparse Events by Spiking Neurons
• Data-Driven Feature Tracking for Event Cameras With and Without Frames
• Representation Learning for Event-based Visuomotor Policies
• Graph-Based Object Classification for Neuromorphic Vision Sensing
• Events-to-Video: Bringing Modern Computer Vision to Event Cameras
• Event-based Camera Simulation using Monte Carlo Path Tracing with Adaptive Denoising
• Event-Driven Visual-Tactile Sensing and Learning for Robots
• An Asynchronous Intensity Representation for Framed and Event Video Sources
• YoloTag: Vision-based Robust UAV Navigation with Fiducial Markers
• Event Probability Mask (EPM) and Event Denoising Convolutional Neural Network (EDnCNN) for Neuromorphic Cameras
• EventNet: Asynchronous Recursive Event Processing
• Zoom-In-to-Check: Boosting Video Interpolation via Instance-level Discrimination
• Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion
• Combined Physics and Event Camera Simulator for Slip Detection
• Realtime Time Synchronized Event-based Stereo
• Event-VPR: End-to-End Weakly Supervised Network Architecture for Event-based Visual Place Recognition
• EvAn: Neuromorphic Event-based Anomaly Detection
• Event-based Timestamp Image Encoding Network for Human Action Recognition and Anticipation
• EventHPE: Event-based 3D Human Pose and Shape Estimation
• Low-Latency Event-Based Velocimetry for Quadrotor Control in a Narrow Pipe
• Globally Optimal Contrast Maximisation for Event-based Motion Estimation
• Asynchronous "Events" are Better For Motion Estimation
• Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow
• Motion Equivariance OF Event-based Camera Data with the Temporal Normalization Transform
• Distance Surface for Event-Based Optical Flow
• EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream
• Non-Uniform Exposure Imaging via Neuromorphic Shutter Control