Unsupervised Learning of Dense Optical Flow, Depth and Egomotion from Sparse Event Data

Abstract

In this work we present a lightweight, unsupervised learning pipeline for \textit{dense} depth, optical flow and egomotion estimation from sparse event output of the Dynamic Vision Sensor (DVS). To tackle this low level vision task, we use a novel encoder-decoder neural network architecture - ECN. Our work is the first monocular pipeline that generates dense depth and optical flow from sparse event data only. The network works in self-supervised mode and has just 150k parameters. We evaluate our pipeline on the MVSEC self driving dataset and present results for depth, optical flow and and egomotion estimation. Due to the lightweight design, the inference part of the network runs at 250 FPS on a single GPU, making the pipeline ready for realtime robotics applications. Our experiments demonstrate significant improvements upon previous works that used deep learning on event data, as well as the ability of our pipeline to perform well during both day and night.

References in corpus (9)

• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
• EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
• Event-based Moving Object Detection and Tracking
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Unsupervised Learning of Depth and Ego-Motion from Video
• Learning Depth from Monocular Videos using Direct Methods
• Improved Bilinear Pooling with CNNs
• Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion

Cited by in corpus (12)

• Focus Is All You Need: Loss Functions For Event-based Vision
• SGE: Structured Light System Based on Gray Code with an Event Camera
• FAITH: Fast iterative half-plane focus of expansion estimation using event-based optic flow
• E3D: Event-Based 3D Shape Reconstruction
• Motion Equivariant Networks for Event Cameras with the Temporal Normalization Transform
• Event-Based Feature Tracking in Continuous Time with Sliding Window Optimization
• Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion
• EvAn: Neuromorphic Event-based Anomaly Detection
• Learning Event-Based Motion Deblurring
• Spatiotemporal Registration for Event-based Visual Odometry
• Motion Equivariance OF Event-based Camera Data with the Temporal Normalization Transform
• Distance Surface for Event-Based Optical Flow