Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy

Abstract

Event cameras are novel vision sensors that sample, in an asynchronous fashion, brightness increments with low latency and high temporal resolution. The resulting streams of events are of high value by themselves, especially for high speed motion estimation. However, a growing body of work has also focused on the reconstruction of intensity frames from the events, as this allows bridging the gap with the existing literature on appearance- and frame-based computer vision. Recent work has mostly approached this problem using neural networks trained with synthetic, ground-truth data. In this work we approach, for the first time, the intensity reconstruction problem from a self-supervised learning perspective. Our method, which leverages the knowledge of the inner workings of event cameras, combines estimated optical flow and the event-based photometric constancy to train neural networks without the need for any ground-truth or synthetic data. Results across multiple datasets show that the performance of the proposed self-supervised approach is in line with the state-of-the-art. Additionally, we propose a novel, lightweight neural network for optical flow estimation that achieves high speed inference with only a minor drop in performance.

arXiv comment9 pages, 6 figures, 5 tables

References in corpus (7)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
• EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
• Event-based Moving Object Detection and Tracking
• Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception
• Event-based Camera Pose Tracking using a Generative Event Model
• How Do Neural Networks Estimate Optical Flow? A Neuropsychology-Inspired Study

Cited by in corpus (4)

• Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras
• SiamEvent: Event-based Object Tracking via Edge-aware Similarity Learning with Siamese Networks
• Deep Learning for HDR Imaging: State-of-the-Art and Future Trends
• EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation