Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation

Abstract

Current optical flow and point-tracking methods rely heavily on synthetic datasets. Event cameras are novel vision sensors with advantages in challenging visual conditions, but state-of-the-art frame-based methods cannot be easily adapted to event data due to the limitations of current event simulators. We introduce a novel self-supervised loss combining the Contrast Maximization framework with a non-linear motion prior in the form of pixel-level trajectories and propose an efficient solution to solve the high-dimensional assignment problem between non-linear trajectories and events. Their effectiveness is demonstrated in two scenarios: In dense continuous-time motion estimation, our method improves the zero-shot performance of a synthetically trained model on the real-world dataset EVIMO2 by 29%. In optical flow estimation, our method elevates a simple UNet to achieve state-of-the-art performance among self-supervised methods on the DSEC optical flow benchmark. Our code is available at https://github.com/tub-rip/MotionPriorCMax.

24 pages, 8 figures, 8 tables, Project Page: https://github.com/tub-rip/MotionPriorCMax

References in corpus (16)

• Event-based Vision: A Survey
• A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
• The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
• EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
• A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation
• Secrets of Event-Based Optical Flow
• Learning Dense and Continuous Optical Flow from an Event Camera
• Dense Continuous-Time Optical Flow from Events and Frames
• CMax-SLAM: Event-based Rotational-Motion Bundle Adjustment and SLAM System using Contrast Maximization
• Event-based Background-Oriented Schlieren
• Event Collapse in Contrast Maximization Frameworks
• A Fast Geometric Regularizer to Mitigate Event Collapse in the Contrast Maximization Framework
• Low-power, Continuous Remote Behavioral Localization with Event Cameras
• EVIMO2: An Event Camera Dataset for Motion Segmentation, Optical Flow, Structure from Motion, and Visual Inertial Odometry in Indoor Scenes with Monocular or Stereo Algorithms
• Stereo Co-capture System for Recording and Tracking Fish with Frame- and Event Cameras
• Event-Based Feature Tracking in Continuous Time with Sliding Window Optimization

Cited by in corpus (3)

• MouseSIS: A Frames-and-Events Dataset for Space-Time Instance Segmentation of Mice
• Event-based Photometric Bundle Adjustment
• FPGA-Based Hardware Architecture for Contrast Maximization in Event-Based Vision