Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow

Abstract

Event cameras have the potential to capture continuous motion information over time and space, making them well-suited for optical flow estimation. However, most existing learning-based methods for event-based optical flow adopt frame-based techniques, ignoring the spatio-temporal characteristics of events. Additionally, these methods assume linear motion between consecutive events within the loss time window, which increases optical flow errors in long-time sequences. In this work, we observe that rich spatio-temporal information and accurate nonlinear motion between events are crucial for event-based optical flow estimation. Therefore, we propose E-NMSTFlow, a novel unsupervised event-based optical flow network focusing on long-time sequences. We propose a Spatio-Temporal Motion Feature Aware (STMFA) module and an Adaptive Motion Feature Enhancement (AMFE) module, both of which utilize rich spatio-temporal information to learn spatio-temporal data associations. Meanwhile, we propose a nonlinear motion compensation loss that utilizes the accurate nonlinear motion between events to improve the unsupervised learning of our network. Extensive experiments demonstrate the effectiveness and superiority of our method. Remarkably, our method ranks first among unsupervised learning methods on the MVSEC and DSEC-Flow datasets. Our project page is available at https://wynelio.github.io/E-NMSTFlow.

Accepted to ICRA 2025. Project Page: https://wynelio.github.io/E-NMSTFlow

References in corpus (11)

• VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator
• Event-based Vision: A Survey
• Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars
• Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios
• EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
• Event-based Moving Object Detection and Tracking
• Secrets of Event-Based Optical Flow
• A compact formula for the derivative of a 3-D rotation in exponential coordinates
• Event Collapse in Contrast Maximization Frameworks
• ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking
• Lightweight Event-based Optical Flow Estimation via Iterative Deblurring