Event-Free Moving Object Segmentation from Moving Ego Vehicle

Abstract

Moving object segmentation (MOS) in dynamic scenes is an important, challenging, but under-explored research topic for autonomous driving, especially for sequences obtained from moving ego vehicles. Most segmentation methods leverage motion cues obtained from optical flow maps. However, since these methods are often based on optical flows that are pre-computed from successive RGB frames, this neglects the temporal consideration of events occurring within the inter-frame, consequently constraining its ability to discern objects exhibiting relative staticity but genuinely in motion. To address these limitations, we propose to exploit event cameras for better video understanding, which provide rich motion cues without relying on optical flow. To foster research in this area, we first introduce a novel large-scale dataset called DSEC-MOS for moving object segmentation from moving ego vehicles, which is the first of its kind. For benchmarking, we select various mainstream methods and rigorously evaluate them on our dataset. Subsequently, we devise EmoFormer, a novel network able to exploit the event data. For this purpose, we fuse the event temporal prior with spatial semantic maps to distinguish genuinely moving objects from the static background, adding another level of dense supervision around our object of interest. Our proposed network relies only on event data for training but does not require event input during inference, making it directly comparable to frame-only methods in terms of efficiency and more widely usable in many application cases. The exhaustive comparison highlights a significant performance improvement of our method over all other methods. The source code and dataset are publicly available at: https://github.com/ZZY-Zhou/DSEC-MOS.

References in corpus (13)

• LoRA: Low-Rank Adaptation of Large Language Models
• The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
• Secrets of Event-Based Optical Flow
• Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation
• A Survey on Deep Learning Technique for Video Segmentation
• Segmenting Moving Objects via an Object-Centric Layered Representation
• Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation
• MOSE: A New Dataset for Video Object Segmentation in Complex Scenes
• E${}^2$(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition
• ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data
• CMDA: Cross-Modality Domain Adaptation for Nighttime Semantic Segmentation
• MPI-Flow: Learning Realistic Optical Flow with Multiplane Images
• Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation