SODFormer: Streaming Object Detection with Transformer Using Events and Frames
arXiv:2308.04047 · doi:10.1109/TPAMI.2023.3298925
Abstract
DAVIS camera, streaming two complementary sensing modalities of asynchronous events and frames, has gradually been used to address major object detection challenges (e.g., fast motion blur and low-light). However, how to effectively leverage rich temporal cues and fuse two heterogeneous visual streams remains a challenging endeavor. To address this challenge, we propose a novel streaming object detector with Transformer, namely SODFormer, which first integrates events and frames to continuously detect objects in an asynchronous manner. Technically, we first build a large-scale multimodal neuromorphic object detection dataset (i.e., PKU-DAVIS-SOD) over 1080.1k manual labels. Then, we design a spatiotemporal Transformer architecture to detect objects via an end-to-end sequence prediction problem, where the novel temporal Transformer module leverages rich temporal cues from two visual streams to improve the detection performance. Finally, an asynchronous attention-based fusion module is proposed to integrate two heterogeneous sensing modalities and take complementary advantages from each end, which can be queried at any time to locate objects and break through the limited output frequency from synchronized frame-based fusion strategies. The results show that the proposed SODFormer outperforms four state-of-the-art methods and our eight baselines by a significant margin. We also show that our unifying framework works well even in cases where the conventional frame-based camera fails, e.g., high-speed motion and low-light conditions. Our dataset and code can be available at https://github.com/dianzl/SODFormer.
arXiv comment18 pages, 15 figures, in IEEE Transactions on Pattern Analysis and Machine Intelligence
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Neural Machine Translation by Jointly Learning to Align and Translate
- Event-based Vision: A Survey
- Deformable DETR: Deformable Transformers for End-to-End Object Detection
- MDLatLRR: A novel decomposition method for infrared and visible image fusion
- Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars
- Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios
- Understanding and Improving Layer Normalization
- Learning to Detect Objects with a 1 Megapixel Event Camera
- VECtor: A Versatile Event-Centric Benchmark for Multi-Sensor SLAM
- TJU-DHD: A Diverse High-Resolution Dataset for Object Detection
- A Large Scale Event-based Detection Dataset for Automotive
- Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation
- Real-Time Face & Eye Tracking and Blink Detection using Event Cameras
- StreamYOLO: Real-time Object Detection for Streaming Perception