VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows
arXiv:2108.05015
Abstract
Different from visible cameras which record intensity images frame by frame, the biologically inspired event camera produces a stream of asynchronous and sparse events with much lower latency. In practice, visible cameras can better perceive texture details and slow motion, while event cameras can be free from motion blurs and have a larger dynamic range which enables them to work well under fast motion and low illumination. Therefore, the two sensors can cooperate with each other to achieve more reliable object tracking. In this work, we propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios, and it is divided into a training and a testing subset, each of which contains 500 and 320 videos, respectively. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods by extending current single-modality trackers into dual-modality versions. More importantly, we further build a simple but effective tracking algorithm by proposing a cross-modality transformer, to achieve more effective feature fusion between visible and event data. Extensive experiments on the proposed VisEvent dataset, FE108, COESOT, and two simulated datasets (i.e., OTB-DVS and VOT-DVS), validated the effectiveness of our model. The dataset and source code have been released on: \url{https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark}.
arXiv commentAccepted by IEEE Transactions on Cybernetics
References in corpus (9)
- High-Speed Tracking with Kernelized Correlation Filters
- GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
- Event-based Moving Object Detection and Tracking
- Asynchronous, Photometric Feature Tracking using Events and Frames
- End-to-end Learning of Object Motion Estimation from Retinal Events for Event-based Object Tracking
- Asynchronous Tracking-by-Detection on Adaptive Time Surfaces for Event-based Object Tracking
- DashNet: A Hybrid Artificial and Spiking Neural Network for High-speed Object Tracking
- Siamese Keypoint Prediction Network for Visual Object Tracking
- Effects of Blur and Deblurring to Visual Object Tracking