Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark
arXiv:2103.16746
Abstract
Tracking by natural language specification is a new rising research topic that aims at locating the target object in the video sequence based on its language description. Compared with traditional bounding box (BBox) based tracking, this setting guides object tracking with high-level semantic information, addresses the ambiguity of BBox, and links local and global search organically together. Those benefits may bring more flexible, robust and accurate tracking performance in practical scenarios. However, existing natural language initialized trackers are developed and compared on benchmark datasets proposed for tracking-by-BBox, which can't reflect the true power of tracking-by-language. In this work, we propose a new benchmark specifically dedicated to the tracking-by-language, including a large scale dataset, strong and diverse baseline methods. Specifically, we collect 2k video sequences (contains a total of 1,244,340 frames, 663 words) and split 1300/700 for the train/testing respectively. We densely annotate one sentence in English and corresponding bounding boxes of the target object for each video. We also introduce two new challenges into TNL2K for the object tracking task, i.e., adversarial samples and modality switch. A strong baseline method based on an adaptive local-global-search scheme is proposed for future works to compare. We believe this benchmark will greatly boost related researches on natural language guided tracking.
arXiv commentAccepted by CVPR 2021
References in corpus (11)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- YOLOv3: An Incremental Improvement
- High-Speed Tracking with Kernelized Correlation Filters
- GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
- Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
- A Novel Performance Evaluation Methodology for Single-Target Trackers
- Deep Learning for Visual Tracking: A Comprehensive Survey
- PTB-TIR: A Thermal Infrared Pedestrian Tracking Benchmark
- 'Skimming-Perusal' Tracking: A Framework for Real-Time and Robust Long-term Tracking
- Describe and Attend to Track: Learning Natural Language guided Structural Representation and Visual Attention for Object Tracking
- Siamese Keypoint Prediction Network for Visual Object Tracking