Learning Where to Focus for Efficient Video Object Detection
arXiv:1911.05253
Abstract
Transferring existing image-based detectors to the video is non-trivial since the quality of frames is always deteriorated by part occlusion, rare pose, and motion blur. Previous approaches exploit to propagate and aggregate features across video frames by using optical flow-warping. However, directly applying image-level optical flow onto the high-level features might not establish accurate spatial correspondences. Therefore, a novel module called Learnable Spatio-Temporal Sampling (LSTS) has been proposed to learn semantic-level correspondences among adjacent frame features accurately. The sampled locations are first randomly initialized, then updated iteratively to find better spatial correspondences guided by detection supervision progressively. Besides, Sparsely Recursive Feature Updating (SRFU) module and Dense Feature Aggregation (DFA) module are also introduced to model temporal relations and enhance per-frame features, respectively. Without bells and whistles, the proposed method achieves state-of-the-art performance on the ImageNet VID dataset with less computational complexity and real-time speed. Code will be made available at https://github.com/jiangzhengkai/LSTS.
arXiv commentAccepted to ECCV 2020
References in corpus (10)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Two-Stream Convolutional Networks for Action Recognition in Videos
- R-FCN: Object Detection via Region-based Fully Convolutional Networks
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Recurrent Models of Visual Attention
- Action Recognition using Visual Attention
- Seq-NMS for Video Object Detection
- Impression Network for Video Object Detection