Probabilistic Semantic Retrieval for Surveillance Videos with Activity Graphs
arXiv:1712.06204 · doi:10.1109/TMM.2018.2865860
Abstract
We present a novel framework for finding complex activities matching user-described queries in cluttered surveillance videos. The wide diversity of queries coupled with unavailability of annotated activity data limits our ability to train activity models. To bridge the semantic gap we propose to let users describe an activity as a semantic graph with object attributes and inter-object relationships associated with nodes and edges, respectively. We learn node/edge-level visual predictors during training and, at test-time, propose to retrieve activity by identifying likely locations that match the semantic graph. We formulate a novel CRF based probabilistic activity localization objective that accounts for mis-detections, mis-classifications and track-losses, and outputs a likelihood score for a candidate grounded location of the query in the video. We seek groundings that maximize overall precision and recall. To handle the combinatorial search over all high-probability groundings, we propose a highest precision subgraph matching algorithm. Our method outperforms existing retrieval methods on benchmarked datasets.
arXiv comment1520-9210 (c) 2018 IEEE. This paper has been accepted by IEEE Transactions on Multimedia. Print ISSN: 1520-9210. Online ISSN: 1941-0077. Preprint link is https://ieeexplore.ieee.org/document/8438958/
References in corpus (8)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Simple Online and Realtime Tracking
- Person Re-Identification by Camera Correlation Aware Feature Augmentation
- Sequential Deep Trajectory Descriptor for Action Recognition with Three-stream CNN
- Evaluating Two-Stream CNN for Video Classification
- Fast Zero-Shot Image Tagging
- Objects2action: Classifying and localizing actions without any video example