Spatio-Temporal Attention Models for Grounded Video Captioning
arXiv:1610.04997
Abstract
Automatic video captioning is challenging due to the complex interactions in dynamic real scenes. A comprehensive system would ultimately localize and track the objects, actions and interactions present in a video and generate a description that relies on temporal localization in order to ground the visual concepts. However, most existing automatic video captioning systems map from raw video data to high level textual description, bypassing localization and recognition, thus discarding potentially valuable information for content localization and generalization. In this work we present an automatic video captioning model that combines spatio-temporal attention and image classification by means of deep neural network structures based on long short-term memory. The resulting system is demonstrated to produce state-of-the-art results in the standard YouTube captioning benchmark while also offering the advantage of localizing the visual concepts (subjects, verbs, objects), with no grounding supervision, over space and time.
To appear in Asian Conference on Computer Vision (ACCV), Taipei, Taiwan, 2016
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Learning to Execute
- Sequence to Sequence -- Video to Text
- Describing Videos by Exploiting Temporal Structure
- Delving Deeper into Convolutional Networks for Learning Video Representations
- A Multi-scale Multiple Instance Video Description Network
- Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
- Jointly Modeling Embedding and Translation to Bridge Video and Language