Grounded Objects and Interactions for Video Captioning
arXiv:1711.06354
Abstract
We address the problem of video captioning by grounding language generation on object interactions in the video. Existing work mostly focuses on overall scene understanding with often limited or no emphasis on object interactions to address the problem of video understanding. In this paper, we propose SINet-Caption that learns to generate captions grounded over higher-order interactions between arbitrary groups of objects for fine-grained video understanding. We discuss the challenges and benefits of such an approach. We further demonstrate state-of-the-art results on the ActivityNet Captions dataset using our model, SINet-Caption based on this approach.
arXiv admin note: substantial text overlap with arXiv:1711.06330
References in corpus (9)
- The Kinetics Human Action Video Dataset
- A simple neural network module for relational reasoning
- Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
- Visual Translation Embedding Network for Visual Relation Detection
- Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
- Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
- Modeling Relationships in Referential Expressions with Compositional Modular Networks
- Semantic Compositional Networks for Visual Captioning
- Video Captioning with Transferred Semantic Attributes