Generating Descriptions with Grounded and Co-Referenced People

Abstract

Learning how to generate descriptions of images or videos received major interest both in the Computer Vision and Natural Language Processing communities. While a few works have proposed to learn a grounding during the generation process in an unsupervised way (via an attention mechanism), it remains unclear how good the quality of the grounding is and whether it benefits the description quality. In this work we propose a movie description model which learns to generate description and jointly ground (localize) the mentioned characters as well as do visual co-reference resolution between pairs of consecutive sentences/clips. We also propose to use weak localization supervision through character mentions provided in movie descriptions to learn the character grounding. At training time, we first learn how to localize characters by relating their visual appearance to mentions in the descriptions via a semi-supervised approach. We then provide this (noisy) supervision into our description model which greatly improves its performance. Our proposed description model improves over prior work w.r.t. generated description quality and additionally provides grounding and local co-reference resolution. We evaluate it on the MPII Movie Description dataset using automatic and human evaluation measures and using our newly collected grounding and co-reference data for characters.

Accepted to CVPR 2017

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Image Captioning with Semantic Attention
• Naive-Deep Face Recognition: Touching the Limit of LFW Benchmark or Not?
• Show and Tell: A Neural Image Caption Generator
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Visual Storytelling
• Attention Correctness in Neural Image Captioning
• Modeling Context in Referring Expressions
• A Dataset for Movie Description
• Deeply learned face representations are sparse, selective, and robust
• Fine-grained Activity Recognition with Holistic and Pose based Features
• Beyond Caption To Narrative: Video Captioning With Multiple Sentences

Cited by in corpus (5)

• Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
• General Partial Label Learning via Dual Bipartite Graph Autoencoder
• Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
• Robust Character Labeling in Movie Videos: Data Resources and Self-supervised Feature Adaptation
• Grounded Video Description