Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

Abstract

While recent deep neural network models have achieved promising results on the image captioning task, they rely largely on the availability of corpora with paired image and sentence captions to describe objects in context. In this work, we propose the Deep Compositional Captioner (DCC) to address the task of generating descriptions of novel objects which are not present in paired image-sentence datasets. Our method achieves this by leveraging large object recognition datasets and external text corpora and by transferring knowledge between semantically similar concepts. Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet. In contrast, our model can compose sentences that describe novel objects and their interactions with other objects. We demonstrate our model's ability to describe novel concepts by empirically evaluating its performance on MSCOCO and show qualitative results on ImageNet images of objects for which no paired image-caption data exist. Further, we extend our approach to generate descriptions of objects in video clips. Our results show that DCC has distinct advantages over existing image and video captioning approaches for generating descriptions of new objects in context.

References in corpus (5)

• Efficient Estimation of Word Representations in Vector Space
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• What value do explicit high level concepts have in vision to language problems?

Cited by in corpus (9)

• Residual Attention Network for Image Classification
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• Dual Attention Networks for Multimodal Reasoning and Matching
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Weakly Supervised Dense Video Captioning
• Weakly-supervised learning of visual relations
• Semantic Compositional Networks for Visual Captioning
• A Semi-supervised Framework for Image Captioning
• Improving Classification by Improving Labelling: Introducing Probabilistic Multi-Label Object Interaction Recognition