Decoupled Novel Object Captioner

Abstract

Image captioning is a challenging task where the machine automatically describes an image by sentences or phrases. It often requires a large number of paired image-sentence annotations for training. However, a pre-trained captioning model can hardly be applied to a new domain in which some novel object categories exist, i.e., the objects and their description words are unseen during model training. To correctly caption the novel object, it requires professional human workers to annotate the images by sentences with the novel words. It is labor expensive and thus limits its usage in real-world applications. In this paper, we introduce the zero-shot novel object captioning task where the machine generates descriptions without extra sentences about the novel object. To tackle the challenging problem, we propose a Decoupled Novel Object Captioner (DNOC) framework that can fully decouple the language sequence model from the object descriptions. DNOC has two components. 1) A Sequence Model with the Placeholder (SM-P) generates a sentence containing placeholders. The placeholder represents an unseen novel object. Thus, the sequence model can be decoupled from the novel object descriptions. 2) A key-value object memory built upon the freely available detection model, contains the visual information and the corresponding word for each object. The SM-P will generate a query to retrieve the words from the object memory. The placeholder will then be filled with the correct word, resulting in a caption with novel object descriptions. The experimental results on the held-out MSCOCO dataset demonstrate the ability of DNOC in describing novel concepts in the zero-shot novel object captioning task.

arXiv commentAccepted to ACM MM 2018

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: A system for large-scale machine learning
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Matching Networks for One Shot Learning
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• One-shot Learning with Memory-Augmented Neural Networks
• Image Captioning with Semantic Attention
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

Cited by in corpus (15)

• nocaps: novel object captioning at scale
• Cascaded Revision Network for Novel Object Captioning
• Diverse Image Captioning with Context-Object Split Latent Spaces
• ClawCraneNet: Leveraging Object-level Relation for Text-based Video Segmentation
• VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
• Learning to Select: A Fully Attentive Approach for Novel Object Captioning
• Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
• Pointing Novel Objects in Image Captioning
• ECOL-R: Encouraging Copying in Novel Object Captioning with Reinforcement Learning
• Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models
• Captioning Images with Novel Objects via Online Vocabulary Expansion
• Image Captioning with Unseen Objects
• Intention Oriented Image Captions with Guiding Objects
• Saying the Unseen: Video Descriptions via Dialog Agents
• Describing Unseen Videos via Multi-Modal Cooperative Dialog Agents