Pointing Novel Objects in Image Captioning

Abstract

Image captioning has received significant attention with remarkable improvements in recent advances. Nevertheless, images in the wild encapsulate rich knowledge and cannot be sufficiently described with models built on image-caption pairs containing only in-domain objects. In this paper, we propose to address the problem by augmenting standard deep captioning architectures with object learners. Specifically, we present Long Short-Term Memory with Pointing (LSTM-P) --- a new architecture that facilitates vocabulary expansion and produces novel objects via pointing mechanism. Technically, object learners are initially pre-trained on available object recognition data. Pointing in LSTM-P then balances the probability between generating a word through LSTM and copying a word from the recognized objects at each time step in decoder stage. Furthermore, our captioning encourages global coverage of objects in the sentence. Extensive experiments are conducted on both held-out COCO image captioning and ImageNet datasets for describing novel objects, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, we obtain an average of 60.9% in F1 score on held-out COCO~dataset.

CVPR 2019

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Image Captioning with Semantic Attention
• Decoupled Novel Object Captioner
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• Jointly Localizing and Describing Events for Dense Video Captioning
• Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
• Describing Natural Images Containing Novel Objects with Knowledge Guided Assitance