Partially-Supervised Image Captioning

Abstract

Image captioning models are becoming increasingly successful at describing the content of images in restricted domains. However, if these models are to function in the wild - for example, as assistants for people with impaired vision - a much larger number and variety of visual concepts must be understood. To address this problem, we teach image captioning models new visual concepts from labeled images and object detection datasets. Since image labels and object classes can be interpreted as partial captions, we formulate this problem as learning from partially-specified sequence data. We then propose a novel algorithm for training sequence models, such as recurrent neural networks, on partially-specified sequences which we represent using finite state automata. In the context of image captioning, our method lifts the restriction that previously required image captioning models to be trained on paired image-sentence corpora only, or otherwise required specialized model architectures to take advantage of alternative data modalities. Applying our approach to an existing neural captioning model, we achieve state of the art results on the novel object captioning task using the COCO dataset. We further show that we can train a captioning model to describe new visual concepts from the Open Images dataset while maintaining competitive COCO evaluation scores.

arXiv commentNeurIPS 2018

References in corpus (14)

• Efficient Estimation of Word Representations in Vector Space
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• YFCC100M: The New Data in Multimedia Research
• Modeling Missing Data in Clinical Time Series with RNNs
• Language Models for Image Captioning: The Quirks and What Works
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Zero-Shot Learning -- The Good, the Bad and the Ugly
• Review Networks for Caption Generation
• Rich Image Captioning in the Wild
• Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search
• Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
• Polyglot Semantic Parsing in APIs

Cited by in corpus (4)

• Unsupervised Image Captioning
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• Captioning Images with Novel Objects via Online Vocabulary Expansion
• Partially-Supervised Novel Object Captioning Leveraging Context from Paired Data