Image Captioning with Unseen Objects
arXiv:1908.00047
Abstract
Image caption generation is a long standing and challenging problem at the intersection of computer vision and natural language processing. A number of recently proposed approaches utilize a fully supervised object recognition model within the captioning approach. Such models, however, tend to generate sentences which only consist of objects predicted by the recognition models, excluding instances of the classes without labelled training examples. In this paper, we propose a new challenging scenario that targets the image captioning problem in a fully zero-shot learning setting, where the goal is to be able to generate captions of test images containing objects that are not seen during training. The proposed approach jointly uses a novel zero-shot object detection model and a template-based sentence generator. Our experiments show promising results on the COCO dataset.
arXiv commentTo appear in British Machine Vision Conference (BMVC) 2019
References in corpus (12)
- Efficient Estimation of Word Representations in Vector Space
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Distributed Representations of Words and Phrases and their Compositionality
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- R-FCN: Object Detection via Region-based Fully Convolutional Networks
- DSSD : Deconvolutional Single Shot Detector
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
- LSDA: Large Scale Detection Through Adaptation
- Zero Shot Recognition with Unreliable Attributes
- Decoupled Novel Object Captioner
- Polarity Loss for Zero-shot Object Detection