Unpaired Image Captioning via Scene Graph Alignments

Abstract

Most of current image captioning models heavily rely on paired image-caption datasets. However, getting large scale image-caption paired data is labor-intensive and time-consuming. In this paper, we present a scene graph-based approach for unpaired image captioning. Our framework comprises an image scene graph generator, a sentence scene graph generator, a scene graph encoder, and a sentence decoder. Specifically, we first train the scene graph encoder and the sentence decoder on the text modality. To align the scene graphs between images and sentences, we propose an unsupervised feature alignment method that maps the scene graph features from the image to the sentence modality. Experimental results show that our proposed model can generate quite promising results without using any image-caption training pairs, outperforming existing methods by a wide margin.

arXiv commentAccepted in ICCV 2019

References in corpus (7)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Multimodal Pivots for Image Caption Translation
• Stack-Captioning: Coarse-to-Fine Learning for Image Captioning

Cited by in corpus (3)

• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Exploring Semantic Relationships for Unpaired Image Captioning
• Recurrent Relational Memory Network for Unsupervised Image Captioning