Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards

Abstract

Generating image descriptions in different languages is essential to satisfy users worldwide. However, it is prohibitively expensive to collect large-scale paired image-caption dataset for every target language which is critical for training descent image captioning models. Previous works tackle the unpaired cross-lingual image captioning problem through a pivot language, which is with the help of paired image-caption data in the pivot language and pivot-to-target machine translation models. However, such language-pivoted approach suffers from inaccuracy brought by the pivot-to-target translation, including disfluency and visual irrelevancy errors. In this paper, we propose to generate cross-lingual image captions with self-supervised rewards in the reinforcement learning framework to alleviate these two types of errors. We employ self-supervision from mono-lingual corpus in the target language to provide fluency reward, and propose a multi-level visual semantic matching model to provide both sentence-level and concept-level visual relevancy rewards. We conduct extensive experiments for unpaired cross-lingual image captioning in both English and Chinese respectively on two widely used image caption corpora. The proposed approach achieves significant performance improvement over state-of-the-art methods.

arXiv commentAccepted by ACMMM 2019

References in corpus (17)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Convolutional Neural Networks for Sentence Classification
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Image Captioning with Semantic Attention
• AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Fluency-Guided Cross-Lingual Image Captioning
• Multimodal Pivots for Image Caption Translation
• Guiding Long-Short Term Memory for Image Caption Generation
• Discriminability objective for training descriptive captions
• Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
• Using Artificial Tokens to Control Languages for Multilingual Image Caption Generation
• Unsupervised Image Captioning
• Unpaired Image Captioning by Language Pivoting
• Depth Structure Preserving Scene Image Generation
• Unsupervised Bilingual Lexicon Induction from Mono-lingual Multimodal Data