Fluency-Guided Cross-Lingual Image Captioning

Abstract

Image captioning has so far been explored mostly in English, as most available datasets are in this language. However, the application of image captioning should not be restricted by language. Only few studies have been conducted for image captioning in a cross-lingual setting. Different from these works that manually build a dataset for a target language, we aim to learn a cross-lingual captioning model fully from machine-translated sentences. To conquer the lack of fluency in the translated sentences, we propose in this paper a fluency-guided learning framework. The framework comprises a module to automatically estimate the fluency of the sentences and another module to utilize the estimated fluency scores to effectively train an image captioning model for the target language. As experiments on two bilingual (English-Chinese) datasets show, our approach improves both fluency and relevance of the generated captions in Chinese, but without using any manually written sentences from the target language.

9 pages, 2 figures, accepted as ORAL by ACM Multimedia 2017

References in corpus (7)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• YFCC100M: The New Data in Multimedia Research
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Multilingual Image Description with Neural Sequence Models
• Boosting Image Captioning with Attributes

Cited by in corpus (13)

• Evaluation of Text Generation: A Survey
• VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning
• Understanding, Categorizing and Predicting Semantic Image-Text Relations
• COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval
• CCMB: A Large-scale Chinese Cross-modal Benchmark
• Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards
• MULE: Multimodal Universal Language Embedding
• Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models
• Object-Centric Unsupervised Image Captioning
• MTVR: Multilingual Moment Retrieval in Videos