Using Artificial Tokens to Control Languages for Multilingual Image Caption Generation

Abstract

Recent work in computer vision has yielded impressive results in automatically describing images with natural language. Most of these systems generate captions in a sin- gle language, requiring multiple language-specific models to build a multilingual captioning system. We propose a very simple technique to build a single unified model across languages, using artificial tokens to control the language, making the captioning system more compact. We evaluate our approach on generating English and Japanese captions, and show that a typical neural captioning architecture is capable of learning a single model that can switch between two different languages.

This work appears as an Extended Abstract at the 2017 CVPR Language and Vision Workshop

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• Multimodal Pivots for Image Caption Translation
• Multilingual Image Description with Neural Sequence Models

Cited by in corpus (6)

• VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
• Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning
• COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval
• Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events
• Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards
• Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models