Unsupervised Image Captioning

Abstract

Deep neural networks have achieved great successes on the image captioning task. However, most of the existing models depend heavily on paired image-sentence datasets, which are very expensive to acquire. In this paper, we make the first attempt to train an image captioning model in an unsupervised manner. Instead of relying on manually labeled image-sentence pairs, our proposed model merely requires an image set, a sentence corpus, and an existing visual concept detector. The sentence corpus is used to teach the captioning model how to generate plausible sentences. Meanwhile, the knowledge in the visual concept detector is distilled into the captioning model to guide the model to recognize the visual concepts in an image. In order to further encourage the generated captions to be semantically consistent with the image, the image and caption are projected into a common latent space so that they can reconstruct each other. Given that the existing sentence corpora are mainly designed for linguistic research and are thus with little reference to image contents, we crawl a large-scale image description corpus of two million natural sentences to facilitate the unsupervised image captioning scenario. Experimental results show that our proposed model is able to produce quite promising results without any caption annotations.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Microsoft COCO Captions: Data Collection and Evaluation Server
• MaskGAN: Better Text Generation via Filling in the______
• Image Captioning with Semantic Attention
• Rich Image Captioning in the Wild
• Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
• Partially-Supervised Image Captioning
• Learning to Guide Decoding for Image Captioning
• A Semi-supervised Framework for Image Captioning

Cited by in corpus (5)

• A Fast and Accurate One-Stage Approach to Visual Grounding
• Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards
• Adversarial reconstruction for Multi-modal Machine Translation
• End-to-End Learning Using Cycle Consistency for Image-to-Caption Transformations
• Experimenting with Self-Supervision using Rotation Prediction for Image Captioning