A Semi-supervised Framework for Image Captioning

Abstract

State-of-the-art approaches for image captioning require supervised training data consisting of captions with paired image data. These methods are typically unable to use unsupervised data such as textual data with no corresponding images, which is a much more abundant commodity. We here propose a novel way of using such textual data by artificially generating missing visual information. We evaluate this learning approach on a newly designed model that detects visual concepts present in an image and feed them to a reviewer-decoder architecture with an attention mechanism. Unlike previous approaches that encode visual concepts using word embeddings, we instead suggest using regional image features which capture more intrinsic information. The main benefit of this architecture is that it synthesizes meaningful thought vectors that capture salient image properties and then applies a soft attentive decoder to decode the thought vectors and generate image captions. We evaluate our model on both Microsoft COCO and Flickr30K datasets and demonstrate that this model combined with our semi-supervised learning method can largely improve performance and help the model to generate more accurate and diverse captions.

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Explain Images with Multimodal Recurrent Neural Networks
• Image Captioning with Semantic Attention
• Show and Tell: A Neural Image Caption Generator
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• From Captions to Visual Concepts and Back
• Review Networks for Caption Generation
• Generating Images from Captions with Attention
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• CIDEr: Consensus-based Image Description Evaluation
• Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
• ConceptLearner: Discovering Visual Concepts from Weakly Labeled Image Collections

Cited by in corpus (3)

• More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval
• Unsupervised Image Captioning
• Approximate Distribution Matching for Sequence-to-Sequence Learning