Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition

Abstract

Recently, there has been a lot of interest in automatically generating descriptions for an image. Most existing language-model based approaches for this task learn to generate an image description word by word in its original word order. However, for humans, it is more natural to locate the objects and their relationships first, and then elaborate on each object, describing notable attributes. We present a coarse-to-fine method that decomposes the original image description into a skeleton sentence and its attributes, and generates the skeleton sentence and attribute phrases separately. By this decomposition, our method can generate more accurate and novel descriptions than the previous state-of-the-art. Experimental results on the MS-COCO and a larger scale Stock3M datasets show that our algorithm yields consistent improvements across different evaluation metrics, especially on the SPICE metric, which has much higher correlation with human ratings than the conventional metrics. Furthermore, our algorithm can generate descriptions with varied length, benefiting from the separate control of the skeleton and attributes. This enables image description generation that better accommodates user preferences.

arXiv commentAccepted by CVPR 2017

References in corpus (14)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Image Captioning with Semantic Attention
• Show and Tell: A Neural Image Caption Generator
• Exploring Nearest Neighbor Approaches for Image Captioning
• Language Models for Image Captioning: The Quirks and What Works
• From Captions to Visual Concepts and Back
• What value do explicit high level concepts have in vision to language problems?

Cited by in corpus (7)

• CNN+CNN: Convolutional Decoders for Image Captioning
• Discriminability objective for training descriptive captions
• A Neural Compositional Paradigm for Image Captioning
• A Comprehensive Survey of Deep Learning for Image Captioning
• Phrase-based Image Captioning with Hierarchical LSTM Model
• simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions
• Searching a Raw Video Database using Natural Language Queries