Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech
arXiv:1805.12589
Abstract
Image captioning is an ambiguous problem, with many suitable captions for an image. To address ambiguity, beam search is the de facto method for sampling multiple captions. However, beam search is computationally expensive and known to produce generic captions. To address this concern, some variational auto-encoder (VAE) and generative adversarial net (GAN) based methods have been proposed. Though diverse, GAN and VAE are less accurate. In this paper, we first predict a meaningful summary of the image, then generate the caption based on that summary. We use part-of-speech as summaries, since our summary should drive caption generation. We achieve the trifecta: (1) High accuracy for the diverse captions as evaluated by standard captioning metrics and user studies; (2) Faster computation of diverse captions compared to beam search and diverse beam search; and (3) High diversity as evaluated by counting novel sentences, distinct n-grams and mutual overlap (i.e., mBleu-4) scores.
arXiv comment12 pages with references and appendix. To appear CVPR'19
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Categorical Reparameterization with Gumbel-Softmax
- Convolutional Sequence to Sequence Learning
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- Exploring Nearest Neighbor Approaches for Image Captioning
- CNN+CNN: Convolutional Decoders for Image Captioning
- Discriminability objective for training descriptive captions
- Generating Diverse and Accurate Visual Captions by Comparative Adversarial Learning