simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions
arXiv:1808.08732
Abstract
The encode-decoder framework has shown recent success in image captioning. Visual attention, which is good at detailedness, and semantic attention, which is good at comprehensiveness, have been separately proposed to ground the caption on the image. In this paper, we propose the Stepwise Image-Topic Merging Network (simNet) that makes use of the two kinds of attention at the same time. At each time step when generating the caption, the decoder adaptively merges the attentive information in the extracted topics and the image according to the generated context, so that the visual information and the semantic information can be effectively combined. The proposed approach is evaluated on two benchmark datasets and reaches the state-of-the-art performances.(The code is available at https://github.com/lancopku/simNet)
arXiv commentAccepted by Conference on Empirical Methods in Natural Language Processing 2018
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- Image Captioning with Semantic Attention
- Sequence to Sequence -- Video to Text
- Visual Translation Embedding Network for Visual Relation Detection
- Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
- Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
- Deconvolution-Based Global Decoding for Neural Machine Translation