Attention on Attention for Image Captioning
arXiv:1908.06954
Abstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Accepted to ICCV 2019 (Oral)
References in corpus (5)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Hadamard Product for Low-rank Bilinear Pooling
- C-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object Detection
Cited by in corpus (7)
- Unifying Multimodal Transformer for Bi-directional Image and Text Generation
- TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
- A Survey on Natural Language Video Localization
- Macroscopic Control of Text Generation for Image Captioning
- TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning
- Group-based Distinctive Image Captioning with Memory Attention
- Recurrent Relational Memory Network for Unsupervised Image Captioning