Normalized and Geometry-Aware Self-Attention Network for Image Captioning
arXiv:2003.08897
Abstract
Self-attention (SA) network has shown profound value in image captioning. In this paper, we improve SA from two aspects to promote the performance of image captioning. First, we propose Normalized Self-Attention (NSA), a reparameterization of SA that brings the benefits of normalization inside SA. While normalization is previously only applied outside SA, we introduce a novel normalization method and demonstrate that it is both possible and beneficial to perform it on the hidden activations inside SA. Second, to compensate for the major limit of Transformer that it fails to model the geometry structure of the input objects, we propose a class of Geometry-aware Self-Attention (GSA) that extends SA to explicitly and efficiently consider the relative geometry relations between the objects in the image. To construct our image captioning model, we combine the two modules and apply it to the vanilla self-attention network. We extensively evaluate our proposals on MS-COCO image captioning dataset and superior results are achieved when comparing to state-of-the-art approaches. Further experiments on three challenging tasks, i.e. video captioning, machine translation, and visual question answering, show the generality of our methods.
arXiv commentAccepted by CVPR 2020
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Self-Attention Generative Adversarial Networks
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Context-Aware Visual Policy Network for Sequence-Level Image Captioning
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
- Multi-modality Latent Interaction Network for Visual Question Answering
- Multimodal Transformer with Multi-View Visual Representation for Image Captioning