Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Abstract

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

arXiv commentCVPR 2018 full oral, winner of the 2017 Visual Question Answering challenge

References in corpus (28)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ADADELTA: An Adaptive Learning Rate Method
• Deep Residual Learning for Image Recognition
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Language Modeling with Gated Convolutional Networks
• VQA: Visual Question Answering
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Image Captioning with Semantic Attention
• Identity Mappings in Deep Residual Networks
• Stacked Attention Networks for Image Question Answering
• Show and Tell: A Neural Image Caption Generator
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• From Captions to Visual Concepts and Back
• Order-Embeddings of Images and Language
• Review Networks for Caption Generation
• CIDEr: Consensus-based Image Description Evaluation
• Zero-Shot Visual Question Answering
• Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
• Boosting Image Captioning with Attributes
• What value do explicit high level concepts have in vision to language problems?
• Revisiting Visual Question Answering Baselines
• End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering

Cited by in corpus (51)

• Attention U-Net: Learning Where to Look for the Pancreas
• Learning to Count Objects in Natural Images for Visual Question Answering
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• Discriminability objective for training descriptive captions
• Stacked Cross Attention for Image-Text Matching
• Action2Vec: A Crossmodal Embedding Approach to Action Learning
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• A Neural Compositional Paradigm for Image Captioning
• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• Normalized and Geometry-Aware Self-Attention Network for Image Captioning
• Masked Non-Autoregressive Image Captioning
• Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
• A Fast and Accurate One-Stage Approach to Visual Grounding
• Attention on Attention: Architectures for Visual Question Answering (VQA)
• Language-Conditioned Imitation Learning for Robot Manipulation Tasks
• Agile Amulet: Real-Time Salient Object Detection with Contextual Attention
• Consensus-based Sequence Training for Video Captioning
• Multimodal Attribute Extraction
• Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
• Scene Graph Generation via Conditional Random Fields
• Learning by Abstraction: The Neural State Machine
• Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
• An Attempt towards Interpretable Audio-Visual Video Captioning
• Joint Image Captioning and Question Answering
• Making History Matter: History-Advantage Sequence Training for Visual Dialog
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• A Comprehensive Survey of Deep Learning for Image Captioning
• Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
• Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• DocVQA: A Dataset for VQA on Document Images
• "Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention
• Question Type Guided Attention in Visual Question Answering
• Gated Hierarchical Attention for Image Captioning
• Towards Diverse and Accurate Image Captions via Reinforcing Determinantal Point Process
• Dual Recurrent Attention Units for Visual Question Answering
• Grounded Objects and Interactions for Video Captioning
• Learning to discover and localize visual objects with open vocabulary
• Integrating Image Captioning with Rule-based Entity Masking
• Long Short-Term Memory Spatial Transformer Network
• Face-Cap: Image Captioning using Facial Expression Analysis
• Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation
• Follow the Attention: Combining Partial Pose and Object Motion for Fine-Grained Action Detection
• A Dataset and Benchmarks for Multimedia Social Analysis
• Zero-Shot Human-Object Interaction Recognition via Affordance Graphs
• ICDAR 2021 Competition on Document VisualQuestion Answering
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• Reproducibility Report for "Learning To Count Objects In Natural Images For Visual Question Answering"
• Towards Diverse Paragraph Captioning for Untrimmed Videos