Hierarchical Question-Image Co-Attention for Visual Question Answering

Abstract

A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.

11 pages, 7 figures, 3 tables in 2016 Conference on Neural Information Processing Systems (NIPS)

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Multimodal Residual Learning for Visual QA
• A Focused Dynamic Attention Model for Visual Question Answering

Cited by in corpus (203)

• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• A General Survey on Attention Mechanisms in Deep Learning
• Attention in Natural Language Processing
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Dynamic Coattention Networks For Question Answering
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Grad-CAM: Why did you say that?
• Modulating early visual processing by language
• Changer: Feature Interaction is What You Need for Change Detection
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Intra- and Inter-epoch Temporal Context Network (IITNet) Using Sub-epoch Features for Automatic Sleep Scoring on Raw Single-channel EEG
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• RUBi: Reducing Unimodal Biases in Visual Question Answering
• Learning to Count Objects in Natural Images for Visual Question Answering
• A Survey on Aspect-Based Sentiment Classification
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• Compositional Attention Networks for Machine Reasoning
• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis
• Visual Reference Resolution using Attention Memory for Visual Dialog
• DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
• Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
• Representation Learning for Natural Language Processing
• Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
• Bilinear Attention Networks
• Focal Visual-Text Attention for Visual Question Answering
• Spatio-Temporal Self-Attention Network for Video Saliency Prediction
• Global-Locally Self-Attentive Dialogue State Tracker
• Fine-grained Visual-textual Representation Learning
• Explainable Identification of Dementia from Transcripts using Transformer Networks
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
• Multi-modal Deep Analysis for Multimedia
• Vision-to-Language Tasks Based on Attributes and Attention Mechanism
• Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
• C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Dual Attention Networks for Multimodal Reasoning and Matching
• ShapeWorld - A new test methodology for multimodal language understanding
• High-Order Attention Models for Visual Question Answering
• Visual Question Answering: A Survey of Methods and Datasets
• Combination of Multiple Global Descriptors for Image Retrieval
• Visual Question Answering for Cultural Heritage
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• Neural ranking models for document retrieval
• Beyond Visual Semantics: Exploring the Role of Scene Text in Image Understanding
• FVQA: Fact-based Visual Question Answering
• Identity-Aware Textual-Visual Matching with Latent Co-attention
• OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese
• Kronecker Attention Networks
• Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions
• Creativity: Generating Diverse Questions using Variational Autoencoders
• End-to-end optimization of goal-driven and visually grounded dialogue systems
• It Takes Two to Tango: Towards Theory of AI's Mind
• A Multi-Modal Chinese Poetry Generation Model
• A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method
• Revisiting Visual Question Answering Baselines
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• Visual Question Generation as Dual Task of Visual Question Answering
• DVQA: Understanding Data Visualizations via Question Answering
• To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression
• CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification
• Structured Attentions for Visual Question Answering
• ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• SRC-Net: Bi-Temporal Spatial Relationship Concerned Network for Change Detection
• Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Survey of Visual Question Answering: Datasets and Techniques
• Task-driven Visual Saliency and Attention-based Visual Question Answering
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
• ORDNet: Capturing Omni-Range Dependencies for Scene Parsing
• Question-Guided Hybrid Convolution for Visual Question Answering
• An Analysis of Visual Question Answering Algorithms
• Multimodal Integration of Human-Like Attention in Visual Question Answering
• Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
• Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
• Passage Retrieval for Outside-Knowledge Visual Question Answering
• Agile Amulet: Real-Time Salient Object Detection with Contextual Attention
• Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
• GuessWhat?! Visual object discovery through multi-modal dialogue
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching
• Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering
• Where is the Model Looking At?--Concentrate and Explain the Network Attention
• Active Learning for Visual Question Answering: An Empirical Study
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System
• Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
• Learning Convolutional Text Representations for Visual Question Answering
• VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
• Intermediate Deep Feature Compression: the Next Battlefield of Intelligent Sensing
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
• ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• Robustness Analysis of Visual QA Models by Basic Questions
• Visual Question Answering with Memory-Augmented Networks
• Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product
• Speech-Based Visual Question Answering
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• SBNet: Sparse Blocks Network for Fast Inference
• Implicit Motion-Compensated Network for Unsupervised Video Object Segmentation
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• Factor Graph Attention
• Reciprocal Attention Fusion for Visual Question Answering
• Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
• Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
• Towards Interpretable R-CNN by Unfolding Latent Structures
• ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
• The Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA)
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network
• A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
• Interpretable Visual Question Answering by Reasoning on Dependency Trees
• DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
• Do Explanations make VQA Models more Predictable to a Human?
• Faithful Multimodal Explanation for Visual Question Answering
• Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration
• Video Highlight Prediction Using Audience Chat Reactions
• Hierarchical Question Answering for Long Documents
• Question Type Guided Attention in Visual Question Answering
• Two-stream Collaborative Learning with Spatial-Temporal Attention for Video Classification
• TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
• Dense Recurrent Neural Networks for Scene Labeling
• Textually Enriched Neural Module Networks for Visual Question Answering
• ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations
• Improving Visual Question Answering by Referring to Generated Paragraph Captions
• Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization
• Zero-Shot Transfer VQA Dataset
• Dual Recurrent Attention Units for Visual Question Answering
• Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding
• Attend in groups: a weakly-supervised deep learning framework for learning from web data
• TallyQA: Answering Complex Counting Questions
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
• AlignVE: Visual Entailment Recognition Based on Alignment Relations
• Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication
• Learning to Disambiguate by Asking Discriminative Questions
• Multimodal Differential Network for Visual Question Generation
• See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks
• QSAN: A Quantum-probability based Signed Attention Network for Explainable False Information Detection
• Multilevel Language and Vision Integration for Text-to-Clip Retrieval
• Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network
• An Entropy Clustering Approach for Assessing Visual Question Difficulty
• ORD: Object Relationship Discovery for Visual Dialogue Generation
• Self-Attentive Neural Collaborative Filtering
• Supervised Visual Attention for Simultaneous Multimodal Machine Translation
• Microsoft AI Challenge India 2018: Learning to Rank Passages for Web Question Answering with Deep Attention Networks
• Modularity Matters: Learning Invariant Relational Reasoning Tasks
• Relationship-Embedded Representation Learning for Grounding Referring Expressions
• Spatial Memory for Context Reasoning in Object Detection
• Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks
• How to Become Instagram Famous: Post Popularity Prediction with Dual-Attention
• Question Guided Modular Routing Networks for Visual Question Answering
• Cross-Dataset Adaptation for Visual Question Answering
• Think Visually: Question Answering through Virtual Imagery
• Linguistically Driven Graph Capsule Network for Visual Question Reasoning
• Cross-Modal Generative Augmentation for Visual Question Answering
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• Where to Look and How to Describe: Fashion Image Retrieval with an Attentional Heterogeneous Bilinear Network
• Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings
• Trying Bilinear Pooling in Video-QA
• Tutorial on Answering Questions about Images with Deep Learning
• The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
• Learning Question-Guided Video Representation for Multi-Turn Video Question Answering
• Attention Guided Semantic Relationship Parsing for Visual Question Answering
• Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
• Holistic Multi-modal Memory Network for Movie Question Answering
• Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networks
• Scene Parsing via Dense Recurrent Neural Networks with Attentional Selection
• Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• Towards Solving Multimodal Comprehension
• Visual Explanations from Hadamard Product in Multimodal Deep Networks
• Multimodal Data Fusion based on the Global Workspace Theory
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• DeepSIC: Deep Semantic Image Compression
• Visual Question Answering based on Local-Scene-Aware Referring Expression Generation
• Conditionally Learn to Pay Attention for Sequential Visual Task
• After All, Only The Last Neuron Matters: Comparing Multi-modal Fusion Functions for Scene Graph Generation
• Quantifying the Suicidal Tendency on Social Media: A Survey
• Attention-Mechanism-based Tracking Method for Intelligent Internet of Vehicles
• Understanding in Artificial Intelligence
• Combining exogenous and endogenous signals with a semi-supervised co-attention network for early detection of COVID-19 fake tweets
• Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models
• Customized Image Narrative Generation via Interactive Visual Question Generation and Answering
• Modulated Self-attention Convolutional Network for VQA
• Reasoning Over History: Context Aware Visual Dialog
• Uncertainty based Class Activation Maps for Visual Question Answering
• CS-VQA: Visual Question Answering with Compressively Sensed Images
• Triple Attention Network architecture for MovieQA
• Recurrent and Contextual Models for Visual Question Answering
• Reason induced visual attention for explainable autonomous driving