A Focused Dynamic Attention Model for Visual Question Answering

Abstract

Visual Question and Answering (VQA) problems are attracting increasing interest from multiple research disciplines. Solving VQA problems requires techniques from both computer vision for understanding the visual contents of a presented image or video, as well as the ones from natural language processing for understanding semantics of the question and generating the answers. Regarding visual content modeling, most of existing VQA methods adopt the strategy of extracting global features from the image or video, which inevitably fails in capturing fine-grained information such as spatial configuration of multiple objects. Extracting features from auto-generated regions -- as some region-based image recognition methods do -- cannot essentially address this problem and may introduce some overwhelming irrelevant features with the question. In this work, we propose a novel Focused Dynamic Attention (FDA) model to provide better aligned image content representation with proposed questions. Being aware of the key words in the question, FDA employs off-the-shelf object detector to identify important regions and fuse the information from the regions and global features via an LSTM unit. Such question-driven representations are then combined with question representation and fed into a reasoning unit for generating the answers. Extensive evaluation on a large-scale benchmark dataset, VQA, clearly demonstrate the superior performance of FDA over well-established baselines.

arXiv commentSubmitted to ECCV 2016

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• VQA: Visual Question Answering
• Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation
• Simple Baseline for Visual Question Answering
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Compositional Memory for Visual Question Answering

Cited by in corpus (37)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Multimodal Residual Learning for Visual QA
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
• Visual Question Answering: A Survey of Methods and Datasets
• Multimodal Unified Attention Networks for Vision-and-Language Interactions
• Structured Attentions for Visual Question Answering
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Task-driven Visual Saliency and Attention-based Visual Question Answering
• An Analysis of Visual Question Answering Algorithms
• Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering
• VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
• ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
• DualNet: Domain-Invariant Network for Visual Question Answering
• VQABQ: Visual Question Answering by Basic Questions
• Compact Trilinear Interaction for Visual Question Answering
• Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• Learning Visual Question Answering by Bootstrapping Hard Attention
• A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
• Interpretable Visual Question Answering by Reasoning on Dependency Trees
• Figure Captioning with Reasoning and Sequence-Level Training
• Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• Tutorial on Answering Questions about Images with Deep Learning
• Linguistically Driven Graph Capsule Network for Visual Question Reasoning
• Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
• Transfer Learning in Visual and Relational Reasoning
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Improved RAMEN: Towards Domain Generalization for Visual Question Answering