Task-driven Visual Saliency and Attention-based Visual Question Answering

Abstract

Visual question answering (VQA) has witnessed great progress since May, 2015 as a classic problem unifying visual and textual data into a system. Many enlightening VQA works explore deep into the image and question encodings and fusing methods, of which attention is the most effective and infusive mechanism. Current attention based methods focus on adequate fusion of visual and textual features, but lack the attention to where people focus to ask questions about the image. Traditional attention based methods attach a single value to the feature at each spatial location, which losses many useful information. To remedy these problems, we propose a general method to perform saliency-like pre-selection on overlapped region features by the interrelation of bidirectional LSTM (BiLSTM), and use a novel element-wise multiplication based attention method to capture more competent correlation information between visual and textual features. We conduct experiments on the large-scale COCO-VQA dataset and analyze the effectiveness of our model demonstrated by strong empirical results.

8 pages, 3 figures

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Visualizing and Understanding Recurrent Networks
• Dynamic Memory Networks for Visual and Textual Question Answering
• Simple Baseline for Visual Question Answering
• Multimodal Residual Learning for Visual QA
• Stacked Attention Networks for Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• A Focused Dynamic Attention Model for Visual Question Answering
• Learning to Compose Neural Networks for Question Answering
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Visual7W: Grounded Question Answering in Images
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• Revisiting Visual Question Answering Baselines
• Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoders

Cited by in corpus (3)

• Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network
• A Deeper Look at Salient Object Detection: Bi-stream Network with a Small Training Dataset
• Recursive Multi-model Complementary Deep Fusion forRobust Salient Object Detection via Parallel Sub Networks