Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering

Abstract

Recently, the Visual Question Answering (VQA) task has gained increasing attention in artificial intelligence. Existing VQA methods mainly adopt the visual attention mechanism to associate the input question with corresponding image regions for effective question answering. The free-form region based and the detection-based visual attention mechanisms are mostly investigated, with the former ones attending free-form image regions and the latter ones attending pre-specified detection-box regions. We argue that the two attention mechanisms are able to provide complementary information and should be effectively integrated to better solve the VQA problem. In this paper, we propose a novel deep neural network for VQA that integrates both attention mechanisms. Our proposed framework effectively fuses features from free-form image regions, detection boxes, and question representations via a multi-modal multiplicative feature embedding scheme to jointly attend question-related free-form image regions and detection boxes for more accurate question answering. The proposed method is extensively evaluated on two publicly available datasets, COCO-QA and VQA, and outperforms state-of-the-art approaches. Source code is available at https://github.com/lupantech/dual-mfa-vqa.

arXiv commentTo appear in AAAI 2018

References in corpus (10)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Exploring Models and Data for Image Question Answering
• Simple Baseline for Visual Question Answering
• Multimodal Residual Learning for Visual QA
• A Focused Dynamic Attention Model for Visual Question Answering
• Video Question Answering via Attribute-Augmented Attention Network Learning
• Person Search with Natural Language Description
• Learning to Answer Questions From Image Using Convolutional Neural Network