High-Order Attention Models for Visual Question Answering
arXiv:1711.04323
Abstract
The quest for algorithms that enable cognitive abilities is an important part of machine learning. A common trait in many recently investigated cognitive-like tasks is that they take into account different data modalities, such as visual and textual input. In this paper we propose a novel and generally applicable form of attention mechanism that learns high-order correlations between various data modalities. We show that high-order correlations effectively direct the appropriate attention to the relevant elements in the different data modalities that are required to solve the joint task. We demonstrate the effectiveness of our high-order attention mechanism on the task of visual question answering (VQA), where we achieve state-of-the-art performance on the standard VQA dataset.
9 pages, 8 figures, NIPS 2017
References in corpus (9)
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
- Training Recurrent Answering Units with Joint Loss Minimization for VQA
- ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
- Dual Attention Networks for Multimodal Reasoning and Matching
- Creativity: Generating Diverse Questions using Variational Autoencoders