VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering

Abstract

We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker. We use our dataset to analyze the similarity between human and neural attentive strategies learned by five state-of-the-art VQA models: Modular Co-Attention Network (MCAN) with either grid or region features, Pythia, Bilinear Attention Network (BAN), and the Multimodal Factorized Bilinear Pooling Network (MFB). While prior work has focused on studying the image modality, our analyses show - for the first time - that for all models, higher correlation with human attention on text is a significant predictor of VQA performance. This finding points at a potential for improving VQA performance and, at the same time, calls for further research on neural text attention mechanisms and their integration into architectures for vision and language tasks, including but potentially also beyond VQA.

arXiv commentCoNLL 2021

References in corpus (18)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Attention is not Explanation
• Multimodal Residual Learning for Visual QA
• Learning to Count Objects in Natural Images for Visual Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Bilinear Attention Networks
• Seeing with Humans: Gaze-Assisted Neural Image Captioning
• Exploring Human-like Attention Supervision in Visual Question Answering
• Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
• Revisiting Visual Question Answering Baselines
• In Defense of Grid Features for Visual Question Answering
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• AiR: Attention with Reasoning Capability

Cited by in corpus (1)

• Multimodal Integration of Human-Like Attention in Visual Question Answering