Exploring Human-like Attention Supervision in Visual Question Answering

Abstract

Attention mechanisms have been widely applied in the Visual Question Answering (VQA) task, as they help to focus on the area-of-interest of both visual and textual information. To answer the questions correctly, the model needs to selectively target different areas of an image, which suggests that an attention-based model may benefit from an explicit attention supervision. In this work, we aim to address the problem of adding attention supervision to VQA models. Since there is a lack of human attention data, we first propose a Human Attention Network (HAN) to generate human-like attention maps, training on a recently released dataset called Human ATtention Dataset (VQA-HAT). Then, we apply the pre-trained HAN on the VQA v2.0 dataset to automatically produce the human-like attention maps for all image-question pairs. The generated human-like attention map dataset for the VQA v2.0 dataset is named as Human-Like ATtention (HLAT) dataset. Finally, we apply human-like attention supervision to an attention-based VQA model. The experiments show that adding human-like supervision yields a more accurate attention together with a better performance, showing a promising future for human-like attention supervision in VQA.

Cited by in corpus (13)

• Attention in Natural Language Processing
• MirrorGAN: Learning Text-to-image Generation by Redescription
• Fine-Grained Fashion Similarity Prediction by Attribute-Specific Embedding Learning
• Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
• Counterfactual Samples Synthesizing for Robust Visual Question Answering
• Multimodal Integration of Human-Like Attention in Visual Question Answering
• Learning compositionally through attentive guidance
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• AiR: Attention with Reasoning Capability
• More Than Just Attention: Improving Cross-Modal Attentions with Contrastive Constraints for Image-Text Matching
• Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining
• Fine-Grained Fashion Similarity Learning by Attribute-Specific Embedding Network