Self-Critical Reasoning for Robust Visual Question Answering

Abstract

Visual Question Answering (VQA) deep-learning systems tend to capture superficial statistical correlations in the training data because of strong language priors and fail to generalize to test data with a significantly different question-answer (QA) distribution. To address this issue, we introduce a self-critical training objective that ensures that visual explanations of correct answers match the most influential image regions more than other competitive answer candidates. The influential regions are either determined from human visual/textual explanations or automatically from just significant words in the question and answer. We evaluate our approach on the VQA generalization task using the VQA-CP dataset, achieving a new state-of-the-art i.e., 49.5% using textual explanations and 48.5% using automatically annotated regions.

arXiv commentIn NeurIPS 2019

Cited by in corpus (24)

• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
• LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering
• Counterfactual Samples Synthesizing for Robust Visual Question Answering
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
• Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
• Learning from Lexical Perturbations for Consistent Visual Question Answering
• Language bias in Visual Question Answering: A Survey and Taxonomy
• Estimating semantic structure for the VQA answer space
• Faithful Multimodal Explanation for Visual Question Answering
• MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
• Loss re-scaling VQA: Revisiting the LanguagePrior Problem from a Class-imbalance View
• Generating Question Relevant Captions to Aid Visual Question Answering
• Visual Grounding Methods for VQA are Working for the Wrong Reasons!
• Greedy Gradient Ensemble for Robust Visual Question Answering
• Overcoming Statistical Shortcuts for Open-ended Visual Counting
• Learning content and context with language bias for Visual Question Answering
• AiR: Attention with Reasoning Capability
• Cross-Modal Generative Augmentation for Visual Question Answering
• Separating Skills and Concepts for Novel Visual Question Answering
• Explaining the Road Not Taken
• On the Significance of Question Encoder Sequence Model in the Out-of-Distribution Performance in Visual Question Answering
• Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering
• X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering