LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering
arXiv:2105.14300 · doi:10.1145/3404835.3462981
Abstract
Most existing Visual Question Answering (VQA) systems tend to overly rely on language bias and hence fail to reason from the visual clue. To address this issue, we propose a novel Language-Prior Feedback (LPF) objective function, to re-balance the proportion of each answer's loss value in the total VQA loss. The LPF firstly calculates a modulating factor to determine the language bias using a question-only branch. Then, the LPF assigns a self-adaptive weight to each training sample in the training process. With this reweighting mechanism, the LPF ensures that the total VQA loss can be reshaped to a more balanced form. By this means, the samples that require certain visual information to predict will be efficiently used during training. Our method is simple to implement, model-agnostic, and end-to-end trainable. We conduct extensive experiments and the results show that the LPF (1) brings a significant improvement over various VQA models, (2) achieves competitive performance on the bias-sensitive VQA-CP v2 benchmark.
arXiv commentAccepted by ACM SIGIR 2021
References in corpus (8)
- SSD: Single Shot MultiBox Detector
- VQA: Visual Question Answering
- RUBi: Reducing Unimodal Biases in Visual Question Answering
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- Self-Critical Reasoning for Robust Visual Question Answering
- On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
- Citation concept analysis (CCA) - A new form of citation analysis revealing the usefulness of concepts for other researchers illustrated by two exemplary case studies including classic books by Thomas S. Kuhn and Karl R. Popper
- Counterfactual VQA: A Cause-Effect Look at Language Bias