Counterfactual VQA: A Cause-Effect Look at Language Bias

Abstract

VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods proposed to exclude the language prior during inference. However, they fail to disentangle the "good" language context and "bad" language bias from the whole. In this paper, we investigate how to mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal effect. Experiments demonstrate that our proposed counterfactual inference framework 1) is general to various VQA backbones and fusion strategies, 2) achieves competitive performance on the language-bias sensitive VQA-CP dataset while performs robustly on the balanced VQA v2 dataset without any augmented data. The code is available at https://github.com/yuleiniu/cfvqa.

arXiv commentAccepted by CVPR 2021

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• RUBi: Reducing Unimodal Biases in Visual Question Answering
• Interventional Few-Shot Learning
• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
• Unshuffling Data for Improved Generalization
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• Learning to Collocate Neural Modules for Image Captioning

Cited by in corpus (11)

• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering
• Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect
• Unbiased Scene Graph Generation from Biased Training
• Distilling Causal Effect of Data in Class-Incremental Learning
• Counterfactual Variable Control for Robust and Interpretable Question Answering
• Greedy Gradient Ensemble for Robust Visual Question Answering
• Learning content and context with language bias for Visual Question Answering
• Improving Weakly-supervised Object Localization via Causal Intervention
• X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering
• The Blessings of Unlabeled Background in Untrimmed Videos