Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
arXiv:2010.10802
Abstract
Many recent datasets contain a variety of different data modalities, for instance, image, question, and answer data in visual question answering (VQA). When training deep net classifiers on those multi-modal datasets, the modalities get exploited at different scales, i.e., some modalities can more easily contribute to the classification results than others. This is suboptimal because the classifier is inherently biased towards a subset of the modalities. To alleviate this shortcoming, we propose a novel regularization term based on the functional entropy. Intuitively, this term encourages to balance the contribution of each modality to the classification result. However, regularization with the functional entropy is challenging. To address this, we develop a method based on the log-Sobolev inequality, which bounds the functional entropy with the functional-Fisher-information. Intuitively, this maximizes the amount of information that the modalities contribute. On the two challenging multi-modal datasets VQA-CPv2 and SocialIQ, we obtain state-of-the-art results while more uniformly exploiting the modalities. In addition, we demonstrate the efficacy of our method on Colored MNIST.
arXiv commentNeurIPS 2020
References in corpus (8)
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Opening the Black Box of Deep Neural Networks via Information
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Self-Critical Reasoning for Robust Visual Question Answering
- Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
- Learning Not to Learn: Training Deep Neural Networks with Biased Data
- Factor Graph Attention
- Visual Grounding Methods for VQA are Working for the Wrong Reasons!