Domain-robust VQA with diverse datasets and methods but no target labels
arXiv:2103.15974
Abstract
The observation that computer vision methods overfit to dataset specifics has inspired diverse attempts to make object recognition models robust to domain shifts. However, similar work on domain-robust visual question answering methods is very limited. Domain adaptation for VQA differs from adaptation for object recognition due to additional complexity: VQA models handle multimodal inputs, methods contain multiple steps with diverse modules resulting in complex optimization, and answer spaces in different datasets are vastly different. To tackle these challenges, we first quantify domain shifts between popular VQA datasets, in both visual and textual space. To disentangle shifts between datasets arising from different modalities, we also construct synthetic shifts in the image and question domains separately. Second, we test the robustness of different families of VQA methods (classic two-stream, transformer, and neuro-symbolic methods) to these shifts. Third, we test the applicability of existing domain adaptation methods and devise a new one to bridge VQA domain gaps, adjusted to specific VQA models. To emulate the setting of real-world generalization, we focus on unsupervised domain adaptation and the open-ended classification task formulation.
arXiv commentTo appear in CVPR 2021
References in corpus (10)
- Domain Separation Networks
- Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
- Exploring Models and Data for Image Question Answering
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
- The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
- Domain Stylization: A Strong, Simple Baseline for Synthetic to Real Image Domain Adaptation
- Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering
- Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"