Cross-Dataset Adaptation for Visual Question Answering

Abstract

We investigate the problem of cross-dataset adaptation for visual question answering (Visual QA). Our goal is to train a Visual QA model on a source dataset but apply it to another target one. Analogous to domain adaptation for visual recognition, this setting is appealing when the target dataset does not have a sufficient amount of labeled data to learn an "in-domain" model. The key challenge is that the two datasets are constructed differently, resulting in the cross-dataset mismatch on images, questions, or answers. We overcome this difficulty by proposing a novel domain adaptation algorithm. Our method reduces the difference in statistical distributions by transforming the feature representation of the data in the target dataset. Moreover, it maximizes the likelihood of answering questions (in the target dataset) correctly using the Visual QA model trained on the source dataset. We empirically studied the effectiveness of the proposed approach on adapting among several popular Visual QA datasets. We show that the proposed method improves over baselines where there is no adaptation and several other adaptation methods. We both quantitatively and qualitatively analyze when the adaptation can be mostly effective.

arXiv commentAccepted at CVPR 2018

References in corpus (9)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Adversarial Discriminative Domain Adaptation
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• A Testbed for Cross-Dataset Analysis
• Survey of Visual Question Answering: Datasets and Techniques
• An Analysis of Visual Question Answering Algorithms
• A Diagram Is Worth A Dozen Images