Passage Retrieval for Outside-Knowledge Visual Question Answering
arXiv:2105.03938 · doi:10.1145/3404835.3462987
Abstract
In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowledge, which in our case we define to be a large unstructured passage collection. We first conduct sparse retrieval with BM25 and study expanding the question with object names and image captions. We verify that visual clues play an important role and captions tend to be more informative than object names in sparse retrieval. We then construct a dual-encoder dense retriever, with the query encoder being LXMERT, a multi-modal pre-trained transformer. We further show that dense retrieval significantly outperforms sparse retrieval that uses object expansion. Moreover, dense retrieval matches the performance of sparse retrieval that leverages human-generated captions.
Accepted to SIGIR'21 as a short paper
References in corpus (9)
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- REALM: Retrieval-Augmented Language Model Pre-Training
- A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
- Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering
- Sparse, Dense, and Attentional Representations for Text Retrieval
- Open-Retrieval Conversational Question Answering
- Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
- Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks