VQA-LOL: Visual Question Answering under the Lens of Logic
arXiv:2002.08325
Abstract
Logical connectives and their implications on the meaning of a natural language sentence are a fundamental aspect of understanding. In this paper, we investigate whether visual question answering (VQA) systems trained to answer a question about an image, are able to answer the logical composition of multiple such questions. When put under this \textit{Lens of Logic}, state-of-the-art VQA models have difficulty in correctly answering these logically composed questions. We construct an augmentation of the VQA dataset as a benchmark, with questions containing logical compositions and linguistic transformations (negation, disjunction, conjunction, and antonyms). We propose our {Lens of Logic (LOL)} model which uses question-attention and logic-attention to understand logical connectives in the question, and a novel Fréchet-Compatibility Loss, which ensures that the answers of the component questions and the composed question are consistent with the inferred logical operation. Our model shows substantial improvement in learning logical compositions while retaining performance on VQA. We suggest this work as a move towards robustness by embedding logical connectives in visual understanding.
Accepted to ECCV 2020
References in corpus (7)
- Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
- A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
- Compositional Vector Space Models for Knowledge Base Completion
- Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning