The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR

Abstract

Visual QA is a pivotal challenge for higher-level reasoning, requiring understanding language, vision, and relationships between many objects in a scene. Although datasets like CLEVR are designed to be unsolvable without such complex relational reasoning, some surprisingly simple feed-forward, "holistic" models have recently shown strong performance on this dataset. These models lack any kind of explicit iterative, symbolic reasoning procedure, which are hypothesized to be necessary for counting objects, narrowing down the set of relevant objects based on several attributes, etc. The reason for this strong performance is poorly understood. Hence, our work analyzes such models, and finds that minor architectural elements are crucial to performance. In particular, we find that \textit{early fusion} of language and vision provides large performance improvements. This contrasts with the late fusion approaches popular at the dawn of Visual QA. We propose a simple module we call Multimodal Core, which we hypothesize performs the fundamental operations for multimodal tasks. We believe that understanding why these elements are so important to complex question answering will aid the design of better-performing algorithms for Visual QA while minimizing hand-engineering effort.

arXiv commentPresented at ECCV'18 Workshop on Shortcomings in Vision and Language

References in corpus (2)

• VQA: Visual Question Answering
• Learning Visual Question Answering by Bootstrapping Hard Attention

Cited by in corpus (7)

• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• Leveraging Medical Visual Question Answering with Supporting Facts
• Answer Them All! Toward Universal Visual Question Answering Models
• Answering Questions about Data Visualizations using Efficient Bimodal Fusion
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• What is needed for simple spatial language capabilities in VQA?
• Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning