Learning to Reason: End-to-End Module Networks for Visual Question Answering

Abstract

Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems. For example, to answer "is there an equal number of balls and boxes?" we can look for balls, look for boxes, count them, and compare the results. The recently proposed Neural Module Network (NMN) architecture implements this approach to question answering by parsing questions into linguistic substructures and assembling question-specific deep networks from smaller modules that each solve one subtask. However, existing NMN implementations rely on brittle off-the-shelf parsers, and are restricted to the module configurations proposed by these parsers rather than learning them from data. In this paper, we propose End-to-End Module Networks (N2NMNs), which learn to reason by directly predicting instance-specific network layouts without the aid of a parser. Our model learns to generate network structures (by imitating expert demonstrations) while simultaneously learning network parameters (using the downstream task loss). Experimental results on the new CLEVR dataset targeted at compositional question answering show that N2NMNs achieve an error reduction of nearly 50% relative to state-of-the-art attentional approaches, while discovering interpretable network architectures specialized for each question.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Neural Architecture Search with Reinforcement Learning
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• Stacked Attention Networks for Image Question Answering
• Learning to Compose Neural Networks for Question Answering
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• Segmentation from Natural Language Expressions
• Modeling Relationships in Referential Expressions with Compositional Modular Networks

Cited by in corpus (55)

• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• A simple neural network module for relational reasoning
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• FiLM: Visual Reasoning with a General Conditioning Layer
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Compositional Attention Networks for Machine Reasoning
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• MAttNet: Modular Attention Network for Referring Expression Comprehension
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Neural Module Networks for Reasoning over Text
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• CoPhy: Counterfactual Learning of Physical Dynamics
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• Discrete and continuous representations and processing in deep learning: Looking forward
• DVQA: Understanding Data Visualizations via Question Answering
• Neural Program Synthesis with Priority Queue Training
• DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Question-Guided Hybrid Convolution for Visual Question Answering
• A Survey on Semantic Parsing
• Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
• Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewards
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• Semantic Parsing with Syntax- and Table-Aware SQL Generation
• Factor Graph Attention
• Multi-Step Inference for Reasoning Over Paragraphs
• Meta Module Network for Compositional Visual Reasoning
• Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• Information Maximizing Visual Question Generation
• Learning Visual Question Answering by Bootstrapping Hard Attention
• A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
• On transfer learning using a MAC model variant
• Zero-Shot Transfer VQA Dataset
• Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding
• HySTER: A Hybrid Spatio-Temporal Event Reasoner
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• Question Guided Modular Routing Networks for Visual Question Answering
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• Modularized Textual Grounding for Counterfactual Resilience
• Neural Compositional Denotational Semantics for Question Answering
• Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Toward Imitating Visual Attention of Experts in Software Development Tasks
• Cascaded Mutual Modulation for Visual Reasoning
• Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks
• Reasoning Over History: Context Aware Visual Dialog
• Can We Automate Diagrammatic Reasoning?
• Sequential Coordination of Deep Models for Learning Visual Arithmetic