Learning to Compose Neural Networks for Question Answering

Abstract

We describe a question answering model that applies to both images and structured knowledge bases. The model uses natural language strings to automatically assemble neural networks from a collection of composable modules. Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning, with only (world, question, answer) triples as supervision. Our approach, which we term a dynamic neural model network, achieves state-of-the-art results on benchmark datasets in both visual and structured domains.

Cited by in corpus (57)

• Neural Architecture Search with Reinforcement Learning
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• DyNet: The Dynamic Neural Network Toolkit
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Modular Multitask Reinforcement Learning with Policy Sketches
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• FiLM: Visual Reasoning with a General Conditioning Layer
• Long Short-Term Memory-Networks for Machine Reading
• Differentiable Learning of Logical Rules for Knowledge Base Reasoning
• Compositional Attention Networks for Machine Reasoning
• End-to-End Differentiable Proving
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Deep Learning with Dynamic Computation Graphs
• TensorLog: A Differentiable Deductive Database
• Towards an integration of deep learning and neuroscience
• Generating Visual Explanations
• Neural Programmer: Inducing Latent Programs with Gradient Descent
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Memorize or generalize? Searching for a compositional RNN in a haystack
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Visual Question Answering: A Survey of Methods and Datasets
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Beyond Visual Semantics: Exploring the Role of Scene Text in Image Understanding
• TensorLog: Deep Learning Meets Probabilistic DBs
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• DVQA: Understanding Data Visualizations via Question Answering
• Neural Program Synthesis with Priority Queue Training
• An Analysis of Visual Question Answering Algorithms
• Task-driven Visual Saliency and Attention-based Visual Question Answering
• Question-Guided Hybrid Convolution for Visual Question Answering
• Utilizing Large Scale Vision and Text Datasets for Image Segmentation from Referring Expressions
• Modular Generative Adversarial Networks
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• Hierarchical Question Answering for Long Documents
• MarioQA: Answering Questions by Watching Gameplay Videos
• Textually Enriched Neural Module Networks for Visual Question Answering
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (Short Version)
• TallyQA: Answering Complex Counting Questions
• Topos and Stacks of Deep Neural Networks
• Visual Question: Predicting If a Crowd Will Agree on the Answer
• Spatial Memory for Context Reasoning in Object Detection
• Multi-Task Spatiotemporal Neural Networks for Structured Surface Reconstruction
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• Web Question Answering with Neurosymbolic Program Synthesis
• Neural Compositional Denotational Semantics for Question Answering
• Cascaded Mutual Modulation for Visual Reasoning
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Memory and attention in deep learning
• Proposing Plausible Answers for Open-ended Visual Question Answering
• Pay Attention to Those Sets! Learning Quantification from Images
• Scaffolding Networks: Incremental Learning and Teaching Through Questioning
• General AI Challenge - Round One: Gradual Learning
• User Personalized Satisfaction Prediction via Multiple Instance Deep Learning