Dynamic Memory Networks for Visual and Textual Question Answering
arXiv:1603.01417
Abstract
Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.
References in corpus (2)
Cited by in corpus (155)
- On the Opportunities and Risks of Foundation Models
- Recent Trends in Deep Learning Based Natural Language Processing
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- A simple neural network module for relational reasoning
- Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
- Pointer Sentinel Mixture Models
- Deep Learning Based Text Classification: A Comprehensive Review
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Quasi-Recurrent Neural Networks
- Collaborative Memory Network for Recommendation Systems
- Learning Multiagent Communication with Backpropagation
- Visual Question Answering: Datasets, Algorithms, and Future Challenges
- Multimodal Residual Learning for Visual QA
- URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection
- Long Short-Term Memory-Networks for Machine Reading
- Multimodal Machine Learning: A Survey and Taxonomy
- ImageNet pre-trained models with batch normalization
- FigureQA: An Annotated Figure Dataset for Visual Reasoning
- Compositional Attention Networks for Machine Reasoning
- Recent Advances in Deep Learning: An Overview
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
- Visual Reference Resolution using Attention Memory for Visual Dialog
- Query-Reduction Networks for Question Answering
- Neural Memory Networks for Seizure Type Classification
- Focal Visual-Text Attention for Visual Question Answering
- Training Recurrent Answering Units with Joint Loss Minimization for VQA
- Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
- Multi-modal Deep Analysis for Multimedia
- Small Sample Learning in Big Data Era
- Attentive Explanations: Justifying Decisions and Pointing to the Evidence
- Vision-to-Language Tasks Based on Attributes and Attention Mechanism
- Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
- Hierarchical Memory Networks
- C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- Dual Attention Networks for Multimodal Reasoning and Matching
- High-Order Attention Models for Visual Question Answering
- Visual Question Answering: A Survey of Methods and Datasets
- Attention-Aware Face Hallucination via Deep Reinforcement Learning
- Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
- Attentive Memory Networks: Efficient Machine Reading for Conversational Search
- An Introductory Survey on Attention Mechanisms in NLP Problems
- FVQA: Fact-based Visual Question Answering
- Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs
- Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
- Creativity: Generating Diverse Questions using Variational Autoencoders
- One-shot Learning for Question-Answering in Gaokao History Challenge
- Pedestrian Attribute Recognition: A Survey
- Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks
- Signed Distance-based Deep Memory Recommender
- Iterative Visual Reasoning Beyond Convolutions
- Auto-Encoding Scene Graphs for Image Captioning
- VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
- Motion-Appearance Co-Memory Networks for Video Question Answering
- MemexQA: Visual Memex Question Answering
- Task-driven Visual Saliency and Attention-based Visual Question Answering
- Understanding Early Word Learning in Situated Artificial Agents
- An Analysis of Visual Question Answering Algorithms
- Passage Retrieval for Outside-Knowledge Visual Question Answering
- Attention on Attention: Architectures for Visual Question Answering (VQA)
- Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
- Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering
- Learning by Abstraction: The Neural State Machine
- Knowledge Graph Embedding with Entity Neighbors and Deep Memory Network
- Learning Convolutional Text Representations for Visual Question Answering
- VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
- Sliced Recurrent Neural Networks
- DualNet: Domain-Invariant Network for Visual Question Answering
- AMC: Attention guided Multi-modal Correlation Learning for Image Search
- Recurrent Autoregressive Networks for Online Multi-Object Tracking
- Visual Question Answering with Memory-Augmented Networks
- Pre-Trained Models: Past, Present and Future
- Unsupervised Learning from Continuous Video in a Scalable Predictive Recurrent Network
- Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition
- Learning Visual Knowledge Memory Networks for Visual Question Answering
- Factor Graph Attention
- Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
- Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modeling
- Visual Text Correction
- Reciprocal Attention Fusion for Visual Question Answering
- Integrating both Visual and Audio Cues for Enhanced Video Caption
- Learning Visual Storylines with Skipping Recurrent Neural Networks
- An Empirical Evaluation of Visual Question Answering for Novel Objects
- A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
- Gated End-to-End Memory Networks
- Interpretable Spatio-temporal Attention for Video Action Recognition
- Sparse Attentive Memory Network for Click-through Rate Prediction with Long Sequences
- Learning Sentence Representation with Guidance of Human Attention
- Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields
- Hierarchical Memory Networks for Answer Selection on Unknown Words
- Learning Visual Question Answering by Bootstrapping Hard Attention
- Fooling Vision and Language Models Despite Localization and Attention Mechanism
- Incorporating Glosses into Neural Word Sense Disambiguation
- DeepLogic: Towards End-to-End Differentiable Logical Reasoning
- Interpretable Visual Question Answering by Reasoning on Dependency Trees
- Dual ResGCN for Balanced Scene GraphGeneration
- Multimodal Memory Modelling for Video Captioning
- Finding ReMO (Related Memory Object): A Simple Neural Architecture for Text based Reasoning
- Attention on Abstract Visual Reasoning
- Neuro-Symbolic Artificial Intelligence: Current Trends
- Video Highlight Prediction Using Audience Chat Reactions
- A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
- VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles
- Opinion Recommendation using Neural Memory Model
- MarioQA: Answering Questions by Watching Gameplay Videos
- Learning Operations on a Stack with Neural Turing Machines
- Zero-Shot Transfer VQA Dataset
- Dual Recurrent Attention Units for Visual Question Answering
- Movie Question Answering: Remembering the Textual Cues for Layered Visual Contents
- Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding
- Working Memory Networks: Augmenting Memory Networks with a Relational Reasoning Module
- Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
- Location-aware Graph Convolutional Networks for Video Question Answering
- Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task
- Be Concise and Precise: Synthesizing Open-Domain Entity Descriptions from Facts
- Visual Fashion-Product Search at SK Planet
- Spatial Memory for Context Reasoning in Object Detection
- Dynamic Time-Aware Attention to Speaker Roles and Contexts for Spoken Language Understanding
- Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms
- Multilevel Language and Vision Integration for Text-to-Clip Retrieval
- Reasoning About Human-Object Interactions Through Dual Attention Networks
- Robust and Scalable Differentiable Neural Computer for Question Answering
- OrthoSeg: A Deep Multimodal Convolutional Neural Network for Semantic Segmentation of Orthoimagery
- Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog
- New Ideas and Trends in Deep Multimodal Content Understanding: A Review
- Multi-sense Definition Modeling using Word Sense Decompositions
- Dynamically Context-Sensitive Time-Decay Attention for Dialogue Modeling
- FAQ-based Question Answering via Knowledge Anchors
- Non-local Recurrent Neural Memory for Supervised Sequence Modeling
- Learning Predicates as Functions to Enable Few-shot Scene Graph Prediction
- Exploiting Contextual Information via Dynamic Memory Network for Event Detection
- Iterative Document Representation Learning Towards Summarization with Polishing
- Extending Compositional Attention Networks for Social Reasoning in Videos
- Think Visually: Question Answering through Virtual Imagery
- The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
- Dynamic Graph Generation Network: Generating Relational Knowledge from Diagrams
- Transfer Learning in Visual and Relational Reasoning
- DCR-Net: A Deep Co-Interactive Relation Network for Joint Dialog Act Recognition and Sentiment Classification
- Improved RAMEN: Towards Domain Generalization for Visual Question Answering
- Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
- Zero-Shot Open-Book Question Answering
- Holistic Multi-modal Memory Network for Movie Question Answering
- Learning Question-Guided Video Representation for Multi-Turn Video Question Answering
- The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
- Pay More Attention - Neural Architectures for Question-Answering
- Uncertainty based Class Activation Maps for Visual Question Answering
- An In-depth Walkthrough on Evolution of Neural Machine Translation
- The emergent algebraic structure of RNNs and embeddings in NLP
- Pay Attention to Those Sets! Learning Quantification from Images
- Semi-supervised Visual Feature Integration for Pre-trained Language Models
- Learning Invariants through Soft Unification
- Traversing the Continuous Spectrum of Image Retrieval with Deep Dynamic Models