A simple neural network module for relational reasoning

Abstract

Relational reasoning is a central component of generally intelligent behavior, but has proven difficult for neural networks to learn. In this paper we describe how to use Relation Networks (RNs) as a simple plug-and-play module to solve problems that fundamentally hinge on relational reasoning. We tested RN-augmented networks on three tasks: visual question answering using a challenging dataset called CLEVR, on which we achieve state-of-the-art, super-human performance; text-based question answering using the bAbI suite of tasks; and complex reasoning about dynamic physical systems. Then, using a curated dataset called Sort-of-CLEVR we show that powerful convolutional networks do not have a general capacity to solve relational questions, but can gain this capacity when augmented with RNs. Our work shows how a deep learning architecture equipped with an RN module can implicitly discover and learn to reason about entities and their relations.

References in corpus (12)

• Semi-Supervised Classification with Graph Convolutional Networks
• Deep Convolutional Networks on Graph-Structured Data
• Learning Convolutional Neural Networks for Graphs
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Tracking the World State with Recurrent Entity Networks
• Towards Deep Symbolic Reinforcement Learning
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Discovering objects and their relations from entangled scene representations
• A Pooling Approach to Modelling Spatial Relations for Image Retrieval and Annotation
• An Analysis of Visual Question Answering Algorithms

Cited by in corpus (140)

• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Towards artificial general intelligence via a multimodal foundation model
• Graph networks as learnable physics engines for inference and control
• Non-Local Recurrent Network for Image Restoration
• Attentional Pooling for Action Recognition
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• Learning to Count Objects in Natural Images for Visual Question Answering
• FiLM: Visual Reasoning with a General Conditioning Layer
• Deformable ConvNets v2: More Deformable, Better Results
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• Compositional Attention Networks for Machine Reasoning
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• DeepProbLog: Neural Probabilistic Logic Programming
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Multicolumn Networks for Face Recognition
• Object Level Visual Reasoning in Videos
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Empirical Bayes Transductive Meta-Learning with Synthetic Gradients
• Ontology Reasoning with Deep Neural Networks
• Relation Networks for Object Detection
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Differentiable Physics-informed Graph Networks
• Hyperbolic Attention Networks
• Learning to Make Analogies by Contrasting Abstract Relational Structure
• Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• CoPhy: Counterfactual Learning of Physical Dynamics
• FreeDOM: A Transferable Neural Architecture for Structured Information Extraction on Web Documents
• Temporal Relational Reasoning in Videos
• Directed Acyclic Graph Neural Networks
• VAIN: Attentional Multi-agent Predictive Modeling
• Discrete and continuous representations and processing in deep learning: Looking forward
• Scones: Towards Conversational Authoring of Sketches
• Text as Neural Operator: Image Manipulation by Text Instruction
• 3D Human Pose Estimation with Relational Networks
• Spatial-Temporal Relation Networks for Multi-Object Tracking
• Iterative Visual Reasoning Beyond Convolutions
• Relational recurrent neural networks
• The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning
• Self-Attentive Associative Memory
• DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer
• Visual Semantic Reasoning for Image-Text Matching
• Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
• Solving the Same-Different Task with Convolutional Neural Networks
• In Defense of Grid Features for Visual Question Answering
• Shallow2Deep: Indoor Scene Modeling by Single Image Understanding
• Relational Generalized Few-Shot Learning
• Graph-Based Global Reasoning Networks
• Recurrent Relational Networks
• MEUZZ: Smart Seed Scheduling for Hybrid Fuzzing
• Tunneling Neural Perception and Logic Reasoning through Abductive Learning
• A Perspective on Objects and Systematic Generalization in Model-Based RL
• TERMCast: Temporal Relation Modeling for Effective Urban Flow Forecasting
• Factorised Neural Relational Inference for Multi-Interaction Systems
• Visual Entailment Task for Visually-Grounded Language Learning
• Multitask Learning on Graph Neural Networks: Learning Multiple Graph Centrality Measures with a Unified Network
• GNN is a Counter? Revisiting GNN for Question Answering
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Dynamically Pruned Message Passing Networks for Large-Scale Knowledge Graph Reasoning
• Large-scale Collaborative Filtering with Product Embeddings
• GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognition
• Knowledge-Aided Open-Domain Question Answering
• DeepLogic: Towards End-to-End Differentiable Logical Reasoning
• Learning Visual Abstract Reasoning through Dual-Stream Networks
• ALBA : Reinforcement Learning for Video Object Segmentation
• Beyond the Self: Using Grounded Affordances to Interpret and Describe Others' Actions
• Compact Global Descriptor for Neural Networks
• Scene Gated Social Graph: Pedestrian Trajectory Prediction Based on Dynamic Social Graphs and Scene Constraints
• Attention on Abstract Visual Reasoning
• Learning Representations of Sets through Optimized Permutations
• A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
• Relational Action Forecasting
• On transfer learning using a MAC model variant
• ToyArchitecture: Unsupervised Learning of Interpretable Models of the World
• 3DRM:Pair-wise relation module for 3D object detection
• LinkNet: Relational Embedding for Scene Graph
• Zero-Shot Transfer VQA Dataset
• SOGNet: Scene Overlap Graph Network for Panoptic Segmentation
• Working Memory Networks: Augmenting Memory Networks with a Relational Reasoning Module
• DenseImage Network: Video Spatial-Temporal Evolution Encoding and Understanding
• Deep execution monitor for robot assistive tasks
• Predicting the Popularity of Online Videos via Deep Neural Networks
• Feed-Forward Neural Networks Need Inductive Bias to Learn Equality Relations
• TallyQA: Answering Complex Counting Questions
• A Study of Compositional Generalization in Neural Models
• One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems
• Grounded Objects and Interactions for Video Captioning
• Graph Neural Processes: Towards Bayesian Graph Neural Networks
• PVRNet: Point-View Relation Neural Network for 3D Shape Recognition
• Untrained neural networks can demonstrate memorization-independent abstract reasoning
• Topos and Stacks of Deep Neural Networks
• Refining Source Representations with Relation Networks for Neural Machine Translation
• The relational processing limits of classic and contemporary neural network models of language processing
• Dynamic Input Structure and Network Assembly for Few-Shot Learning
• Modularity Matters: Learning Invariant Relational Reasoning Tasks
• Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment
• Sentence Encoding with Tree-constrained Relation Networks
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• On the Transferability of VAE Embeddings using Relational Knowledge with Semi-Supervision
• Question Guided Modular Routing Networks for Visual Question Answering
• Pairwise Relations Discriminator for Unsupervised Raven's Progressive Matrices
• Knowledge Integration Networks for Action Recognition
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• Linguistically Driven Graph Capsule Network for Visual Question Reasoning
• Learning Graph Representation via Formal Concept Analysis
• Visual Sensor Network Reconfiguration with Deep Reinforcement Learning
• Semi-Supervised Few-Shot Atomic Action Recognition
• Few Shot Learning with Simplex
• Neural Consciousness Flow
• Multi-layer Relation Networks
• Dilated DenseNets for Relational Reasoning
• Cascaded Mutual Modulation for Visual Reasoning
• Neural Compositional Denotational Semantics for Question Answering
• Evaluating Theory of Mind in Question Answering
• Compositional Attention: Disentangling Search and Retrieval
• Relational dynamic memory networks
• Towards Robust Pattern Recognition: A Review
• A New Benchmark and Progress Toward Improved Weakly Supervised Learning
• Transfer Learning in Visual and Relational Reasoning
• Reasoning-Modulated Representations
• Image interpretation by iterative bottom-up top-down processing
• Learning Actor Relation Graphs for Group Activity Recognition
• HRGE-Net: Hierarchical Relational Graph Embedding Network for Multi-view 3D Shape Recognition
• Subgraph Retrieval Enhanced by Graph-Text Alignment for Commonsense Question Answering
• Two pathways to resolve relational inconsistencies
• Compact Deep Aggregation for Set Retrieval
• Learning to Represent and Predict Sets with Deep Neural Networks
• Spot What Matters: Learning Context Using Graph Convolutional Networks for Weakly-Supervised Action Detection
• Behaviour-conditioned policies for cooperative reinforcement learning tasks
• Query-Focused Scenario Construction
• Relationships from Entity Stream
• Relational Mimic for Visual Adversarial Imitation Learning
• Face-Focused Cross-Stream Network for Deception Detection in Videos
• Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning
• Composing Text and Image for Image Retrieval - An Empirical Odyssey
• SBNet: Segmentation-based Network for Natural Language-based Vehicle Search