Measuring abstract reasoning in neural networks

Abstract

Whether neural networks can learn abstract reasoning or whether they merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation `regimes' in which the training and test data differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test sets differ only minimally, and we present a novel architecture, with a structure designed to encourage reasoning, that does significantly better. When we vary the way in which the test questions and training data differ, we find that our model is notably proficient at certain forms of generalisation, but notably weak at others. We further show that the model's ability to generalise improves markedly if it is trained to predict symbolic explanations for its answers. Altogether, we introduce and explore ways to both measure and induce stronger abstract reasoning in neural networks. Our freely-available dataset should motivate further progress in this direction.

arXiv commentICML 2018

Cited by in corpus (68)

• How Powerful are Graph Neural Networks?
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• Abstraction and Analogy-Making in Artificial Intelligence
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• Weakly-Supervised Disentanglement Without Compromises
• What Can Neural Networks Reason About?
• Analysing Mathematical Reasoning Abilities of Neural Models
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Generalization and Representational Limits of Graph Neural Networks
• On the Binding Problem in Artificial Neural Networks
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Multi-Label Contrastive Learning for Abstract Visual Reasoning
• The Expressive Power of Graph Neural Networks: A Survey
• RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
• Kandinsky Patterns
• The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning
• Solving the Same-Different Task with Convolutional Neural Networks
• Learning by Abstraction: The Neural State Machine
• Solving Raven's Progressive Matrices with Neural Networks
• Emergent Symbols through Binding in External Memory
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• From Local Structures to Size Generalization in Graph Neural Networks
• Effective Abstract Reasoning with Dual-Contrast Network
• Learning Visual Abstract Reasoning through Dual-Stream Networks
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• Neuro-Symbolic Artificial Intelligence: Current Trends
• HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
• Disentangled Representations from Non-Disentangled Models
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• A Study of Compositional Generalization in Neural Models
• Neural Theorem Provers Do Not Learn Rules Without Exploration
• Topos and Stacks of Deep Neural Networks
• Generating Correct Answers for Progressive Matrices Intelligence Tests
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Neural Status Registers
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• Distinguishing rule- and exemplar-based generalization in learning systems
• Pairwise Relations Discriminator for Unsupervised Raven's Progressive Matrices
• Few-Shot Abstract Visual Reasoning With Spectral Features
• Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
• Multi-Granularity Modularized Network for Abstract Visual Reasoning
• On the Transferability of VAE Embeddings using Relational Knowledge with Semi-Supervision
• A Memory-Augmented Neural Network Model of Abstract Rule Learning
• Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
• Odd-One-Out Representation Learning
• Multi-layer Relation Networks
• SPARE3D: A Dataset for SPAtial REasoning on Three-View Line Drawings
• Selective Replay Enhances Learning in Online Continual Analogical Reasoning
• Modelling the development of counting with memory-augmented neural networks
• Evaluating the Apperception Engine
• Abstract Reasoning via Logic-guided Generation
• Modeling Gestalt Visual Reasoning on the Raven's Progressive Matrices Intelligence Test Using Generative Image Inpainting Techniques
• Data augmentation by morphological mixup for solving Raven's Progressive Matrices
• ACRE: Abstract Causal REasoning Beyond Covariation
• Computational principles of intelligence: learning and reasoning with neural networks
• Unsupervised Abstract Reasoning for Raven's Problem Matrices
• Analogical and Relational Reasoning with Spiking Neural Networks
• Toward Building Science Discovery Machines
• Understanding in Artificial Intelligence
• DAReN: A Collaborative Approach Towards Reasoning And Disentangling
• Logic and the $2$-Simplicial Transformer
• PQA: Perceptual Question Answering
• Learning Canonical Transformations