Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning

Abstract

Humans have an inherent ability to learn novel concepts from only a few samples and generalize these concepts to different situations. Even though today's machine learning models excel with a plethora of training data on standard recognition tasks, a considerable gap exists between machine-level pattern recognition and human-level concept learning. To narrow this gap, the Bongard problems (BPs) were introduced as an inspirational challenge for visual cognition in intelligent systems. Despite new advances in representation learning and learning to learn, BPs remain a daunting challenge for modern AI. Inspired by the original one hundred BPs, we propose a new benchmark Bongard-LOGO for human-level concept learning and reasoning. We develop a program-guided generation technique to produce a large set of human-interpretable visual cognition problems in action-oriented LOGO language. Our benchmark captures three core properties of human cognition: 1) context-dependent perception, in which the same object may have disparate interpretations given different contexts; 2) analogy-making perception, in which some meaningful concepts are traded off for other meaningful concepts; and 3) perception with a few samples but infinite vocabulary. In experiments, we show that the state-of-the-art deep learning methods perform substantially worse than human subjects, implying that they fail to capture core human cognition properties. Finally, we discuss research directions towards a general architecture for visual reasoning to tackle this benchmark.

arXiv comment22 pages, NeurIPS 2020

References in corpus (11)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• A Simple Neural Attentive Meta-Learner
• Neural-Symbolic Learning and Reasoning: A Survey and Interpretation
• The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
• Measuring abstract reasoning in neural networks
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• Analysing Mathematical Reasoning Abilities of Neural Models
• Visual Concept-Metaconcept Learning
• Solving Bongard Problems with a Visual Language and Pragmatic Reasoning

Cited by in corpus (8)

• Abstraction and Analogy-Making in Artificial Intelligence
• A Review of Emerging Research Directions in Abstract Visual Reasoning
• Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices
• One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems
• Flexible Compositional Learning of Structured Visual Concepts
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• Tracking Without Re-recognition in Humans and Machines