Learning Compositional Rules via Neural Program Synthesis

Abstract

Many aspects of human reasoning, including language, require learning rules from very little data. Humans can do this, often learning systematic rules from very few examples, and combining these rules to form compositional rule-based systems. Current neural architectures, on the other hand, often fail to generalize in a compositional manner, especially when evaluated in ways that vary systematically from training. In this work, we present a neuro-symbolic model which learns entire rule systems from a small set of examples. Instead of directly predicting outputs from inputs, we train our model to induce the explicit system of rules governing a set of previously seen examples, drawing upon techniques from the neural program synthesis literature. Our rule-synthesis approach outperforms neural meta-learning techniques in three domains: an artificial instruction-learning domain used to evaluate human learning, the SCAN challenge datasets, and learning rule-based translations of number words into integers for a wide range of human languages.

arXiv commentNeurIPS 2020. Code can be found at https://github.com/mtensor/rulesynthesis

References in corpus (9)

• Deep Learning: A Critical Appraisal
• DeepCoder: Learning to Write Programs
• RobustFill: Neural Program Learning under Noisy I/O
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Neural Sketch Learning for Conditional Program Generation
• Automatic Program Synthesis of Long Programs with a Learned Garbage Collector
• Learning to Infer Program Sketches
• Write, Execute, Assess: Program Synthesis with a REPL
• Selecting Representative Examples for Program Synthesis

Cited by in corpus (19)

• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• A Benchmark for Systematic Generalization in Grounded Language Understanding
• Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
• Compositional Generalization by Learning Analytical Expressions
• Human-like generalization in a machine through predicate learning
• A Theory of Relation Learning and Cross-domain Generalization
• An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
• Emergent Symbols through Binding in External Memory
• Compositional Generalization via Neural-Symbolic Stack Machines
• Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
• Learning Task-General Representations with Generative Neuro-Symbolic Modeling
• Revisiting Iterative Back-Translation from the Perspective of Compositional Generalization
• The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• Conditional Teaching Size
• Grounded Graph Decoding Improves Compositional Generalization in Question Answering
• Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
• Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling
• Enhancing Reinforcement Learning with discrete interfaces to learn the Dyck Language