Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
arXiv:1906.00180
Abstract
Can neural nets learn logic? We approach this classic question with current methods, and demonstrate that recurrent neural networks can learn to recognize first order logical entailment relations between expressions. We define an artificial language in first-order predicate logic, generate a large dataset of sample 'sentences', and use an automatic theorem prover to infer the relation between random pairs of such sentences. We describe a Siamese neural architecture trained to predict the logical relation, and experiment with recurrent and recursive networks. Siamese Recurrent Networks are surprisingly successful at the entailment recognition task, reaching near perfect performance on novel sentences (consisting of known words), and even outperforming recursive networks. We report a series of experiments to test the ability of the models to perform compositional generalization. In particular, we study how they deal with sentences of unseen length, and sentences containing unseen words. We show that set-ups using LSTMs and GRUs obtain high scores on these tests, demonstrating a form of compositionality.
12 pages, 5 figures
References in corpus (8)
- ADADELTA: An Adaptive Learning Rate Method
- Reasoning about Entailment with Neural Attention
- Deep Learning: A Critical Appraisal
- Neural GPUs Learn Algorithms
- Logic Tensor Networks: Deep Learning and Logical Reasoning from Data and Knowledge
- Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
- Learning Continuous Semantic Representations of Symbolic Expressions
- Graded Entailment for Compositional Distributional Semantics
Cited by in corpus (7)
- Compositional Generalization by Learning Analytical Expressions
- Hierarchical Poset Decoding for Compositional Generalization in Language
- Revisiting Iterative Back-Translation from the Perspective of Compositional Generalization
- Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?
- Word Interdependence Exposes How LSTMs Compose Representations
- How BPE Affects Memorization in Transformers
- LSTMs Compose (and Learn) Bottom-Up