Learning to Reason with Third-Order Tensor Products

Abstract

We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data. This improves symbolic interpretation and systematic generalisation. Our architecture is trained end-to-end through gradient descent on a variety of simple natural language reasoning tasks, significantly outperforming the latest state-of-the-art models in single-task and all-tasks settings. We also augment a subset of the data such that training and test data exhibit large systematic differences and show that our approach generalises better than the previous state-of-the-art.

Cited by in corpus (17)

• Hopfield Networks is All You Need
• Long-term Forecasting using Higher Order Tensor RNNs
• On the Binding Problem in Artificial Neural Networks
• Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
• Metalearned Neural Memory
• Training Agents using Upside-Down Reinforcement Learning
• HUBERT Untangles BERT to Improve Transfer across NLP Tasks
• A Perspective on Objects and Systematic Generalization in Model-Based RL
• Learning Associative Inference Using Fast Weight Memory
• On the Memory Mechanism of Tensor-Power Recurrent Models
• Memory and attention in deep learning
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
• Deep Learning: Our Miraculous Year 1990-1991
• A Simple Recurrent Unit with Reduced Tensor Product Representations
• Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages
• Learning with Holographic Reduced Representations
• Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization