Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

Abstract

We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-problems. The essential component of the model is a novel attention mechanism, called TP-Attention, which explicitly encodes the relations between each Transformer cell and the other cells from which values have been retrieved by attention. TP-Attention goes beyond linear combination of retrieved values, strengthening representation-building and resolving ambiguities introduced by multiple layers of standard attention. The TP-Transformer's attention maps give better insights into how it is capable of solving the Mathematics Dataset's challenging problems. Pretrained models and code will be made available after publication.

References in corpus (9)

• Training Very Deep Networks
• Visualizing and Measuring the Geometry of BERT
• Stabilizing Transformers for Reinforcement Learning
• Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
• Augmenting Self-attention with Persistent Memory
• Learning to Reason with Third-Order Tensor Products
• Systematic Generalization: What Is Required and Can It Be Learned?
• Open Sesame: Getting Inside BERT's Linguistic Knowledge
• A Perspective on Objects and Systematic Generalization in Model-Based RL

Cited by in corpus (21)

• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Deep Learning Based Text Classification: A Comprehensive Review
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Logic Tensor Networks
• Scaling Laws for Autoregressive Generative Modeling
• Hopfield Networks is All You Need
• On the Binding Problem in Artificial Neural Networks
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks
• Evaluating Logical Generalization in Graph Neural Networks
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Vector Symbolic Finite State Machines in Attractor Neural Networks
• LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning
• Learning Symbolic Rules for Reasoning in Quasi-Natural Language
• The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
• Teaching Temporal Logics to Neural Networks
• A Reinforcement Learning Environment for Mathematical Reasoning via Program Synthesis
• Learning Associative Inference Using Fast Weight Memory
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language
• Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages
• Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
• Generating Symbolic Reasoning Problems with Transformer GANs