Logic and the $2$-Simplicial Transformer

Abstract

We introduce the $2$-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive bias for logical reasoning in the context of deep reinforcement learning.

References in corpus (13)

• Neural Message Passing for Quantum Chemistry
• Learning Important Features Through Propagating Activation Differences
• Understanding Black-box Predictions via Influence Functions
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Generating Long Sequences with Sparse Transformers
• Universal Transformers
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• Measuring abstract reasoning in neural networks
• Understanding intermediate layers using linear classifier probes
• Neural Logic Machines
• Simplex2Vec embeddings for community detection in simplicial complexes
• Modeling Compositionality with Multiplicative Recurrent Neural Networks
• Sobczyk's simplicial calculus does not have a proper foundation