Centered Self-Attention Layers

Ameen Ali, Tomer Galanti, Lior Wolf

The self-attention mechanism in transformers and the message-passing mechanism in graph neural networks are repeatedly applied within deep learning architectures. We show that this…