Rethinking Graph Transformers with Spectral Attention

Abstract

In recent years, the Transformer architecture has proven to be very successful in sequence processing, but its application to other data structures, such as graphs, has remained limited due to the difficulty of properly defining positions. Here, we present the $\text{Spectral Attention Network}$ (SAN), which uses a learned positional encoding (LPE) that can take advantage of the full Laplacian spectrum to learn the position of each node in a given graph. This LPE is then added to the node features of the graph and passed to a fully-connected Transformer. By leveraging the full spectrum of the Laplacian, our model is theoretically powerful in distinguishing graphs, and can better detect similar sub-structures from their resonance. Further, by fully connecting the graph, the Transformer does not suffer from over-squashing, an information bottleneck of most GNNs, and enables better modeling of physical phenomenons such as heat transfer and electric interaction. When tested empirically on a set of 4 standard datasets, our model performs on par or better than state-of-the-art GNNs, and outperforms any attention-based model by a wide margin, becoming the first fully-connected architecture to perform well on graph benchmarks.

arXiv commentAccepted in Proceedings of NeurIPS 2021

References in corpus (17)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Geometric deep learning: going beyond Euclidean data
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• How Powerful are Graph Neural Networks?
• Residual Gated Graph ConvNets
• Principal Neighbourhood Aggregation for Graph Nets
• Benchmarking Graph Neural Networks
• Differentiable Graph Module (DGM) for Graph Convolutional Networks
• On the Bottleneck of Graph Neural Networks and its Practical Implications
• Are Transformers universal approximators of sequence-to-sequence functions?
• Partial Functional Correspondence
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• Deep Green Function Convolution for Improving Saliency in Convolutional Neural Networks

Cited by in corpus (9)

• Do Transformers Really Perform Bad for Graph Representation?
• SIGformer: Sign-aware Graph Transformer for Recommendation
• Graph Neural Networks with Learnable Structural and Positional Representations
• LPFormer: An Adaptive Graph Transformer for Link Prediction
• A Retrieve-and-Read Framework for Knowledge Graph Link Prediction
• Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures
• Improving the Expressiveness of $K$-hop Message-Passing GNNs by Injecting Contextualized Substructure Information
• Hierarchical Position Embedding of Graphs with Landmarks and Clustering for Link Prediction
• Rankformer: A Graph Transformer for Recommendation based on Ranking Objective